Enterprise Integration Zone is brought to you in partnership with:

Bilgin Ibryam is a software engineer with Master's degree in Computer Science and currently working for Red Hat in London. He is also the author of "Instant Apache Camel Message Routing" book, an open source enthusiast, Apache OFBiz, and Apache Camel committer. In his spare time, he enjoys contributing to open source projects and blogging at ofbizian.com. Bilgin is a DZone MVB and is not an employee of DZone and has posted 22 posts at DZone. You can read more from them at their website. View Full User Profile

Image Loader Powered by Apache Camel

04.26.2013
| 4355 views |
  • submit to reddit

 

Live #Olympics Image Stream

(The application has been improved and moved to www.livephotostream.com read about the move here)
This is a very short post about a very simple application way back during the London 2012 Olympics and based on Apache Camel examples, I've created an application that displays twitter images related to the Olympics in real time. It listens for tweets containing images, filters out duplicates, and sends the images to the browser using websockets every 5 seconds. See it live running on amazon micro instance or check it out from my github account.

This is an end-to-end push application: once a user pushes an image to twitter, twitter pushes the image to the camel application, and then the camel application pushes the image to all the clients. It is built using only "free stuff": Twitter's free streaming API, Apache Camel framework, Amazon's free micro instance and done during my free time. Here is the essence:
public void configure() throws Exception {

    from("twitter://streaming/filter?type=event&keywords=" + searchTerm)

            .to("log:tweetStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

            .process(new ImageExtractor())

            .process(new Statistics())

            .filter(body().isInstanceOf(Tweet.class))

            .idempotentConsumer(header(UNIQUE_IMAGE_URL), MemoryIdempotentRepository.memoryIdempotentRepository(10000))

            .to("log:imageStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")

            .throttle(1).timePeriodMillis(5000).asyncDelayed().callerRunsWhenRejected(false)

            .marshal().json(JsonLibrary.Jackson)

            .to("websocket:camelympics?sendToAll=true");
}

Line by line explanation
Twitter pushes tweets containing the #Olympics and #London2012 tags to the app:
from("twitter://streaming/filter?type=event&keywords=" + searchTerm)
Log statistical information about the number of messages every minute. Not visible the users:
.to("log:tweetStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")
Extract images from the tweets which has media associated with:
.process(new ImageExtractor())
Put the current number of tweets and images in the message:
.process(new Statistics())
Filter out all the tweets which doesn't contain images:
.filter(body().isInstanceOf(Tweet.class))
Filter out duplicated images, identified by their url:
.idempotentConsumer(header(UNIQUE_IMAGE_URL), MemoryIdempotentRepository.memoryIdempotentRepository(10000))
Log again, the messages which reached this far in the route:
.to("log:imageStream?level=INFO&groupInterval=60000&groupDelay=60000&groupActiveOnly=false")
Let images go with 5 seconds difference, so the user can enjoy it. Also important - don't block the twitter listener by using callerRunsWhenRejected if the image buffer fills up, Twitter will block you:
.throttle(1).timePeriodMillis(5000).asyncDelayed().callerRunsWhenRejected(false)
Serialize into json:
.marshal().json(JsonLibrary.Jackson)
Push it to the users:
.to("websocket:camelympics?sendToAll=true");
The application can be run also locally and allows filtering images not only for the Olympics but for any keywords passed as argument. Don't forget to use your own twitter oath tokens when using locally though.

Enjoy the Olympics.

Published at DZone with permission of Bilgin Ibryam, author and DZone MVB. (source)

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)