Enterprise Integration Zone is brought to you in partnership with:

Masoud Kalali has a software engineering degree and has been working on software development projects since 1998. He has experience with a variety of technologies (.NET, J2EE, CORBA, and COM+) on diverse platforms (Solaris, Linux, and Windows). His experience is in software architecture, design, and server-side development. Masoud has published several articles at Java.net and Dzone. He has authored multiple refcards, published by Dzone, including but not limited to Using XML in Java, Java EE Security and GlassFish v3 refcardz. He is one of the founder members of NetBeans Dream Team and a GlassFish community spotlighted developer. Recently Masoud's new book, GlassFish Security has been published which covers GlassFish v3 security and Java EE 6 security. Masoud's main area of research and interest includes service-oriented architecture and large scale systems' development and deployment and in his leisure time he enjoys photography, mountaineering and camping. Masoud's can be followed at his Twitter account. Masoud has posted 82 posts at DZone. You can read more from them at their website. View Full User Profile

Navigating the Integration Landscape - Claus Ibsen on Apache Camel

04.02.2009
| 20734 views |
  • submit to reddit

 DZone had a chance to sit down with Claus Ibsen, author of the recently published Refcard Enterprise Integration Patterns with Apache Camel. In this interview, Claus, a Principal Engineer at Progress Software in the FUSE team, discusses best practices and pitfalls associated with application integration, and how Apache Camel addresses some of these challenges.  He surveys the open source ESB landscape, describes where Camel fits in and gives some valuable advice on how to properly evaluate and select the right ESB solution.

DZone: Hi Claus, could you tell us a little bit about yourself and your background in software integration?

Claus Ibsen: Yeah sure. I hold a position as Principal Engineer at Progress Software in the FUSE Team (http://fusesource.com), where we specialize in open source integration. I am very fortunate to work full time on Apache Camel and our FUSE empowered products and tools.

I started in the IT industry back in 1998 as a traditional developer on the AS/400 platform. I later changed to develop software on the PC platform doing integration to the AS/400. And in the year 2000 I jumped on board the Java platform developing middleware on top of the AS/400 platform. So I started my career doing software integration using legacy and proprietary protocols such as EDI, MQ, socket, batch files etc. The dot com era took me to the land of the J2EE and web. I was technical architect and lead developer on several major projects using the latest J2EE stack money could buy. The bubble burst and in 2003 I was contracted as a technical architect and consultant to work in China (Shanghai), on a new integration infrastructure for custom clearance for the local government in Shanghai. The system handles a very high load of transactions per day as its handling payments between governments, banks and transport agencies for to entire Shanghai region. It was a great experience to live and work in a different culture than the western. A year later I returned to Denmark, and the time thereafter I have mostly been a client side consultant helping clients with their integration needs.

In terms of open source, my “career” started as many others by trying frameworks and contributing back patches and fixes. As I am curios I downloaded the source code and started digging into the internals. I helps me better understand the framework and also lear new skills. That lead my into as a committer for both Apache Camel and many years ago on the WebWork/Xwork projects.

I have also developed a plugin for IDEA and released it as open source. I started using IDEA in 2002 and at that time I needed it to be able to dump logging information for my model objects and therefore I developed the generate toString() plugin. This plugin is now in safe hands as its bundled out of the box with IDEA.

DZone: How did you become involved in the Apache Camel project?

Claus: It started, when I was working as a consultant for my last client, as we where looking for a new integration platform, to replace Sun JCAPS. My client was facing difficulties to find talented people that had or wanted to work with this platform. And the platform itself has a very poor productivity and ease of use. They where looking for something that was much more agile, flexible, leaner and most importantly can run in their existing platform and infrastructure. Something that SUN had promised them, but they couldn't keep.
As I used my spare time to track emerging technologies and try out frameworks that potentially can make my day work easier, I spotted several open source frameworks such as Apache Camel, Apache ServiceMix and Mule that could fit in my clients organization. I started digging into Camel and along the way I submitted patches for bug fixes and new features etc. 3 months later James Strachan invited me in as a committer and I think the Camel Community can testimony that I have been active and devoted since. 

DZone: Apache Camel is in the line with some other Apache Foundation integration projects including ServiceMix, and Apache Synapse. Is there any relation between these two projects and Apache Camel?

Claus: Camel grew out of our experiences on ServiceMix. There was a component in ServiceMix called the eip-component which acted as the inspiration for Camel; we basically decided to spin off a separate EIP implementation using a very simple JBI-like API that could be used within ServiceMix to route between JBI endpoints - but also be usable in any other environment such as a web application, a JMS broker or web services gateway. 

DZone: It would appear that many much of the functionality that you cover in the Camel Refcard can be implemented, in a fairly non-intrusive way, using a message mediator like Apache Synapse. Why would I choose to go with Camel instead of Apache Synapse?

Claus: Here's a link to a comparison.

Camel is way more popular with a bigger more active and diverse community - its almost as popular as Struts and way more popular than all of the Apache Web Services projects put together (so more popular than Axis + Synapse et al): http://www.nabble.com/Apache-f90.html

Camel is developed first and foremost around the Enterprise Integration Patterns as a first class citizen - that is the logical model in Camel which you can work with in the DSL of your choice: currently either Java, XML, Scala, Groovy, Ruby - or Java beans with annotations. So it tries to raise the abstraction level to that of the patterns rather than be low level plumbing like previous integration technologies tended to be.

For example Synapse started life as a mediation engine for Axis which is a very different focus. 

DZone: Other than Apache Synapse, are there any other projects out there that provide the level of integration that Camel provides?

Claus: I would say that Mule and Apache ServiceMix are the only projects I am aware of that are on par with the level we have in Camel. Spring Integration is a fairly new project and it has a lot of catch up to do.

DZone: Performance is always a big concern for integration professionals. Could you share some of your experiences with Camel performance in the context of high throughput systems?

Claus: Camel is a lightweight framework; it typically doesn't affect performance in and of itself a whole lot - what does affect performance greatly is the actual endpoints and their configuration. e.g. if you are using JMS then the pooling and concurrency configuration is way more relevant to whether you use Camel or native JMS etc.

Another common performance gotcha is when doing lots of XML/XPath/XQuery routes its often advisable to transform payloads into some XML format (DOM or JAXB bean or ADS data objects) once to avoid multiple parses for each expression evaluation.

Finally another trick to boost performance when doing transacted messaging; maybe with XA - is to use transaction batches which can boost latency or perceived throughput for a single thread (though if you are running many parallel processing threads it often doesn't make a huge difference to overall system throughput). 

DZone: What 'best practices' would you advocate for folks using Camel on integration projects?

Claus: Start simple and test early using our testing framework.

James Strachan has really put great innovation into this kit that we use extensively to test Camel itself. It works well with Spring Testing or regular Junit classes. You can use spring property placeholders for the actual server names so you can build a single integration route as skeleton for all your environments.

Camel is so powerful and easy to get things done; most patterns are a single line of Java code - that folks often run before they can walk. By testing early and iteratively adding more features & requirements its easier to spot why things go wrong.

You can mix the Spring XML configuration with the power of the Java DSL.

Decouple your business and integration logic from the Camel API using Camels powerful Bean Binding. This allows you to have all your logic with no dependency at the Camel API at all.

A recent addition is the Camel web console.

That is used for runtime visualization of routes. You can browse the messages on the endpoints and you can also use it to send in new messages. Its also useable to control  routes by stopping, starting or even edit them at runtime.

And for old fashioned logging in log files we have the Tracer.

That will trace all messages and how they are routed at runtime. If it all goes to fast for you to keep track you can globally reduce the pace of the routing and thus at runtime tail a log and easier spot what happens.

DZone:  What are the most common pitfalls when using Apache Camel?

Claus: A given problem will often have multiple solutions in Camel. So choosing which solution to use in your situation or what is the best practice can lead to a bit confusion at first. Folks don't tend to use the EIP designer and debugger that often so when things go wrong its often a bit hard to visualize the routes, put breakpoints at points in the flow and debug the route in an EIP way. Its trivial to add a bean/processor within a route and use a regular Java debugger; but its way simpler using the FUSE tooling. 

DZone: When evaluating an ESB product, what are the main comparison criteria?

Claus: It tends to depend on what you need from an integration solution. Do you want a web services intermediary; a general purpose integration platform and framework or just a way to develop distributed SOA applications?

Write down your requirements. You are looking for a product that meets your requirements and not which products that has most features marked in a feature matrix.

An in the open source space you should look for a product that has a professional organization backing it. You need someone to engage for support, training, and being active maintaining and growing the community around the product.

And you should definitely look at the community. So look for how big and active is the community? How diverse it is. For instance if its a single VC backed company behind it, then who knows if it'll get dropped as a product soon?

You should also look for if the product is actively maintained and developed. The world in the IT landscape changes very rapidly, so you don't want to end up running your business on a product that gradually decline to nothing. Look for activities in the forums, mailing lists, commit log, issue tracker, blog entries, articles and books and the likes.

And when you try out the product put questions out on the forum or mailing lists to see if there is any help and response. If the product lack a feature or you have found a glitch then submit a ticket to the issue tracker and check the response. If your feature requests isn't in the roadmap for the product maybe the product isn't moving in a direction that suits your business.

Then the product itself should support standards at both the API and network layers. Standard APIs such as: REST, WS-*, JMS/Stomp/AMQP for messaging, XMPP and other of the new ones.
The product should be modular and plugin friendly. It should be possible to provide your own modules/components to the product so they work seamless with the product as if they are shipped of the out of the box.

And last but not at least about the product itself, how lightweight and flexible is it? Can you use any of your existing platform and infrastructure as runtime, or do you need to introduce a new platform to the operations division? They would sure love if it can be hosted seamless into their existing mix of servers and infrastructure.

And often overlooked but one of the key points in my optic - Power back to the developer. At any given problem it should be possible for the developer to handle a problem. I use the picture - there should be a “sweat spot” where the cursor is blinking and the developer feel home where he can write code that solves the problem. I have often seen integration with legacy system where you need to cater for all kind of corrupt data it spit out. Its often harder and more expensive to get the legacy vendor to admit and fix the problem, than have your own integration team capable of dealing with it. Standards are fine but it should be possible to deal with systems that unexpected differs from the standard.

DZone:  What advice would you give to our members that are currently involved in integration projects?

Claus: No shoe fit all sizes. You should definitely take your time to try the different products. Take responsibly for your business and integration, your vendor does not run your business and your problems might not be on their priority list. That is why I think you should build a team with experienced resources that is capable of taking up this responsibility. Its is naive to let your business in the hands of the vendor and its army of endless consultants. Use less $$$ in the hardware and platform itself and more on bringing in the right talent to your team, and/or hire independent consultants in the role as your guard dog, capable of playing in the ballparks with the vendors. Integration is more about having the right team than the platform and hardware itself. No vendor have build the nirvana product that can substitute talent and experience.

If you want to learn a bit more about what Camel is and what it can do then take time and read this excellent article by Jonathan Anstey a fellow Camel committer.

And my last advice. Take a ride on the Camel, you wont regret it.
 

AttachmentSize
claus.jpg3.27 KB
8996_small.png70.81 KB
Published at DZone with permission of its author, Masoud Kalali.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Asankha Perera replied on Thu, 2009/04/02 - 9:44am

Claus I find it very amusing how the ActiveMQ Project Management Commitee (PMC) of the ASF ignored and never replied to the email I sent on behalf of the Synapse PMC on the 24th of June 2008, about the thoroughly outdated and incorrect link you are still hosting off the Apache server. The comment "Whereas Synapse is more focussed on being a mediation engine for Axis 2; assuming a web services environment with mostly SOAP as the payloads." just shows how incompetent those who wrote the page and still refuse to update it are. Does the power of Synapse really scare you guys off so much to not update that page when you know its wrong? cheers asankha

Asankha Perera replied on Thu, 2009/04/02 - 10:02am

Claus

"Camel is way more popular with a bigger more active and diverse community - its almost as popular as Struts and way more popular than all of the Apache Web Services projects put together (so more popular than Axis + Synapse et al):"

Did you mess up your addition? Lets really check this one:

Camel Users: 121

Camel Developers: 62

Synapse Users: 160

Synapse Developers: 146

Axis2 Users: 1759

Axis2 Developers: 661

Reference: Ken Coar's page at http://people.apache.org/~coar/mlists.html

cheers

asankha

James Strachan replied on Thu, 2009/04/02 - 11:22am in response to: Asankha Perera

I've no idea what email you're talking about from the Synapse PMC - I've never seen such a thing. We didn't use old history - we used the current activity of the communities to decide. See for yourself where Apache Camel is in the rankings of current popularity and activity...

Claus Ibsen replied on Thu, 2009/04/02 - 11:30am

Using nabble to look at the number of threads posted on the user forums since 1st jan 2009 until today.

I get these figures:

Camel user forum: approx 267 posted threads on the user forum:
http://www.nabble.com/Camel---Users-f36428i245.html

Synapse user forum: approx 48 posted thread son the user forum:
http://www.nabble.com/Synapse---User-f14466i35.html

Asankha Perera replied on Thu, 2009/04/02 - 12:13pm in response to: James Strachan

James

I sent two emails on 24/6/2008 to both the ActiveMQ PMC and ServiceMIX PMC. I even have a reply to one from Guillaume Nodet time stamped "06/24/2008 01:06 PM" and copied back to the Synapse PMC saying:

"..We'll update the page asap..."

On "07/11/2008 04:15 PM" I replied back to him with a reminder:

"I guess with a busy schedule, you have not yet got the time to update that page about Synapse.. but I believe you soon will :-)"

Now check the last edited timestamp on the page: http://servicemix.apache.org/how-does-servicemix-compare-to-synapse.html which reads: "Added by James Strachan, last edited by Guillaume Nodet on Jul 11, 2008".. 

Interesting ah? ... but now what happens next.. the page then goes on to quote Dave Chappell "one of the founders of Synapse" .. about something said *before* Synapse ever started to exist. Some of those who started Synapse didn't want it to be called an ESB at that time, for reasons related to their business. For the record, some of these people never wrote a line of Synapse, so they can never know what it is! if you want stats look here: http://people.apache.org/~asankha/synapse/statsvn/. and you are better off quoting what I say about Synapse!

Then the page *again* says ".. Building an SOA on Apache Synapse would presume that all exchanges in the SOA would be made through SOAP-based Web Services..." ?? Duh! Do you expect those reading these pages to be so silly as those who write them?

James, You've proven many times on record that you never can learn about Synapse, or be truthful:

  • http://markmail.org/message/fquqptdmuficadqz
  • http://www.theserverside.com/news/thread.tss?thread_id=45743
  • Emails to the ActiveMQ PMC [*]

[*] BTW.. if you have trouble locating emails on the ActiveMQ PMC.. I can resend the old emails. Both Camel and Synapse being Apache projects, I would bet more on Ken's pages for real stats on the numbers of subscribed *users* than on something outside the ASF.. but your mileage may vary, of course!

cheers

asankha

 

 

 

 

 

James Strachan replied on Thu, 2009/04/02 - 12:46pm in response to: Asankha Perera

Sorry to hear you are so obviously upset over a small, old FAQ page. I've updated the FAQ page is that better now?

Claus Ibsen replied on Sun, 2009/04/05 - 3:09am

Asankha Perera

My mileage may vary? Well I trust well established mail archives such as nabble and markmail. And of course also google and twitter.

When I track a OS project its about the community and of course the project itself.
When a project has a live community then its a key factor indicating the project itself is sound and supports business needs.

There are many ways of tracking this but a good starting point is measuring the activities
a) activity in the community such as user forum, irc chat, blogs, twitter tweets, articles, books, session on conferences etc.
b) activity in the project such as commits, JIRA tracker, number of releases, other projects uses it as foundation, etc.

User forum activity:
We use markmail as it supports a great and simple visualization and stats:

Apache Synapse markmail stats:
Total number of posts on user forum in the year 2008: 1222
http://apache.markmail.org/search/?q=synapse+list%3Aorg.apache.ws.synapse-user+date%3A200801-200812

Total number of posts on user forum, in the first three months in 2009: 279
http://apache.markmail.org/search/?q=synapse+list%3Aorg.apache.ws.synapse-user+date%3A200901-200903


Apache Camel markmail stats:
Total number of posts on user forum in the year 2008: 4135
http://markmail.org/search/?q=camel+list%3Aorg.apache.camel.users+date%3A200801-200812

Total number of posts on user forum, in the first three months in 2009: 1298
http://markmail.org/search/?q=camel+list%3Aorg.apache.camel.users+date%3A200901-200903


Blog activity:
We use Google Blogs search to as a quick indicator for the activity related to blogs.
Note: When I type the search criteria in Google I type it as: "Apache Synapse" and "Apache Camel" eg with " " enclosing the text.

Apache Synapse: Results 1 - 10 of about 1,592 for "Apache Synapse". (0.06 seconds)
http://blogsearch.google.se/blogsearch?hl=en&ie=UTF-8&q=%22Apache+Synapse%22&btnG=Search+Blogs

Apache Camel: Results 1 - 10 of about 6,495 for "Apache Camel". (0.15 seconds)
http://blogsearch.google.se/blogsearch?hl=en&ie=UTF-8&q=%22Apache+Camel%22&btnG=Search+Blogs

And lets narrow the search to entries for this year, eg we click the "Choose dates" and type in 1/1/2009 - 4/5/2009 as date range
We want to see how the latest activity is and not if the project had peeked in the past.

Apache Synapse: Results 1 - 10 of about 48 for "Apache Synapse". (0.08 seconds)
http://blogsearch.google.se/blogsearch?hl=en&ie=UTF-8&q=%22Apache+Synapse%22&as_maxm=4&as_miny=2009&as_maxy=2009&as_minm=1&as_mind=1&as_maxd=5&as_drrb=b&ctz=-120&c1cr=1%2F1%2F2009&c2cr=4%2F5%2F2009&btnD=Go

Apache Camel: Results 1 - 10 of about 1,458 for "Apache Camel". (0.27 seconds)
http://blogsearch.google.se/blogsearch?hl=en&ie=UTF-8&q=%22Apache+Camel%22&as_maxm=&as_miny=2009&as_maxy=&as_minm=1&as_mind=1&as_maxd=&as_drrb=b&ctz=-120&c1cr=1%2F1%2F2009&c2cr=&btnD=Go

As you can see the situation changes quite dramatically.
For Apache Synapse out of the total 1592 hits the 48 of them are from 2009, approx 3%.
For Apache Camel out of the total 6495 hits the 1458 of them are from 2009, approx 22%.


Twitter tweets activity
I use twitter search to search for tweets. I type "Apache Synapse" and "Apache Camel" also with " " enclosing the text.
There are no total count, so you have to count them manually.

Apache Synapse: 13 in total
http://search.twitter.com/search?q=%22Apache+Synapse%22

Apache Camel: 87 in total
http://search.twitter.com/search?q=%22Apache+Camel%22

Activity in project, number of commits
I use markmail to count the number of commit messages

Apache Synapse: about 1108
http://markmail.org/search/?q=apache%20synapse#query:apache%20synapse%20list%3Aorg.apache.synapse.commits+page:1+state:facets

Apache Camel: about 6488
http://markmail.org/search/?q=apache%20camel#query:apache%20camel%20list%3Aorg.apache.camel.commits+page:1+state:facets


And for last year alone, eg year 2008
As the projects didn't start at same time, then we look at last year as both projects started before 2008.

Apache Synapse: about 861
http://markmail.org/search/?q=apache%20synapse#query:apache%20synapse%20list%3Aorg.apache.synapse.commits%20date%3A200801-200812%20+page:1+state:facets

Apache Camel: about 3425
http://markmail.org/search/?q=apache%20camel#query:apache%20camel%20list%3Aorg.apache.camel.commits%20date%3A200801-200812%20+page:1+state:facets


And the first three months in 2009:

Apache Synapse: about 222
http://markmail.org/search/?q=apache%20synapse#query:apache%20synapse%20list%3Aorg.apache.synapse.commits%20date%3A200901-200903%20+page:1+state:facets

Apache Camel: about 1618
http://markmail.org/search/?q=apache%20camel#query:apache%20camel%20list%3Aorg.apache.camel.commits%20date%3A200901-200903%20+page:1+state:facets

Comments: So yes my mileage vary, but they are based on google, twitter, nabble, markmail and other key services. Its all about the activities, the community is only alive if there is activity on all fronts: forums, chat, blogs, tweets, talks on conferences, books being written, google summer of code, forked projects, or the community creating services around the project or add new or related features.

For instance a company specialized in health care have build their integration platform on top of Camel: Open eHealth Integration Platform

Claus Ibsen replied on Sun, 2009/04/05 - 4:04am

I took a closer look at Kens stat site you used for your numbers, without writing what the numbers was based on, its the number of subscribers, e.g. they are not necessary related to activity.

I was a bit puzzled by the number of subscribers vs. the activity. For instance if we look at a project such as Shale, that is well frankly degrading (e.g. it is outrun by other popular web frameworks).

Kens stat for Shale:
http://people.apache.org/~coar/mlists.html#shale.apache.org
Mailing list user @ shale.apache.org
Current subscribers: 173
Total posts (814 days): 2309
Mean posts per day: 2.84

And then we look at the mail archives such as markmail to see the activity:

Number of posts in user forum in the last 6 months: 17
http://markmail.org/search/?q=%20date%3A200901-200903%20%20list%3Aorg.apache.shale.user%2F#query:list%3Aorg.apache.shale.user%2F%20date%3A200808-200904%20+page:1+state:facets

Number of posts in user forum since 2008 - today: 194, with an average of (194 posts in 465 days) = 0.4 post/day.
http://markmail.org/search/?q=list%3Aorg.apache.shale.user%2F+date%3A200801-200904+

The user activity in the user forum is degrading/very low but there are still 173 subscribers? Are these subscribers active = no.

In my point of view the number of subscribers is a questionable figure to measure or compare projects. At least the figure should not be the only figure you should aggregate other stats to give you better more sound facts when evaluating and comparing projects.

You can do the same measurement for Camel and Synapse as well, comparing the activity vs. the number of subscribers of the mailing list.

And the time is changing how people interact with a project. They use other services such as: google, blogs, twitter, feed aggregators, irc, jabber, etc. So the mailing lists is only a part of the overall activity in the community.

At the end its you who should make your own investigations and most importantly try the projects and get in contact with the community to see if its there to help you when you need assistance. If none replies your mails or posts on user forums then its frustrating and might not be the right community for you.

Daniel Kulp replied on Mon, 2009/04/06 - 8:21am

One more note about Ken's stats: not all mailing lists are "created equal". For example, all of the mailing lists in the WebServices project (and possibly Synapse, haven't checked) REQUIRE the user to subscribe to the list in order to post. The lists are not moderated at all.

Camel, on the other hand, is a moderated list. People can (and frequently do) interact with the mailing list though services such as Nabble so they DON'T have to subscribe. I know on CXF, a couple of the committers/developers don't even subscribe to the list, they interact via Nabble. That allows a lot more interaction with the community as there are a lot of people that won't go through the trouble of subscribing just to post a question/comment.

Thus, number of subscribers is pretty bogus. It definitely doesn't reflect they number of people actively participating in the community which is the important number to consider.

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.