Enterprise Integration Zone is brought to you in partnership with:

Jonathan Anstey is a software engineer with varied experience in manufacturing control systems, build infrastructure, and enterprise integration. Lately, Jon has been working on Apache Camel as a PMC member and an active committer while at FuseSource. He is also co-author of Camel in Action. Jonathan has posted 2 posts at DZone. View Full User Profile

Apache Camel: Integration Nirvana

03.23.2009
| 151444 views |
  • submit to reddit

Take any integration project and you have multiple applications talking over multiple transports on multiple platforms. As you can imagine, in large enterprise applications this can get complex very fast. Much of the complexity stems from two issues:

1.    dealing with the specifics of applications and transports, and
2.    coming up with good solutions to integration problems.

Making your applications speak transports and APIs is relatively easy on its own. I'm sure everyone knows how to send JMS messages to their broker of choice; though it still requires in depth knowledge of the JMS specification, which many developers may not have. On top of that, what happens when you want to route that JMS message to another application? You then have to take care of mapping the JMS message to the application plus handle any new concepts related to the application. Add a dozen other applications into the mix and you've got quite a headache on your hands.

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. Fortunately, most solutions to enterprise integration problems have been formalized already. Gregor Hohpe and Bobby Woolfe's book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions, boils down years of experience from enterprise architects into a set of sixty five Enterprise Integration Patterns (EIPs). This is great but we still have to hand code all parts of these patterns; these are not packaged solutions, only recommendations.   

Apache Camel was created with the intention of addressing these two issues. In this article I'll show you how it actually does this.

What is Camel?

Apache Camel is an open source Java framework that focuses on making integration easier and more accessible to developers. It does this by providing:

•    concrete implementations of all the widely used EIPs
•    connectivity to a great variety of transports and APIs
•    easy to use Domain Specific Language (DSL) to wire EIPs and transports together

Figure 1 shows how these three items actually map to Camel concepts. To give you a good understanding of how Camel is organized, we will discuss Components, Endpoints, Processors, and the Domain Specific Language (DSL). There is of course a lot more going on here under the hood but we'll leave that for another discussion.

 
 

 

Figure 1: High level view of Camel's architecture.

Components are the extension point in Camel to add connectivity to other systems. The core of Camel is very small to keep dependencies low, promote embeddability, etc. and as a result contains only 12 essential components. There are over 60 components outside the core. To expose these systems to the rest of Camel, Components provide an Endpoint interface. By using URIs, you can send or receive messages on Endpoints in a uniform way. For instance, to receive messages from a JMS queue aQueue and send them to a file system directory "c:/tmp", you could use URIs like "jms:aQueue" and "file:c:\tmp".

Processors are used to manipulate and mediate messages in between Endpoints. All of the EIPs are defined as Processors or sets of Processors. As of writing, Camel supports 41 patterns from the EIP book, 6 other integration patterns, and many other useful Processors.

To wire Processors and Endpoints together, Camel defines a Java DSL. The term DSL is used a bit loosely here as it usually implies the involvement of a compiler or interpreter that can process keywords specific to a particular domain. In Camel, DSL means a fluent Java API that contains methods named like terms from the EIP book. Its best explained with an example

from("jms:aQueue")
.filter().xpath("/person[@name='Jon']")
.to("file:c:\tmp");



Here we define a routing rule in a single Java statement that will consume messages from the "jms:aQueue" Endpoint, send them through a Message Filter Processor, which will then send on messages passing the XPath condition to the "file:c:\tmp" endpoint. Messages failing the condition will be dropped.

You can also configure your routes in a XML-based Spring configuration file. This configuration file is a lot more verbose and less auto complete friendly than the Java DSL; many prefer it though because of its direct access to Spring concepts and no requirement for compilation after changes. Here is what the earlier example would look like in Spring:

<camelContext xmlns="http://camel.apache.org/schema/spring">
<route>
<from uri="jms:aQueue"/>
<filter>
<xpath>/person[@name='Jon']</xpath>
<to uri="file:c:\tmp"/>
</filter>
</route>
</camelContext>



AttachmentSize
camelArch1.jpg47.49 KB
riderAutoEips1.jpg25.64 KB
DZone_Camel_Article_JonathanAnstey.pdf647.95 KB
Published at DZone with permission of its author, Jonathan Anstey.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Craig Wickesser replied on Tue, 2009/03/24 - 4:37am

Any chance you could provide a word document or PDF version of this article? I'd like to print it out but need something more "printer" friendly.

James Strachan replied on Tue, 2009/03/24 - 4:58am

Great article Jon! BTW the image just before "Figure 1" isn't being displayed - nor is the image on the second page

Claus Ibsen replied on Tue, 2009/03/24 - 6:09am

Great article Jonathan. Maybe the first DSL example (only 1 line of code) can be split into two lines to avoid it being clipped by the button bar.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:04am in response to: James Strachan

Weird. No clue why the images broke - it was working yesterday! I've hopefully fixed them for good now.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:06am in response to: Claus Ibsen

Good suggestion. Its fixed now.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:49am in response to: Craig Wickesser

Craig,

 I just uploaded a PDF version of this article. Enjoy!

 http://architects.dzone.com/sites/all/files/DZone_Camel_Article_JonathanAnstey.pdf

Javadevel Javad... replied on Tue, 2009/03/24 - 10:29am

Simply Awesome! Could you expand on this and give design guidelines about non-functional requirements such as security. For example, how would one secure (or add authentication) to the http endpoint used in your example. Other interesting issues in the integration space are dealing with extremely large input files, what is the best way to deal with large files and what are Camels true capabilities.

Jonathan Anstey replied on Tue, 2009/03/24 - 12:08pm in response to: Javadevel Javadevelva

To secure the HTTP endpoint you could use SSL. See http://camel.apache.org/jetty.html for more details.

To handle large messages you could do a couple of things...

1. Make sure you use input streams instead of loading messages into memory. For the HTTP endpoint example, the message is streamed by default. However, you may need to tune your route for these kinds of messages. For instance, in Listing 3 the convertBodyTo(String.class) method is used to convert the incoming stream into a String so that we can do things like JAXB unmarshalling on it. This will of course load the message into memory so we may have to do something else here to handle the large message.

2. Use the Claim Check EIP (http://camel.apache.org/claim-check.html) to store the larger portion of your message for retrieval later.

For other Camel capabilities, you can take a peek at the online documentation, Camel User Guide, and FUSE Mediation Router documentation. For specific inquires, you can alway find help on the Apache and FUSE forums.

Scott Stanlick replied on Thu, 2009/03/26 - 7:26am

Great article brother! Can you list a few of the competitors? Is Mule competing in this space? Peace, Scott

Jonathan Anstey replied on Thu, 2009/04/02 - 1:26pm in response to: Scott Stanlick

Glad you enjoyed it! Yeah, Mule is the other competitor with a few years under the belt like Camel. Two new projects that implement EIPs are Spring Integration and Project Fuji.

 I should also mention ServiceMix here which has had support for EIPs since before Camel was started. The latest version of ServiceMix supports EIPs through Camel in favour of the old EIP component. 

Charles Moulliard replied on Tue, 2009/03/31 - 5:47am

Hi John,

Congratulations for this awesome article. You have described in a 4 pages document the "quintessence" of what Camel is.

BTW : May I suggest that you add also in your dataformat section : Camel-bindy which is a new DataFormat component who allows you to map CSV file directly to your pojo (using Java Annotation), Fix messages format, and more in the future.

remark : I will publish soon a tutorial on Camel with Bindy, OSGI, CXF and ServiceMix4

Regards,

Charles

Jonathan Anstey replied on Tue, 2009/03/31 - 7:36am in response to: Charles Moulliard

Charles,

Just added it to the list. I actually wanted to use camel-bindy in the example but it is a new feature in Camel 2.0, which is not released yet (I'm using Camel 1.6). To give readers a tease, with camel-bindy you can add annotations for CSV to POJO mapping just like we did for the XML to POJO mapping. The Order class would have extra annotations like

@CsvRecord(separator = ",")
public class Order implements Serializable {
    @DataField(pos = 0)
    private String name;
    @DataField(pos = 1)
    private int amount;

 and then in our OrderRouter we can simplify the CSV transformation as follows

public class OrderRouter extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        JaxbDataFormat jaxb = new JaxbDataFormat("org.fusesource.camel");
        BindyCsvDataFormat csv = new BindyCsvDataFormat("org.fusesource.camel");        
        ...
        from("jms:incomingOrderQueue")
         .convertBodyTo(String.class)
         .choice()
           .when().method("orderHelper", "isXml")
             .unmarshal(jaxb)
             .to("jms:orderQueue")
           .when().method("orderHelper", "isCsv")
             .unmarshal(csv)        
             .to("jms:orderQueue");
    }
}

Notice how we have added a new bindy DataFormat and used it instead of the default CSV unmarshaller. In this case we have also elimintaed the need for the OrderNormalizer bean as the bindy DataFormat handles all of the CSV to POJO transformation.

Ben slim replied on Tue, 2009/03/31 - 1:17pm

Where can we get the beautifully IEP icons in riderAutoEips1_0.jpg ?

Jonathan Anstey replied on Tue, 2009/03/31 - 2:18pm in response to: Ben slim

I used the Microsoft Visio scencil from the EIP books website

http://www.enterpriseintegrationpatterns.com/download/EIP_Visio_stencil.zip

 Of course, I added a lot of extra beautification too, which just takes time :)

Ben slim replied on Tue, 2009/03/31 - 4:24pm in response to: Jonathan Anstey

Some weeks ago I tried this stencil for visio but your beautification made it look completely different. Great work !

Claus Ibsen replied on Fri, 2009/04/03 - 8:12am

Bruce Snyder posted a blog entry about EIP pattern stencils for OmniGraffle

Link to blog entry: eip-patterns-in-omnigraffle

Kai Wähner replied on Tue, 2011/01/04 - 11:22am

Very nice introduction to Apache Camel! Thank you.

Raj Kapoor replied on Mon, 2011/03/14 - 12:58pm

Great article and tutorial. Just a short note to say that I did get some errors when I tried to run this using Maven. Simply adding some repositories and plugin repositories to the POM.xml file solved my problems. Maybe you guys didn't spot this, as they already had the correct libraries previously downloaded into their repositories. Maybe you could update the pom.xml to add some repositories. Just wanted to help out. Thanks.

Jonathan Anstey replied on Thu, 2011/05/19 - 7:56am

FYI I just posted an updated version of this example at http://java.dzone.com/articles/open-source-integration-apache using the latest version of Camel as well as information about some new Eclipse tooling. Cheers, Jon

Manish Chowdhary replied on Mon, 2012/04/02 - 7:44pm

This article needs to be updated badly! GoECart

Jemmi John replied on Thu, 2012/04/19 - 12:02am

Hello,I love reading through your blog, I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. superdry// superdry sale//

Mike Lurset replied on Fri, 2013/05/17 - 10:25am

I did get some errors when I tried to run this using Maven. Simply adding some repositories and plugin repositories to the POM.xml file solved my problems. driversedguy.tumblr.com

Mike Lurset replied on Sun, 2013/05/19 - 3:30am

Apache Camel was created with the intention of addressing these two issues. In this article I'll show you how it actually does this. things to do in Las Vegas

Mike Lurset replied on Mon, 2013/06/03 - 9:36am

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. mobile news

Mike Lurset replied on Wed, 2013/07/17 - 4:10am

Here we define a routing rule in a single Java statement that will consume messages from the "jms:aQueue" Endpoint, send them through a Message Filter Processor www.esdtech.fr

Mike Lurset replied on Wed, 2013/07/17 - 9:24am

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. plombierathlon.fr

Ronaeg Osama replied on Sun, 2014/02/23 - 7:47pm

i read a lot of stuff and i found that the way of writing to clearifing that exactly want to say was very good so i am impressed and ilike to come again in future..شركة تنظيف بالرياض شركة مكافحة حشرات بالرياض

Jason Sparks replied on Wed, 2014/04/02 - 11:15pm

Got it working first time. Thanks to this!

Spectra 

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.