Enterprise Integration Zone is brought to you in partnership with:

Jonathan Anstey is a software engineer with varied experience in manufacturing control systems, build infrastructure, and enterprise integration. Lately, Jon has been working on Apache Camel as a PMC member and an active committer while at FuseSource. He is also co-author of Camel in Action. Jonathan has posted 2 posts at DZone. View Full User Profile

Apache Camel: Integration Nirvana

03.23.2009
| 168174 views |
  • submit to reddit
These are the concepts that Camel was built upon. Since then many other interesting features have been added. Details of these are left up to the reader to investigate. To get you started, some of these include:

•    Pluggable data formats and type converters for easy message transformation between Artix Data Services, CSV, EDI, Flatpack, HL7, JAXB, JSON, XmlBeans, XStream, Zip, Camel-bindy, etc.
•    Pluggable languages to create expressions or predicates for use in the DSL. Some of these languages include: EL, JXPath, Mvel, OGNL, BeanShell, JavaScript, Groovy, Python, PHP, Ruby, SQL, XPath, XQuery, etc.  
•    Support for the integration of beans and POJOs in various places in Camel.
•    Excellent support for testing distributed and asynchronous systems using a messaging approach
•    and much more...

Example

A motorcycle parts business, Rider Auto Parts, supplies parts to motorcycle manufacturers. Over the years they've changed the way they receive orders several times. Initially, orders were placed by uploading CSV files to an FTP server. The message format was later changed to XML. Currently they provide a web site to submit orders as XML messages over HTTP. All of these messages are converted to an internal POJO format before processing.

Rider Auto Parts states to any new customers to use the web interface to place orders. However, because of existing agreements with customers, they must keep all the old message formats and interfaces up and running.
Solution using EIPs

Rider Auto Parts faces a pretty common problem; over years of operation businesses acquire software baggage in the form of transports/data formats that are popular at the time. Using patterns from the EIP book we can envision the solution as something like Figure 2.

 
 

 

Figure 2: This shows the solution to Rider Auto Parts integration problem using notation from the Enterprise Integration Patterns book.

So we have several patterns in use here.

1. There are two Message Endpoints; one for FTP connectivity and another for HTTP.

2. Messages from these endpoints are fed into the incomingOrderQueue Message Channel

3. The messages are consumed from the incomingOrderQueue and routed by a Content-Based Router to one of two Message Translators. As the EIP name implies, the routing destination depends on the content of the message. In this case we need to route based on whether the content is a CSV or XML file.

4. Both Message Translators convert the message content into a POJO, which is fed into the orderQueue Message Channel.

The whole section that uses a Content-Based Router and several Message Translators is referred to as a Normalizer. This composite pattern has a unique graphic to depict it but was left out here in favor of its sub-patterns to make things clearer.

Implementation using Camel

As mentioned before, Camel has a small core set of components included by default. The rest of the components exist as separate modules. In applications that require many types of connectivity it is useful to figure out what Camel modules to include. Listing 1 shows the dependencies using Apache Maven for the Camel implementation of the Rider Auto Parts example. Of course, you don't need to use Apache Maven for dependencies - it is just the easiest way to rapidly add new dependencies to your applications. The list of dependencies includes support for core Camel, ActiveMQ, JAXB marshaling, CSV marshaling, and HTTP. To make the example easier to try out, I've opted to use the File endpoint instead of the FTP. If we were using the FTP endpoint we would need to add a dependency on the camel-ftp module as well.

Listing 1: Maven dependencies for the Camel implementation

  <dependencies>
<!-- Core Camel support -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-core</artifactId>
<version>${camel-version}</version>
</dependency>
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-spring</artifactId>
<version>${camel-version}</version>
</dependency>

<!-- ActiveMQ connectivity for Camel -->
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-camel</artifactId>
<version>${activemq-version}</version>
</dependency>

<!-- Add support for JAXB marshaling -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jaxb</artifactId>
<version>${camel-version}</version>
</dependency>

<!-- Add support for CSV marshaling -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-csv</artifactId>
<version>${camel-version}</version>
</dependency>

<!-- Add support for HTTP -->
<dependency>
<groupId>org.apache.camel</groupId>
<artifactId>camel-jetty</artifactId>
<version>${camel-version}</version>
</dependency>

<!-- Embedded ActiveMQ broker -->
<dependency>
<groupId>org.apache.activemq</groupId>
<artifactId>activemq-core</artifactId>
<version>${activemq-version}</version>
</dependency>
<dependency>
<groupId>org.apache.xbean</groupId>
<artifactId>xbean-spring</artifactId>
<version>${xbean-spring-version}</version>
</dependency>
</dependencies>




AttachmentSize
camelArch1.jpg47.49 KB
riderAutoEips1.jpg25.64 KB
DZone_Camel_Article_JonathanAnstey.pdf647.95 KB
Published at DZone with permission of its author, Jonathan Anstey.

(Note: Opinions expressed in this article and its replies are the opinions of their respective authors and not those of DZone, Inc.)

Comments

Craig Wickesser replied on Tue, 2009/03/24 - 4:37am

Any chance you could provide a word document or PDF version of this article? I'd like to print it out but need something more "printer" friendly.

James Strachan replied on Tue, 2009/03/24 - 4:58am

Great article Jon! BTW the image just before "Figure 1" isn't being displayed - nor is the image on the second page

Claus Ibsen replied on Tue, 2009/03/24 - 6:09am

Great article Jonathan. Maybe the first DSL example (only 1 line of code) can be split into two lines to avoid it being clipped by the button bar.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:04am in response to: James Strachan

Weird. No clue why the images broke - it was working yesterday! I've hopefully fixed them for good now.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:06am in response to: Claus Ibsen

Good suggestion. Its fixed now.

Jonathan Anstey replied on Tue, 2009/03/24 - 7:49am in response to: Craig Wickesser

Craig,

 I just uploaded a PDF version of this article. Enjoy!

 http://architects.dzone.com/sites/all/files/DZone_Camel_Article_JonathanAnstey.pdf

Javadevel Javad... replied on Tue, 2009/03/24 - 10:29am

Simply Awesome! Could you expand on this and give design guidelines about non-functional requirements such as security. For example, how would one secure (or add authentication) to the http endpoint used in your example. Other interesting issues in the integration space are dealing with extremely large input files, what is the best way to deal with large files and what are Camels true capabilities.

Jonathan Anstey replied on Tue, 2009/03/24 - 12:08pm in response to: Javadevel Javadevelva

To secure the HTTP endpoint you could use SSL. See http://camel.apache.org/jetty.html for more details.

To handle large messages you could do a couple of things...

1. Make sure you use input streams instead of loading messages into memory. For the HTTP endpoint example, the message is streamed by default. However, you may need to tune your route for these kinds of messages. For instance, in Listing 3 the convertBodyTo(String.class) method is used to convert the incoming stream into a String so that we can do things like JAXB unmarshalling on it. This will of course load the message into memory so we may have to do something else here to handle the large message.

2. Use the Claim Check EIP (http://camel.apache.org/claim-check.html) to store the larger portion of your message for retrieval later.

For other Camel capabilities, you can take a peek at the online documentation, Camel User Guide, and FUSE Mediation Router documentation. For specific inquires, you can alway find help on the Apache and FUSE forums.

Scott Stanlick replied on Thu, 2009/03/26 - 7:26am

Great article brother! Can you list a few of the competitors? Is Mule competing in this space? Peace, Scott

Jonathan Anstey replied on Thu, 2009/04/02 - 1:26pm in response to: Scott Stanlick

Glad you enjoyed it! Yeah, Mule is the other competitor with a few years under the belt like Camel. Two new projects that implement EIPs are Spring Integration and Project Fuji.

 I should also mention ServiceMix here which has had support for EIPs since before Camel was started. The latest version of ServiceMix supports EIPs through Camel in favour of the old EIP component. 

Charles Moulliard replied on Tue, 2009/03/31 - 5:47am

Hi John,

Congratulations for this awesome article. You have described in a 4 pages document the "quintessence" of what Camel is.

BTW : May I suggest that you add also in your dataformat section : Camel-bindy which is a new DataFormat component who allows you to map CSV file directly to your pojo (using Java Annotation), Fix messages format, and more in the future.

remark : I will publish soon a tutorial on Camel with Bindy, OSGI, CXF and ServiceMix4

Regards,

Charles

Jonathan Anstey replied on Tue, 2009/03/31 - 7:36am in response to: Charles Moulliard

Charles,

Just added it to the list. I actually wanted to use camel-bindy in the example but it is a new feature in Camel 2.0, which is not released yet (I'm using Camel 1.6). To give readers a tease, with camel-bindy you can add annotations for CSV to POJO mapping just like we did for the XML to POJO mapping. The Order class would have extra annotations like

@CsvRecord(separator = ",")
public class Order implements Serializable {
    @DataField(pos = 0)
    private String name;
    @DataField(pos = 1)
    private int amount;

 and then in our OrderRouter we can simplify the CSV transformation as follows

public class OrderRouter extends RouteBuilder {
    @Override
    public void configure() throws Exception {
        JaxbDataFormat jaxb = new JaxbDataFormat("org.fusesource.camel");
        BindyCsvDataFormat csv = new BindyCsvDataFormat("org.fusesource.camel");        
        ...
        from("jms:incomingOrderQueue")
         .convertBodyTo(String.class)
         .choice()
           .when().method("orderHelper", "isXml")
             .unmarshal(jaxb)
             .to("jms:orderQueue")
           .when().method("orderHelper", "isCsv")
             .unmarshal(csv)        
             .to("jms:orderQueue");
    }
}

Notice how we have added a new bindy DataFormat and used it instead of the default CSV unmarshaller. In this case we have also elimintaed the need for the OrderNormalizer bean as the bindy DataFormat handles all of the CSV to POJO transformation.

Ben slim replied on Tue, 2009/03/31 - 1:17pm

Where can we get the beautifully IEP icons in riderAutoEips1_0.jpg ?

Jonathan Anstey replied on Tue, 2009/03/31 - 2:18pm in response to: Ben slim

I used the Microsoft Visio scencil from the EIP books website

http://www.enterpriseintegrationpatterns.com/download/EIP_Visio_stencil.zip

 Of course, I added a lot of extra beautification too, which just takes time :)

Ben slim replied on Tue, 2009/03/31 - 4:24pm in response to: Jonathan Anstey

Some weeks ago I tried this stencil for visio but your beautification made it look completely different. Great work !

Claus Ibsen replied on Fri, 2009/04/03 - 8:12am

Bruce Snyder posted a blog entry about EIP pattern stencils for OmniGraffle

Link to blog entry: eip-patterns-in-omnigraffle

Kai Wähner replied on Tue, 2011/01/04 - 11:22am

Very nice introduction to Apache Camel! Thank you.

Raj Kapoor replied on Mon, 2011/03/14 - 12:58pm

Great article and tutorial. Just a short note to say that I did get some errors when I tried to run this using Maven. Simply adding some repositories and plugin repositories to the POM.xml file solved my problems. Maybe you guys didn't spot this, as they already had the correct libraries previously downloaded into their repositories. Maybe you could update the pom.xml to add some repositories. Just wanted to help out. Thanks.

Jonathan Anstey replied on Thu, 2011/05/19 - 7:56am

FYI I just posted an updated version of this example at http://java.dzone.com/articles/open-source-integration-apache using the latest version of Camel as well as information about some new Eclipse tooling. Cheers, Jon

Manish Chowdhary replied on Mon, 2012/04/02 - 7:44pm

This article needs to be updated badly! GoECart

Jemmi John replied on Thu, 2012/04/19 - 12:02am

Hello,I love reading through your blog, I wanted to leave a little comment to support you and wish you a good continuation. Wishing you the best of luck for all your blogging efforts. superdry// superdry sale//

Mike Lurset replied on Fri, 2013/05/17 - 10:25am

I did get some errors when I tried to run this using Maven. Simply adding some repositories and plugin repositories to the POM.xml file solved my problems. driversedguy.tumblr.com

Mike Lurset replied on Sun, 2013/05/19 - 3:30am

Apache Camel was created with the intention of addressing these two issues. In this article I'll show you how it actually does this. things to do in Las Vegas

Mike Lurset replied on Mon, 2013/06/03 - 9:36am

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. mobile news

Mike Lurset replied on Wed, 2013/07/17 - 4:10am

Here we define a routing rule in a single Java statement that will consume messages from the "jms:aQueue" Endpoint, send them through a Message Filter Processor www.esdtech.fr

Mike Lurset replied on Wed, 2013/07/17 - 9:24am

Ignoring the mechanics of how to connect with multiple transports and APIs, we can focus on the high level design of how applications interact. plombierathlon.fr

Ronaeg Osama replied on Sun, 2014/02/23 - 7:47pm

i read a lot of stuff and i found that the way of writing to clearifing that exactly want to say was very good so i am impressed and ilike to come again in future..شركة تنظيف بالرياض شركة مكافحة حشرات بالرياض

Jason Sparks replied on Wed, 2014/04/02 - 11:15pm

Got it working first time. Thanks to this!

Spectra 

Ronaeg Osama replied on Sun, 2014/08/24 - 2:28am

This is really good information I have visited this blog to read something fresh and I really admire you efforts in doing so. jailbreak jailbreak wizard application for iphone  لعبة سب واي لعبة سب واي لعبة سب واي لعبة سب واي

Peterfreeman Osama replied on Wed, 2014/06/18 - 10:11pm

 

Great Article it its really informative and innovative keep us posted with new updates. its was really valuable. thanks a lot. ايفون 6 iPhone 6 تداول العملات شركة تداول العملات افضل شركات تداول العملات

Comment viewing options

Select your preferred way to display the comments and click "Save settings" to activate your changes.