Roberto Rojas's Blog

Open Source Integration Happy Hour

2009-07-16T06:57:00.000-07:00

Last week I had the pleasure to enjoy a few beers with my co-workers (Rod Biresch, Steve Smith, Tom Purcell, and Ken Rimple) and have a nice conversation about SOA, Integration, related technologies, use of ESBs, and the current state of things. We spoke about our experiences on integration projects, best practices, and the various technologies we have used in the past. For those interested in listening in on our conversation the PostCast(Episode 35), as well other interesting Postcasts, can be found here. I hope you find it useful and enjoy it.

Camel Component For Amazon SQS

2009-01-16T20:58:00.000-08:00

What Is Amazon SQS?
As its name implies Amazon Simple Queue Service (Amazon SQS) is a simplified version of a Messaging system. The great thing about it is its reliability and high scalability features.
Amazon SQS makes it easy to build an automated workflow, working in close conjunction with the Amazon Elastic Compute Cloud (Amazon EC2) and the other AWS infrastructure web services.
You can learn more about Amazon SQS here.

The Problem
My colleagues (Tom Purcell and Rod Biresch) and myself have been working on a demo application for Chariot Solutions' Emerging Technologies for the Enterprise conference and the need for a transport component to simplify the access of Amazon SQS arose.
We have decided to use FUSE ESB(ServiceMix) 4 as the services integration platform for the demo application, and knowing that Apache Camel collaborates very well with ServiceMix, we decided to use it for the routing and transformation needs.

Currently, neither ServiceMix, nor Camel provide a component to access Amazon SQS. Besides this limitation the remaining capabilities provided by these products still made it a great choice for the demo application.

To learn more about the demo application, please visit Tom Purcell's blog.
I also recommend attending to the Tom Purcell's presentation titled "Bus In the Cloulds" where he'll be speaking about Could Computing use cases and how Open Source Integration technologies can help resolve issues encountered while integrating with business partners.

Another important part in production applications is management and monitoring. To learn more about how to monitor this application, please visit Rod Biresch's blog.

A Solution
Apache Camel makes it relatively easy to compose your own components, so the next logical step was to try to create one. Having seen Dave Kavanagh's Typica API in action and how easy it makes the communication with SQS, we decided to try and come up with such Camel component using this API as the back-bone.

The Camel Amazon SQS Component
The basic idea is to create a component that could be used in the following fashion:
<camel:route>
<camel:from uri="file:data-in" />
<camel:convertBodyTo type="java.lang.String" />
<camel:to uri="sqs:sqsProducer?accessId=...&secretKey=...&queueName=dataQueue"/>
</camel:route>

<camel:route>
<camel:from
uri="sqs:sqsConsumer?accessId=...&secretKey=...&queueName=dataQueue" />
<camel:to uri="file:data-out" />
</camel:route>

The above example shows a Camel route that sends data found in files placed in the data directory to a SQS queue named "dataQueue".
In the to URI "sqs:sqsProducer?accessId=....", the protocol "sqs" will tell Camel what component to configure and attributes will be mapped the properties of either the Endpoint, or the Producer class.

The documentation on writing Camel Components can be found here.

Let's dive into the design of the Camel SQS component.

The main steps for writing a Camel Component are:

Write a POJO which implements the Component interface. The simplest approach is just to derive from DefaultComponent. In our case, the class SQSComponent was created. This class extends DefaultComponent.

To support auto-discovery of your component add a file to META-INF/services/org/apache/camel/component/sqs

where 'sqs' is the URI scheme for your component and any related endpoints created on the fly. The sqs file will contain the class attribute with the name of Class that implements the Component:
class=com.chariotsolutions.soalab.camel.component.sqs.SQSComponent

The Camel SQS Component is composed of the following artifacts:

The SQSComponent Class
This class serves as factory of Endpoints. When the component is being configured by Camel, the method of the createEndpoint is called. This method creates and configures the Endpoint (SQSEndpoint).

The SQSEndpoint Class
This class serves as a factory for Producers, Consumers, and Exchanges.
After the SQSComponent.createEndpoint is execute, an instance of the class SQSEndoint is created.
This class contains the properties used by the SQSConsumer class. These properties are:
private String accessId;
private String secretKey;
private String queueName;

This class extends ScheduledPollEndpoint and is used to create instances of the SQSProducer and SQSConsumer. The use of the ScheduledPollEndpoint is with the intention to use polling facilities it provides to query the SQS Queue at a specified time intervals.

The SQSConsumer Class
This class extends the ScheduledPollConsumer class to indicate that it will be excueted in poll intervals.
The poll intervals are configured in the Component URI using the consumer.initialDelay and consumer.delay attributes like so:

<from uri="sqs:sqsConsumer?...&consumer.initialDelay=1000&consumer.delay=25000" />

The poll() will be executed based on the configuration specified above.
This method uses the Typica API to interact with the Amazon SQS Web services. If a message was found in the SQS Queue, it will
be forward to the handleMessage. This method will set the message in a wrapper class called SQSObject, place it in the SQSExchange
and send it flow of the route using the synchronous Processor by default. The attribute asyncProcess=true would cause it to use
the AsyncProcess.

The SQSProducer Class
This class extends the DefaultProducer class.
It processes the message coming from the Camel flow and sends it to the SQS queue, then it sets the assigned message id to
the SQSObject, which is later added to the SQSExchange in case other components need it down the road.

The SQSExchange Class
This The base message exchange interface providing access to the request, response and fault instances.

The SQSObject Class
This is convenient POJO that wraps the payload and properties coming from the SQS Queue.

Taking the Camel SQS Component For A Ride
To see the Camel SQS Component in action, I created a small sample application.
The sample application can be found in here.

Here are the steps to build and run the application:

If you haven't already done so, create an Amazon SQS account using the "Sign Up" button in this page.
Extract the sample code to a directory of your choice. For example, /tmp/sqs-sample. You should see 2 directories in there, camel-sqs (camel sqs component) and camel-sqs-route (sample camel route).
Navigate to the camel-sqs directory and build the code using: mvn clean install
Navigate to the camel-sqs-route directory.
Edit the file src/main/resources/aws.properties. Specify aws.accessId and aws.secretKey for your Amazon SQS account.
Edit the file src/main/resources/META-INF/spring/camel-context.xml. Find the file endpoints and modify them to point to the directories of your choice. The File-In endpoint is where you would supply the data files to be sent to the SQS queue. The File-Out endpoint is the directory where the files containing the contents of the SQS queue will be placed on.
Build the code using: mvn clean install
Run the code using: mvn camel:run
You should be able to place your sample text files directory pointed by the File-In endpoint. After a few seconds the output files will be placed in directory pointed by File-Out.

Conclusion
Amazon SQS combined with ServiceMix and Camel are a great platform for building your IaaS (Infrastructure as a Service) projects.
Hopefully future versions of Apache Camel will include such component. The Camel CAMEL-1432 JIRA issue has been created for this reason.

Happy Cloud Computing!

Recommended Sites
Tom Purcell's Blog
Rod Biresch's Blog
Amazon SQS
FUSE ESB (ServiceMix) 4
Apache Camel
Chariot Solutions

RESTful Web Services using IONA FUSE ESB 4.0 Preview and Spring Dynamic Modules

2008-09-12T06:44:00.000-07:00

I recently started doing a substantial amount of research on ServiceMix 4 and OSGi (particularly Spring Dynamic Modules) and it's interesting to see the capabilities you can gain when you combine the two.

I have been playing around with the samples applications and looking at the code behind both ServiceMix 4 and Spring DM, and I felt like I should share some of my findings.
In this installment we'll look at a sample application that combines RESTful style of Web Services and Spring Dynamic Modules (Spring DM).

One way of providing RESTful services from within ServiceMix is by using the CXF Components that come with the product. As explained here, CXF provides different ways of exposing RESTful services. For the sample application I'll use the JAX-RS (JSR-311) API to expose the sample Weather application that comes with Spring Dynamic Modules from within FUSE ESB 4.0 Preview (ServiceMix 4).

The Ingredients

IONA FUSE ESB 4.0 Preview
Spring Dynamic Modules 1.2.0 M1
Java 1.5+
Maven 2.0.9
CXF-SpringDM-Weather Sample application SVN Repository
Subversion Client (SVN command line, Tortoisevn)

IONA FUSE ESB 4.0 Preview Configuration

Download IONA FUSE ESB 4.0 Preview from the link above and install it. The installation of product is really straight forward and choosing the default configuration should be just fine.
Go to the examples directory of the FUSE ESB product and build the cxf-osgi project. This step prepares your local maven repository with the artifacts required to build the sample application. If you run the following, you should be just fine: mvn install

Spring Dynamic Modules Installation

Download Spring Dynamic Modules from the link above and install it. For the sample I'll assume the installation directory is C:\dev\spring-osgi-1.2.0-m1
Go to the samples directory (C:\dev\spring-osgi-1.2.0-m1\src\samples) of the Spring DM distribution and execute: mvn -Dmaven.test.skip=true clean install -P felix This step will help with the deployment of the weather service from within the ServiceMix console.

Sample Application Installation and Build

Using a Subversion client, checkout the sample application from here:
SVN Repository
I had to provide a URL link instead of the URL itself to avoid the word wrapping caused by such long URL. You can open your browser to the URL above, copy and paste the address to your svn client. I'll assume the code will be checked out to the directory C:\dev\cxf-springdm-weather
Install the required bundles for the sample application.
From the directory
C:\dev\cxf-springdm-weather\lib execute the following command:
mvn install:install-file -DgroupId=javax.ws.rs -DartifactId=jsr311-osgi-api -Dversion=0.8 -Dpackaging=jar -Dfile=jsr311-osgi-api-0.8.jar
Change to the C:\dev\cxf-springdm-weather directory and build the application with:
mvn clean install
If everything goes well, maven will create a target directory with a jar file with the name
cxf-springdm-weather-4.0.0.0-fuse.jar

Deployment

The CXF implementation of JAX-RS depends on two OSGI bundle jars (jettison-1.0.1.jar and jsr311-osgi-api-0.8.jar). I had to convert these jars to OSGi bundles and I'm providing them within lib directory of the sample application so that you don't have to convert them yourself.

Here is how you would deploy the sample application from within ServiceMix console. These steps use the "osgi" command to install and start the required bundles:

Launch the ServiceMix Kernel by running
bin/servicemix
in the root dir of FUSE ESB distribution.
osgi install file:/C:/dev/cxf-springdm-weather/lib/jettison-1.0.1.jar
osgi install -s file:/C:/dev/cxf-springdm-weather/lib/jsr311-osgi-api-0.8.jar
Use the command "osgi list" to find the bundle id for the bundle with the name "Apache CXF Bundle Jar (2.1.1.fuse)" and execute the command "osgi refresh ". In my system the bundle id is 56, so the command I used was: osgi refresh 56.
osgi install -s mvn:org.springframework.osgi.samples/weather-dao/1.2.0-m1
osgi install -s mvn:org.springframework.osgi.samples/weather-service/1.2.0-m1
osgi install -s file:/C:/dev/cxf-springdm-weather/target/cxf-springdm-weather-4.0.0.0-fuse.jar

At this point you should see an output similar to this one within the ServiceMix console:

servicemix> Instantiating org.apache.servicemix.examples.restful.springdm.WeatherServiceFrontEnd
class org.apache.servicemix.examples.restful.springdm.WeatherServiceFrontEnd setting weatherService: org.springframework.osgi.samples.weather.service.WeatherServiceImpl@5c0dbc55
Sep 12, 2008 10:41:40 AM org.apache.cxf.endpoint.ServerImpl initDestination
INFO: Setting the server's publish address to be /weather

Testing The Application

Now, just open your browser and go to the following url:

http://localhost:8080/cxf/weather/service/historical-high/11-11-2002

The date must be in the format mm-dd-yyyy. If you provide an invalid date an exception will be displayed in the ServiceMix console and today's date will be used for processing.

Your browser should display this:

Sample Application Code Walk-through

The following diagram depicts the request/response flow of the Sample application. Notice how the CXF component is serving as a proxy between the HTTP/WS component and the WeatherServiceFrontEnd Bean. Similarly, Spring DM serves as a proxy to the registered OSGi services, in this case, Weather-Service service and the Weather-dao service.

Spring DM's Weather Application

First let me start explaining little bit about the Spring DM's weather sample application.
The Spring DM weather application comes with some other OSGi bundles, but we only needed the weather-service and weather-dao bundles for the sample applciation.
If you look at the context file
weather/src/main/resources/META-INF/spring/weather-service-context.xml, you'll find the <osgi:service ref="weatherService"> .... </osgi:service> tag. This tag registers the weatherService bean (defined in the same context file) inside the OSGi registry. Later, we'll look at how we get a reference to this service and inject it to our RESTful web service bean.

The cxf-springdm-weather bundle (the front end of the sample application) depends on the service exposed by the weather-service bundle and the weather-service bundle depends on the service exposed by the weather-dao bundle. This is a typical design of applications that take advantage of the modularity provided by OSGi. Of course, Spring DM makes it really easy to accomplish such design by applying IoC principals.

The WeatherServiceFrontEnd Bean

On the CXF/JAX-RS side, we start looking at the org.apache.servicemix.examples.restful.springdm.WeatherServiceFrontEnd class.
This class is nothing more than a POJO with JAX-RS annotations and with a dependency on an object implementing the org.springframework.osgi.samples.weather.service.WeatherService
interface. An instance of this interface will be injected using Spring DM.

In a nutshell, the JAX-RS is a combination of annotations that help mapping the URI and HTTP request parameters to a POJO.
The @Path("/service/") maps the URL immediately after the /cxf, which is the root of this application as we specified it in the xbeans.xml context file. We'll be talking about the configuration in the context file in just a minute.
The @ProduceMime("application/xml") says that this application will be returning xml data.

If you look at the method getHistoricalHighForDate(...), you'll notice that here we used some other JAX-RS annotations.
The @Path("/historical-high/{date}") maps this to a URI underneath /cxf/weather/service.
The @PathParam("date") maps the {date} specified by the above @Path to an argument of the getHistoricalHighForDate method.
I purposely didn't use any XML API to make the code of the sample application easier to read and follow. As you can see the code is really straight forward and all it does is delegate the call to implementation of the WeatherService and wrap in a XML string the results.
More details on JAX-RS and how is used with CXF can be found here.

Note: JAX-RS needs to know what JAX-RS implementation would be used. This is done by creating a file called "javax.ws.rs.ext.RuntimeDelegate" in the META-INF/services directory. The contents of the file specify the name of the class performing this implementation.

The Sample Application Context File

The context file, located in META-INF/spring/xbean.xml, is used to configure the RESTful service, CXF and inject the weather service. At the start of the context file, you'll find the usual namepace definitions, including those for JAX-RS and Spring DM. Then, there are a set of context imports:

<import resource="classpath:META-INF/cxf/cxf.xml" />
<import resource="classpath:META-INF/cxf/cxf-extension-jaxrs-binding.xml" />
<import resource="classpath:META-INF/cxf/cxf-extension-http.xml" />
<import resource="classpath:META-INF/cxf/osgi/cxf-extension-osgi.xml" />

These imports configure CXF's HTTP transport, hooks JAX-RS with CXF, and configure CXF within ServiceMix and the OSGi container.
Then, there is the definition of the WeatherServiceFrontEnd bean.

<bean id="weatherServiceFrontEnd" class="org.apache.servicemix.examples.restful.springdm.WeatherServiceFrontEnd" >
<property name="weatherService">
<springdm-osgi:reference interface="org.springframework.osgi.samples.weather.service.WeatherService" timeout="5000" />
</property>
</bean>

This is where the magic of Spring DM and Spring IoC is used to inject the weather-service to the weather RESTful service bean.
The <springdm-osgi:reference ...> tag finds a service in the OSGi registry with the interface org.springframework.osgi.samples.weather.service.WeatherService
and injects it to the weatherService property of the org.apache.servicemix.examples.restful.springdm.WeatherServiceFrontEnd bean.

Lastly, the configuration of the RESTful web service by mapping it to the "/cxf/weather" URI using the address attribute of the JAX-RS component tag:

<jaxrs:server id="weatherService" address="/weather" >
<jaxrs:serviceBeans>
<ref bean="weatherServiceFrontEnd" />
</jaxrs:serviceBeans>
</jaxrs:server>

As you can see in the snippet of code above, the reference of the WeatherServiceFrontEnd bean is injected into the JAX-RS component.

OSGi Bundles

The last thing to explain is how to help the OSGi container find the required classes for the various bundles of an application. This is usually done by specifying package imports/exports in the META-INF/MANIFEST.MF file. If you look closely at the provided pom.xml file, you'll notice that I'm using a ServiceMix 4 specific artifact as the parent for this project. This is due to the fact that I've used one of the sample's pom.xml file as starting point for this project. I'm guessing that down the road FUSE ESB 4.0 will come with maven archetypes, as its predecessor did, to facilitate the start of new projects. Back with the maven project, the maven-bundle-plugin plug-in is used to take care of generating the MANIFEST.MF file for us. This is done at the moment when the application is being built by the maven build process. All we have to do is specify the packages to import/export in the plug-in's configuration section. In there you'll find the packages of JAX-RS, CXF, the weather-service, among others.

For a more in-depth look OSGi bundles, visit Costin Leau's Blog.

Final Words

The combination of an ESB like ServiceMix on top of an OSGi Container, all that with the flexibility and easy of use of Spring Dynamic Modules makes this an interesting product to use for your Enterprise Application Integration needs.
I look forward to the GA release of the FUSE ESB 4.0 product.

Recommended Reading