Message Ordering in JMS - Using Coherence and Hibernate

As I continue to work with JMS and WebLogic, I wonder about Message Ordering. I have looked for strategies and best practices and have finally done some experimentation that I would like to share.

One of the common patterns when working with Enterprise Integration and Messaging is the Competing Consumer Design Pattern. The same is explained very well in the book "Enterprise Integration Patterns" by Hohpe and Woolf.

As shown above we have consumers that are competing for messages on the queue. As each consumer can process out a message, the consumers scale out horizontally and improve the performance while at the same time guaranteeing high availability.

For the sake of discussion, consider that by some stroke of chance the messages 2 and 3 are for the same item, simply published one after another. With the competing consumer pattern, it is possible that message 3 gets processed before message 2 resulting in an erroneous information propagated to downstream systems such as a database thus compromising the integrity of the data.

One of the ways to prevent this scenario from happening is using a single consumer as shown in the figure below where messages in the queue will be consumed (maybe persisted to the database) in serial order by the consumer:

One problem with this approach is that now we have compromised on the availability and performance of the system as we have only a single consumer performing the task.

The JMS Specification is clear that a messaging system must deliver message in FIFO (First in-First out) order to consumers. However, the specification is silent regarding ordering when related to re-delivery of a message. In the case of a single consumer if there is a problem processing the message and it gets rolled back due to say a problem with the underlying database which the consumer is communicating with, then most providers will not attempt to redeliver the message immediately but do so after a configurable delay. Between this delay, if the database problem is cured and the consumer picks Message 3 and processes the same, then when Message 2 is delivered and processed, the downstream database would contain stale data.

One could say that any rejected message should be placed at the very front of the queue. However, if the message was a poison message and the logic to detect the same and dequeue the message were not built in the consumer, then putting the message back to the front of the queue would render the queue non-consumable, unless there is retry count configured in which case the poison message could be expired or placed in a dead letter queue after the retry count.

A strategy that can be employed where the ordering of the message would not really matter is to use the message queue simply as a notifier with the consumers going back to the source system to get the most current record to process as shown in the figure below:

In the example shown above, the producer sends a message to the queue that is only a notifier with the "RECORD_ID" of the record from the source database. The consumer upon receiving the message goes back to the source database to obtain the payload information and then process the message. By doing so, regardless of the messaging order, the most recent payload for a record is selected.

The above approach works great when one does have access to the source system and represents a tight coupling in the system. From a performance perspective, we save on the size of messages and the fact that only selects need to be issued on the source system. Further there is something to say about the simplicity of the solution. It however crumbles when the consuming end of the application does not have access to the source system resource to obtain the latest data for the record in question.

So what can we do to ensure that we process in the correct order and also have high availability and performance at the same time ? Please note that I my focus is only on the consumption ordering of the message in the consumer system. Regarding production of messages, ordering of placement and strategies is something I am not covering. That said, I am making an assumption that on the production end of the system messages will always be made available in order. Following on this assumption, I am taking the liberty of assuming that the JMS Time stamp uniquely identifies the messages in the order they were sent. Note that this assumption can be wrong but for keeping my examples tractable I am taking this liberty.

That said, I can see two strategies that could be employed. There maybe others that I would love to hear about as well.

1. Database Optimistic Lock based Versioning:

Every JMS Message sent has a time stamp associated with it. If the down stream database that the consumer accesses can maintain a table of RECORD_ID-LAST_UPDATE_DATE then by using Optimistic Versioning one can reject records whose time stamps are older than the one in the database. One can use Pessimistic locking as well, however for high scalable systems, Optimistic locking is definitely preferred.

With an optimistic locking solution employed, only if a record is more recent than the database would the message be processed. If all that one is doing is updating a downstream table with information after any business processing, then you might be able to tag a time stamp column to the table to serve the same purpose. Look at the figure below where both competing consumers are attempting to update the same table which has a last time stamp column for optimistic locking:

In the above shown example, let us say we are receiving updates to existing PERSON records. If M1 and M1' represents updates to the same person with M1 issued at time t and M1' issued at t + dT then, if C1 is the first one to update the Person, then all is good as C2 will update shortly and the record will have the most current value. If C2 updates the database record first and C1 tries to update, it will fail as the JMS time stamp on M1 is older than what is in the database for the Person.

I used Hibernate and HSQL to test this out. Hibernate has a feature for optimistic automatic versioning for detached objects. In order to employ the same, one has to use a Version property. The Version property cannot be set programatically but is auto-updated by Hibernate. For this reason the JMS Time stamp passed in cannot be used to version but can be used as part of the optimistic strategy.

I have a Person object that represents the database model and it looks like the following:

@Entity
@Table(name = "PERSON")
public class Person {
 @Id
 @AccessType(value = "property")
 private Long recordId;

 @Column(name = "FIRST_NAME", nullable = false)
 private String firstName;

 @Column(name = "LAST_NAME", nullable = false)
 private String lastName;

 @Temporal(TemporalType.TIMESTAMP)
 @Column(name = "JMS_TIMESTAMP", nullable = false)
 private Date jmsTimestamp;

 @Version
 @Column(name = "OPT_VER", nullable = false)
 @AccessType(value = "property")
 private Long version;
...
}

Using this code shown above, and the corresponding Data access tier, one can witness optimistic locking in action where messages are rejected or consumed as shown in the output below. Record 13 was rejected as there was a previous record in the database with a newer JMS Time stamp. Record 12 is accepted as although there is an existing record with ID 12, the time stamp on the database record is older than the JMS Time stamp of the message:

18:58:27 ERROR - com.welflex.activemq.AbstractListener.onMessage(35) | Rejecting the Stale Record [13]
com.welflex.exception.StaleObjectException: Stale Record. Database time for the record is [1266631107150], attempted to update with older timestamp [1266614643023]
at com.welflex.hibernate.dao.PersonDaoImpl.saveOrUpdate(PersonDaoImpl.java:27)
at com.welflex.service.PersonServiceImpl.savePerson(PersonServiceImpl.java:29)
at ...
18:58:28 INFO - com.welflex.activemq.AbstractListener.onMessage(32) | Successfully processed Record with ID [12]
......

2. Cache based Pessimistic Lock based Versioning:

In this strategy, I have employed Coherence from Oracle for the demonstration. The strategy employed is that when a message arrives it attempts to acquire a coherence lock for the record in question, upon obtaining the lock a check is performed to see if the corresponding date associated with the record is older than the date of the message being worked on. If the cached record is older, then the process is completed. In the case of the example the resulting operation is updating the record in the database. If the cached record in Coherence was older, then the message is rejected. In either case, the lock is release for the next record. A clear example of Pessimistic locking here. Coherence might allow for optimistic locking and transactions but I have not experimented with the same as yet. A figure demonstrating the strategy is shown below where a record is only updated to the database if the Cache lock is successfully acquired:

One thing to note with this solution, if there are many records, then one would need to be careful regarding how often the cache is expired of its entities. The decision of the same could be made if one has some idea on the frequency in which duplicates could be expected. Another point to note is that if all the node's in the cache collapse, and there were message's that were rolled back to be delivered later, unless the cache is primed or restored in some fashion, the solution could fail.

The code for the pessimistic locking in the cache is shown below:

public <T, E extends Exception> T doCacheAction(String cacheKey, Long timeStamp,
    CacheAction<T, E> action) throws E {
 try {
  idCache.lock(cacheKey, -1);
  Long cacheTime = (Long) idCache.get(cacheKey);
  if (cacheTime == null || timeStamp.longValue() > cacheTime.longValue()) {
    T result = action.doAction();
    idCache.put(cacheKey, timeStamp);
    return result;
  }
  throw new StaleObjectException("Coherence Cached Date [" + cacheTime
     + "] for Record    [" + cacheKey + "] is newer than passed in time ["
     + timeStamp + "]. Rejecting the record....");
  }
  finally {
   idCache.unlock(cacheKey);
 }
}

When an example is run, one can witness the following where Record 10 is successfully processed while record 13 (no wonder) is rejected:

19:24:35 INFO - com.welflex.activemq.AbstractListener.onMessage(32) | Record updated :10
19:24:35 ERROR - com.welflex.activemq.AbstractListener.onMessage(35) | Rejecting the Stale Record [13]
com.welflex.exception.StaleObjectException: Coherence Cached Date [1266632674247] for Record [13] is newer than passed in time [1266632673051]. Rejecting the record....
at com.welflex.service.CoherenceCacheServiceImpl.doCacheAction(CoherenceCacheServiceImpl.java:36)
at com.welflex.activemq.CacheMessageListener.doWithMessage(CacheMessageListener.java:28)
at com.welflex.activemq.AbstractListener.onMessage(AbstractListener.java:31)
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:543)

The Sample Code:

Attached herewith is a maven project that demonstrates the above concepts. The example sends messages of a fixed number of record id's to an Active MQ queue and has consumers that demonstrate optimistic or pessimistic locking to reject or accept messages. Spring, the only framework that has my undying admiration is used only for its Listener Container. The Coherence cache in the tests does not demonstrate a clustered cache but the same can be experimented with easily using the code provided. The tests will demonstrate both the cases mentioned above and should be runnable out of the box. I have taken some liberties with Hibernate, JMS and Coherence so don't bother pinning me to the stake for the same, I could care less, I have pinned myself in advance...;-)

Download the code from HERE and execute "mvn test" to see the demonstration of locking/unlocking.

Parting Thoughts:

1. Single Consumer:

The single consumer approach seems to be the most widespread pattern, whether in EJB/MDB land or otherwise for most message ordering solutions. It however is not a very scalable one. Further, if used, I believe it will have to maintain some state as well regarding previous messages received as it would need to deal with messages that were re-delivered. A posting on the Server side regarding EJB/MDB and ordering of messages.

2. The "Going back to the Source" pattern:

The pattern is acceptable clearly if you have access to the source. In terms of simplicity it is definitely the best possible solution to the problem. All the JMS Message becomes is an event notifier with a record id. I must however say that from a purely emotional perspective, it just feels ugly due to the tight coupling between the producer and consumer ends of the application. Again, note that it will not work when the consumer cannot access the source system.

3. Database based ordering:

Database for message ordering works well when you have control on the schema. If working with a legacy system, this can prove an impediment. Apart from not having a dependency on the source system, one advantage to the database approach is that due to the persistent nature of the data, surviving failures and message rollbacks will work really well which might prove a problem with a Caching solution.

4. Caching based ordering:

The caching based solution works when one does not have access to a datastore and cannot introduce database versioning. I am not certain how this will perform when the messages are varied, i.e., many records and have not experimented with cache priming or recovery.

Both in the case of the Database and the Caching based solution, the expectancy is definitely that each message contains a identifier of a record, where it is a synthetic id or natural id or a complex unique key that can differentiate it from other messages as its the identifier that is primal for the locking strategies to work.

It is also possible that message ordering is not an issue for many cases as one can always build a data integrity validating process to ensure integrity after the fact or depend on the fact that a future update will fix the data integrity issue if present. Another case maybe for manual intervention to fix the one off issues. Do however note that it does become an issue if real time information is required.

If the message updates are for inserts or creations, then ordering is hardly an issue. If they are updates of information then ordering becomes more important. In cases where incremental updates and their order needs to be preserved is important, then the above mentioned locking strategies will not work.

The book I mentioned talks for re-sequencing of messages and restoring order, I am however not convinced that performance will not be impacted by the same.

Oracle WebLogichas a message ordering solution in the form of Unit Of Order. However, from experiments we have found that for a clustered highly available environment, the solution does not work. For any questions on the same, mail or post.

As always, for each use case there is a specific solution that will work...I would love to hear about people's experiences with message ordering or just any thoughts. If however you are trying to promote your jewelery site, please don't post here :-)

Defensive Programming in Java

Something crossed my mind today and I thought of Blogging about it. Defensive programming seems to be running in my decaying grey cells :-).
Consider the following best practice,

public class Foo {
   public void doSomething() {
      Date date;
      Bar b = new Bar(date);
      ....
   }
}

public class Bar {
   private Date date;
   public Bar(Date date) {
     // Create a based of the provided date
     this.date = immutable(date);
   } 
   ...
}

In the above example, Bar is being safe, it is ensuring that the Date object even if passed and changed subsequently externally, will be safely used within Bar.

What I ask, is that, when working with external API, shouldn't the caller be safe as well regarding the objects it passes around ?

As an example, when one works with Streams, Sockets, Connections etc, I am of the opinion that the creator or opener of the Resource should be the one to close it. For example,

public class Foo {
  public void someOperation() {
    OutputStream os = ...;
    try {
      doSomething(os);
      ....
      // Do some other stuff with the stream
      doSomethingElse(os);
    } finally {
      os.close();
    }    
  }
  
  public void doSomething(OutputStream os) {
    Bar b = new Bar(os);
    try {
      bar.process();
    } finally {
       // Closes Bar and as a side effect closes os
       bar.close();
    }
  }
}

Now for the sake of discussion let us assume that Bar uses the Stream, but it also has a close() method associated with it. The behavior of the close() method could be to close the output stream that we created in Foo. As Bar is a third party software, one would not know the behavior of the close() method without looking at Bar's javadoc or actual code (might require decompilation). For the sake of discussion, let us say that Bar is the following code:

public class Bar {
 private OutputStream os;

 public Bar(OutputStream os) {
   this.os = os;
 }
 
 public void process() {
   ....
 }
 ...
 public void close() {
   close(os);
 }
}

Clearly our Foo code would fail as Bar's close() method is closing out the provided stream and rendering it unusable for the code in Foo. At this time, the close() of Bar method does not do anything else apart from closing the provided stream. Can we rely on the same being the only operation performed by the close() code in the future? Can we safely agree that as long as we do not call the close() method on Bar, our code in Foo is good? Can we safely walk away knowing that we don't call the close() method on Bar?

One concern that I see is that if we do not call close() on Bar and should tomorrow a new version of Bar get included in the build that is binary compatible but has other additional resources it releases as part of the close() method, what happens then ? Maybe leakage of some sort?

For example let us say that the next version of Bar does some socket management internally as well as shown below,

public class Bar {
  private Socket s;
  private OutputStream os;
  public Bar(OutputStream os) {
    this.os = os;
    this.s = new Socket(...);
  }
  ...
  public void close() {
    close(os);
    close(s);
  }
}

Third party implementation cannot be guaranteed unless the documentation certifies the same. We cannot imagine that the behavior of the close() method will not require the closing of addition resources opened by Bar.

So how does one be defensive in development in this case ? As defensive programmers, what if we provided a stream to Bar that even if Bar attempted to close() the stream, the operation would have no effect?

The same could be achieved using decorator or maybe a Dynamic Proxies where we allow every other method to go through but prevent the closing of the provided stream by Bar.

public class Foo {
  public void someOperation() {
    OutputStream os = ...;
    Bar b = new Bar();
    // Created proxy will do nothing on close() method call
    OutputStream proxy = createProxy(os);
    try {
      doSomething(proxy);
      ....
      doSomethingElse(proxy);
    } finally {
      os.close();
    }
  }
  
  public void doSomething(OutputStream os) {
    Bar b = new Bar(os);
    try {
      bar.process();
    } finally {
      // This operation now will honor the close but
      // prevent the closing of our stream as the proxy will
      // intercept and do nothing on the call to close()
      bar.close();
    }
  }
}

In the above example, the caller Foo is being careful and guaranteeing the integrity of the Stream, a way of immutability. I am probably being too dramatic. At the same time, I think there is value in being defensive. I also think that things work well when the creator is the destroyer :-). BTW this is coded entirely developed using BLOG SPOT's editor...wonder how many mistakes I have made...

RESTful HTTP and the Representation

I was looking at a BLOG posting by Jacob Kaplan-Moss and found his Conflating models and resources an interesting point. When working with a technology like Hibernate, if we were to have an Object like SuperHero, when retreiving the same, where do we stop the retreival and let lazy loading kick in? For example,

class SuperHero {
 private List<Power> powers;
 private Set<Superhero> superBuddies;
 private Set<Colleague> colleagues;
 private Set<Trophy> trophies;
 ....
}

Do we load every thing when load "Superman"? That would be quite a lot of data. What Hibernate does is use proxies that will lazily load each attribute only when asked for. One could also perform queries such that only parts of the information are returned.

If one were working in a RESTful system, when one accessed the resource, /superheroes/superman what should one get back? Would we expect the entire Superman representation? In a deeply nested model, the cost of returning such data can be considerable. I think it is important to consider the idea of lazy loading when working in REST as well and especially the word "State Transfer". I imagine one would typically want to navigate a RESTful system through different states or URLS if in HTTP. For example,

/superheroes/superman
/superheroes/superman/powers
/superheroes/superman/powers/heatvision
/superheroes/superman/superBuddies
/superheroes/superman/superBudies/Green%20Lantern

etc etc

Every Representation along the way providing information that is sufficient enough to represent the resource being queried with links to additional immediate resources. This improves the "linkedness" of your application and allows one to traverse your application by entering and leaving different states. Should you need to represent superman in his entierity, then a resource to the equivalent of /superheroes/superman/full can serve to load the shebang of information :-)

Quite Interesting, need to get my own quote someday

The 25 Best Quotes from Tech History. Think of my colleagues at work will enjoy the one on JINI. My favourite quote of course lies with HAL.

http://www.computerworld.com/s/article/9141008/25_best_quotes_from_tech_history

Ubuntu Karmic and Windows 7

Did two upgrades this weekend. Upgraded a laptop from Vista to Windows 7 and a desktop from Ubuntu Jaunty to Karmic, both 64 bit. Pleased with both upgrades so far. Vista was one of MS biggest lows as far as their OS version goes, Windows 7 seems more peppier in general so far.

I have been very pleased with Ubuntu ever since I installed the same. Have previously been a user of Mandriva, Fedora and Suse. Found mandriva comprehensive in its software offering but not very stable. I feel that Suse may be great to compete with Redhat on the server market but is no where as user friendly as Ubuntu. What I like about Ubuntu is that its pretty solid, convenient and just so easy to use. It detects the need for propriety drivers and installs in a breeze. Have dual monitor support without any xorg.conf tweaking. Printer, web cam all worked out of the box. Have it now running Skype as well without a problem.

After installing Ubuntu, I followed the following BLOG on a TODO list that really helped.
Anyway, got rid of the standard 1948 created Gnome menu in favour of AWN and now my desktop rocks.

A note for those installing eclipse 64 bit. One will find that clicking on some buttons does not work and one has use the keyboard actions to activate the buttons. The way around this problem is to set the following "export GDK_NATIVE_WINDOWS=true" prior to launching eclipse.

On the Windows 7 front, I prefer using Chrome over firefox or IE as its so much more faster on many sites that I frequent. The upgrade from Dell from Vista to Win 7 was pretty simple. Cannot watch youtube on 64 bit IE yet due to 64 flash plugin not available (no so on my Ubuntu though :-)) Don't get me wrong, I don't have anything against Win 7 (except $$$) but when compared with what FREE brings, I find it difficult to justify the extra $$$. I however do need to a Win machine due to the fact that there is no software for the iphone of any value on Linux and that there are sites that I go to such as CWTV to watch videos requiring Win or Mac. Another seller for the Win 7 is my few PC games that I have. BTW, I did not buy Win 7, its a free upgrade option.

The Mrs also is happy with Ubuntu...and as long as she is happy, I am as well :-)

Exceptions in RESTful HTTP with Restlet and JAXB

REST, or rather, RESTful HTTP and how exceptions can be handled therein is what this posting is about.

For those of us who have worked with SOAP and faults, we are familiar with how exceptions are handled. SOAP Stacks upon encountering an exception will throw a SOAP fault which is then marshalled into an exception on the consumer end of the service. SOAP faults in a way provide a protocol over standard HTTP.

When working with REST we have the concept of Resources and their Representations. An exception in a RESTful architecture is really a tuple of an HTTP Status Code and a Representation that describes the same.

The HTTP Status code in themselves are quite indicative, delving a bit into the same:

• 1XX Series - Informational - Request received and continuing processing
• 2XX Success - Action received, understood and accepted
• 3XX Redirection - Further action required to complete the request
• 4XX Client Error - Request is badly formatter or the server cannot process the request
• 5XX Server Error - Server failure on a a potentially valid request

When working with architectures that have Java on both ends of a Service, shying away from Exception handling is IMHO a self imposed handicap. In a RESTful system, one would typically have an expected representation of a resource and in the event of exceptions, have them represented as alternate results along with the appropriate error codes. The JAX-RS specification has the concept of a WebServiceException.

What I am trying to do in this BLOG is a demonstrate how exception handling can be accomplished with a framework like Restlet. I am not demonstrating a specification by any means or a wrapper protocol. For the sake of this demonstration, I will be using Restlet and JAXB. In particular, the example will rely on the XmlAdapters of JAXB extensively coupled with the Java Reflection API and Annotations. The XmlAdapters serve to translate to and from Exceptions to JAXB Representions

The scope of this example is as follows:

a. Exceptions supported are of a particular type and its subclasses only:
Exceptions are restricted to WebServiceException and its sub classes. In addition, the WebServiceException is a RuntimeException. No checked exceptions. Any exceptions thrown by a Restlet Resource that are not an instance of WebServiceException will be converted to the same prior to being marshalled. Any WebServiceException marshalled, must have a JAXB Object associated with it.

@XmlJavaTypeAdapter(WebServiceExceptionAdapter.class)
public class WebServiceException extends RuntimeException {
// Http Status code
private int status;
}

// This class extends WebServiceException but has the same body as
// WebServiceException. For this reason, there is no special XmlAdapter required.
public class OrderException extends WebServiceException {
..
}

b. XML as the media type for Transferring Exceptions:
XML is the sole media type supported for exceptions in this example. Note that it is quite trivial to change the same to accommodate additional media types like JSON etc if required. Every Exception's representation is wrapped around with a WebServiceExceptionWrapperDto. The same is to allow for sub typing. Note that the targetException will be marshalled by an appropriate XmlAdapter at Runtime.

@XmlType(name = "ExceptionWrapper", propOrder = { "targetException" })
@XmlRootElement
public class WebServiceExceptionWrapperDto<E extends WebServiceException> {
@XmlElement(name = "targetException", required = true)
public E targetException;
}

c. Rich Exceptions:
Exceptions that can be richer, i.e., more than just message, cause and stack. For example validation messages, codes, inner objects etc. A simple example of the same is shown below where the OrderValidationException contains a description and a custom validation error code. The same could easily have been a complex object if required:

@XmlJavaTypeAdapter(OrderValidationExceptionAdapter.class)
public class OrderValidationException extends WebServiceException {
// A code representing the error
private int validationCode;

// A description indicating the cause of the validation error
private String validationMessage;
...
}

// Note that the OrderValidationException Adapter is behaving like a Template method pattern
// Base marshalling to an exception is handled by the parent class with the child only providing
// specific additions. The methods marshall() and unmarshall() are implemented in the parent class
// and will populate the base properties of the Exception and representation accordingly
public class OrderValidationExceptionAdapter extends
AbstractExceptionXmlAdapter<OrderValidationExceptionDto,OrderValidationException> {

@Override
public OrderValidationException toException(OrderValidationException created,
  OrderValidationExceptionDto dto) throws Exception {

 created.setValidationCode(dto.validationCode);
 created.setValidationMessage(dto.validationMessage);
 return created;
}

@Override
public OrderValidationExceptionDto toRepresentation(OrderValidationExceptionDto created,
  OrderValidationException t) throws Exception {

 created.validationCode = t.getValidationCode();
 created.validationMessage = t.getValidationMessage();
 return created;
}

d. Control over which parts of an Exception are marshalled:
Provide server stack, cause and message to client. At the same time, ability to prevent certain exceptions from transmitting stack. One might or might not want to marshall server stack in certain cases. The same will be accomplished via an annotation that resides on an Exception thrown. Note that the same can be enhanced for further control. In the example shown below, throwing of the BarException will not result in the marshalling of the stack or cause:

/**
* Foo Exception when thrown should not include server stack and any cause
*/
@ExceptionMarshallingParams(includeStack=false, marshallCause=false)
public class BarException extends WebServiceException {
}

e. Simple mechanism to throw exceptions on the Client:
Clients are aware of the Exceptions that can be expected from a Service and can expect them to be thrown appropriately. For example:

public OrderDto createOrder(OrderDto order) throws OrderValidationException, OrderException {
JaxbRepresentation<OrderDto> orderRep = new JaxbRepresentation<OrderDto>(order);
Response response = client.post(baseUri + ORDERS_RESOURCE_URI, orderRep);
if (response.getStatus().isSuccess()) {
orderRep = new JaxbRepresentation<OrderDto>(response.getEntity(), OrderDto.class);
try {
 return orderRep.getObject();
}
catch (IOException e) {
throw new OrderException(e.getMessage(), e, response.getStatus());
}
}

// Not a Response we expected..throw one of the following exceptions expected instead
throw ExceptionUtil
.getWebServiceException(response.getEntity(), OrderValidationException.class,  OrderException.class);
}

public class ExceptionUtil {
...
public static WebServiceException getWebServiceException(Representation xmlRep,
  Class... expectedExceptions) {
 StringBuilder packages = new StringBuilder(20);
 packages.append(WebServiceException.class.getPackage().getName());

 for (Class clazz : expectedExceptions) {
   String ctxPath = getContextPath(clazz);
   packages.append(":").append(ctxPath);
 }

 JaxbRepresentation<WebServiceExceptionWrapperDto<WebServiceException>> wsRep
   = new JaxbRepresentation<WebServiceExceptionWrapperDto<WebServiceException>>(
     xmlRep, packages.toString());
 try {
   return wsRep.getObject().targetException;
 }
 catch (IOException e) {
   throw new RuntimeException("Error unmarshalling exception", e);
 }
}
...
}

Note that it would have been nice if we could specify the exceptions that a method could throw using generics. That feature however is restricted to a single exception type and thus accommodating checked exceptions are hard.

f. Resource Classes throw Exceptions on failed code states:
Resource classes will be able to throw any type of WebServiceException and have it marshalled to the client. The Resource classes must however ensure they populate the Exception with the appropriate Http Status Code. For example, an OrderResource on create might throw an OrderNotValidException or an OrderException when a POST operation is invoked on it. At the same time, a GET on the OrderResource might throw a OrderNotFoundException.

public class OrderResource extends ServerResource {
@Post("xml")
public OrderDto createOrder(OrderDto dto)  {
 try {
   dto = persistOrder(dto);
   setStatus(Status.SUCCESS_CREATED);
   return dto;
 } catch (OrderValidationException ove) {
   ove.setStatus(Status.CLIENT_ERROR_BAD_REQUEST);
   throw ove;
 }
 catch (OrderException e) {
   getResponse().setStatus(Status.SERVER_ERROR_INTERNAL);
   throw e;
 }
}
@Get("xml")
public OrderDto getOrder() {
 ....
 Order order = null;
 try {
   order = orderService.getOrder(orderId);
 }
 catch (OrderNotFoundException nfe) {
   nfe.setStatus(Status.CLIENT_ERROR_NOT_FOUND);
   throw nfe;
 }
 return (OrderDto) beanMapper.map(order, OrderDto.class);
}
....
}

public class HealthResource extends ServerResource {
private static int counter = 0;
@Get
public String isHealthy() {
  // Simply to prove a point....
  switch (counter) {
    case 0:
      counter++;
      // Throw a WebServiceException
      throw new WebServiceException("System is unhealthy", Status.CLIENT_ERROR_UNAUTHORIZED);

    case 1:
      counter++;
      // throw a RuntimeExeception..this will be transparently marshalled
      throw new RuntimeException("Some unexpected exception thrown as a Runtime");
  }

  getResponse().setStatus(Status.SUCCESS_OK);
  return "OK YEAH!";
 }
}

g. Marshalling of exceptions occurs via a Restlet Status Service:

A custom extension of the Restlet Status Service is responsible for marshalling all thrown Exceptions. The same is shown below:

public class JaxbExceptionStatusService extends StatusService {
....
public Status getStatus(Throwable throwable, Request request, Response response) {
  Throwable cause = throwable.getCause();
  WebServiceException exception;
  String contextPath = ExceptionUtil.BASE_EXCEPTION_PACKAGE;

  if (cause == null || !(cause instanceof WebServiceException)) {
    // If not a WebService Exception then throw a WebService Exception
    exception = new WebServiceException(..., Status.SERVER_ERROR_INTERNAL); 
   }  else {
     exception = (WebServiceException) cause;
     String clazzContext = ExceptionUtil.getContextPath(cause.getClass());

     if (!contextPath.equals(cause.getClass().getPackage().getName())
        && !clazzContext.equals(contextPath)) {
       contextPath = contextPath + ":" + clazzContext;
     }

     if (exception.getStatus() == 0) {
       exception.setStatus(Status.SERVER_ERROR_INTERNAL);
     }
   }

   // Create the Exception Wrapper
   WebServiceExceptionWrapperDto<WebServiceException> exceptionWrapper = new         WebServiceExceptionWrapperDto<WebServiceException>();
   exceptionWrapper.targetException = exception;

   JaxbRepresentation<WebServiceExceptionWrapperDto<?>> rep = new    JaxbRepresentation<WebServiceExceptionWrapperDto<?>>(
     exceptionWrapper);

   rep.setContextPath(contextPath);
   response.setEntity(rep);
   return Status.valueOf(exception.getStatus());
 }
}

I have included herewith a Maven example that will demonstrate the client, service and exception management in use. The maven module titled "exceptionnmodule" contains the exception handling code. The integration will demonstrate different exceptions that are thrown by the server and re-thrown by the client code. In particular, you can see the the following abilities of the framework:

a. Ability to throw a simple RuntimeException from the Service and see it marshalled/unmarshalled as WebServiceException
b. Ability to throw an instance of WebServiceException
c. Ability to throw an exception that is a subtype of WebServiceException but without any addition body. For example, the OrderException.
d. Ability to throw a sub type of the WebServiceException that has additional body elements. For example, OrderValidationException.

One can witness the server stack, cause, HTTP Status, specific exception types. Simply run a "mvn install" from the top level project.

Although the example uses Restlet 2.0, the framework has no dependencies on the same. There might be many areas for improvement in the concept and I would like to hear about them if a reader of this blog has ideas. Also, in this example, I have used code from a posting by Ian Robertson on Artima regarding Reflecting Generics.

Finally, I don't know how this BLOG will format. I am trying something new, I hope it woiks!

Download the Code HERE!

Restlet 2.0 example using Spring, Maven, Annotations and a Custom Converter

Its been some time since I posted regarding REST. Restlet 2.0 is on its way, I figured its time I updated the example I had created some time ago to a Restlet 2.0 milestone release. If you are continuing to read this BLOG, I would recommend a read of the Restlet 1.1 BLOG I mentioned prior to reading this one. So whats new with Restlet 2.0? A read of the Resource API refactoring page is recommended. One of the major features that I liked about JAX-RS over the core Restlet API while working with the former is that it is annotation driven versus class driven. Automatic content negotiation and the calling of the appropriate method in your Server Resource class for different Content-Types is really best left to the container. A method intended to serve content should not have if/else blocks for different content, something one would need to do in Restlet 1.1. Restlet 2.0 seems to take the same into consideration by providing the "best of both worlds" There were major changes that occurred on Restlet version 1.2 that the team at Restlet decided to change the release to a 2.0. Further details of the change can be obtained at http://blog.noelios.com/2009/05/27/restlet-2-0-m3-released/ Restlet 2.0 has a major change in the way Resources are viewed. In particular a split into a ClientResource and a ServerResource. Details of the same can be viewed on the Restlet Resource API refactoring WIKI page. In addition, there has been a refactoring of certain core classes to different packages. I also noticed that the Restlet API and Restlet engine are shipped as a single jar org.restlet.jar. I could not find the com.neolios packages any more. Finally, there is a jse and j2ee edition of Restlet available. So, eager to try out the Restlet 2.0 annotations, I reworked my example to use Restlet 2.0 while simplifying the Spring integration present. Compared to my previous example, I have reduced my Spring applicationContext.xml to be more thinner by using a more annotated approach.

<context:component-scan base-package="com.welflex">
</context:component-scan>

<!--  Spring Application. Note there are no mapping of resources here -->
<bean id="application" class="org.restlet.Application">
  <property name="name" value="orderApplication"></property>
  <property name="root" ref="root" />
   <!-- Added to handle Exceptions centrally -->
  <property name="statusService" ref="applicationStatusService"/>
</bean>

<bean id="root" name="router" class="com.welflex.order.rest.SpringBeanRouter"/>
<bean id="beanMapper"
 class="net.sf.dozer.util.mapping.DozerBeanMapper">
  <property name="mappingFiles">
    <list>
 <value>dozerBeanMapping.xml</value>
    </list>
</property>
</bean>

The OrdersResource has changed to:

@Component(value = "/orders")
@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrdersResource extends ServerResource {
  ......
  @Post
  public JaxbRepresentation<OrderDTO> createOrder(Representation rep) {
    ...
    return new JaxbRepresentation<OrderDTO>(MediaType.APPLICATION_XML, dto);      
  }

}

Note that the OrderResource class now extends ServerResource unlike in my previous example which extended the Resource class. Also note that the mapping of the resources to paths are defined in the OrderResource class itself unlike in the xml file in my previous example. The POST method, called createOrder() accepts and returns XML. The return type is a JaxbRepresentation. If we desired, we could have another method in the code that accepts JSON content and returns JSON content that is titled createJSONRepresentation(). Depending on the content type requested, the appropriate method would be called. This is much better than having an if/else block in an acceptRepresentation(Variant v) method. Take a look at the Products Resource, that returns a JSON representation when a GET method is invoked. Note that this method will only return JSON content.

@Component(value = "/products")
@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class ProductsResource extends ServerResource {
  ......
  @Get
  public JsonRepresentation getProducts() {
   .....
   return new JsonRepresentation(jsonString);
  }
}

Pretty nice, I would say. To spoil the party, one of the things I liked about the JAXRS implementations was that I could annotate a Resource with the media type it supported and if it were a JAXB object it would automatically marshall the XML, if it were JSON, it would automatically marshal the same as well. The same can be accomplished in Restlet by using a Converter Service that converts a java object to and from a media type. There are plans for adding JAXB converters amongst others in 2.0 M5. In short, by using a Converter Service, I would like the createOrder() method to have the following signature, i.e., method returns a JAXB Object, however the container knows how to marshal that into XML back to the caller:

@Post("xml":"xml")
public OrderDTO createOrder(OrderDTO inOrder) { return outOrderDTO; }

So what about the ClientResource class? How does that work? Well with Restlet 2.0, one can continue to use the Restlet Client class or use the ClientResource class. The ClientResource is not thread safe. An example of how the ClientResource can be used is shown below in the OrderClientImpl class:

public class OrderClientImpl implements OrderClient {
  private static final String ORDERS_RESOURCE_URI = "/orders";
  private final String baseUri;

  public OrderClientImpl(String baseUri) { this.baseUri = baseUri; }

  public OrderDTO createOrder(OrderDTO order) throws IOException {

    ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI);

    try {
      return ordersResource.post(order, OrderDTO.class);
    }
    catch (ResourceException e) {
      throw new IOException(e);
    }
  }

  public void updateOrder(OrderDTO order) throws OrderException {

    JaxbRepresentation<OrderDTO> orderCmd = new JaxbRepresentation<OrderDTO>(
        MediaType.APPLICATION_XML, order);

    ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI + "/"
      + String.valueOf(order.getOrderId()));

    try {
      ordersResource.put(orderCmd);
    }
    catch (ResourceException e) {
      throw new OrderException("Order Update Failed", e);
    }
  }

  public OrderDTO getOrder(Long orderId) throws OrderNotFoundException, IOException { ....}
  }

  public void deleteOrder(Long orderId) throws OrderException {....}
}

I quite like the use of ClientResource versus the Restlet Client as I feel its more RESTful, i.e., talking to a Resource using HTTP verbs. However, again we witness the JAXB representation creeping in as we saw with the service due to the lack of a converter. I decided to create my own Converter for JAXB to see if I can get it to work and simplify the code base. One of the things I wanted to add just for my pleasure is a way to propagate server stack traces to the client. This is NOT what one would typically include in a representation of a resource as one would not want clients getting details of the working of a server, however, as this is my playground, anything goes ;-). After some tinkering, the following is my Jaxb Converter, it assumes that all JAXB root level objects have the annotation @XmlRootElement:

public class CustomXmlConverter extends XmlConverter {

  ......
  @Override
  public List<Class<?>> getObjectClasses(Variant source) {
    List<Class<?>> result = super.getObjectClasses(source);

    result = addObjectClass(result, JaxbRepresentation.class);
    result = addObjectClass(result, Object.class);

    return result;
  }

  @Override
  public List<VariantInfo> getVariants(Class<?> source) {
    List<VariantInfo> result = super.getVariants(source);

    if (source.getAnnotation(XmlRootElement.class) != null

      || source.isAssignableFrom(JaxbRepresentation.class)) {

      result = addVariant(result, VARIANT_APPLICATION_ALL_XML);
      result = addVariant(result, VARIANT_APPLICATION_XML);
      result = addVariant(result, VARIANT_TEXT_XML);
    }

    return result;
  }

  @SuppressWarnings("unchecked")
  @Override
  public <T> T toObject(Representation source, Class<T> target, UniformResource resource) throws IOException {    
    .....    
    if (target.getAnnotation(XmlRootElement.class) != null) {      
      return new JaxbRepresentation<T>(source, target).getObject();
    }

    throw new IllegalStateException("Should not have got here");
  }

  @Override
  public Representation toRepresentation(Object source, Variant target, UniformResource resource) throws IOException {

    Representation result = null;
    
    if (source.getClass().getAnnotation(XmlRootElement.class) != null) {
      result = new JaxbRepresentation<Object>(source);
    }
    else {
      result = super.toRepresentation(source, target, resource);
    }
    return result;
  }
  .....
}

Great, so we have a Converter, how do we make Restlet use this converter for XML. In my common maven module (shared between client and web), I define a file in META-INF/services/org.restlet.engine.converter.ConverterHelper, and in that file, I have a single entry defining my custom converter. Now when the client and service code use the common library, they will use the Custom Converter I have defined for JAXB. Thus, to illustrate the change, my new resources for Orders and Order are shown below:

@Component(value = "/orders")
@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrdersResource extends ServerResource { 
 ..... 
 @Post("xml")

  public OrderDTO createOrder(OrderDTO dto) {
     ....
     return createdOrderDto;
  }
  ...
}

@Component(value = "/orders/{id}")
@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrderResource extends ServerResource {

  @Put(value = "xml")
  public void updateOrder(OrderDTO orderDTO) {
    try {
      String id = (String) getRequest().getAttributes().get("id");

      orderDTO.setOrderId(Long.parseLong(id));
      persistOrder(orderDTO);
    }
    catch (RuntimeException e) {
      getResponse().setStatus(Status.CLIENT_ERROR_BAD_REQUEST);
      throw new OrderException("Failed to update an order", e);
    }
  }
 .....
}

We are no longer returning Representations in the above code. Quite similar to JAXRS, wouldn't you say ;-)? On the Client end, the OrderClientImpl no longer deals with Representations but only OrderDTOs as shown below:

public class OrderClientImpl implements OrderClient {
  ....
  public OrderDTO createOrder(OrderDTO order) throws OrderException {
    ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI);

    try {
      return ordersResource.post(order, OrderDTO.class);
    }
    catch (ResourceException e) {
      throw new OrderException("Error Creating an Order", e);
    }
  }

  public void updateOrder(OrderDTO order) throws OrderException {
    ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI + "/"
      + String.valueOf(order.getOrderId()));
    try {
      ordersResource.put(order);
    }
    catch (ResourceException e) {
      throw new OrderException("Order Update Failed", e);
    }
  }
}

I like the simplicity of the above and cannot wait to see the release of Restlet 2.0 that has the additional converters built in. For those interested, I have a maven project example available for download. The project provided uses the custom JAXB converter. To execute the project, one would need to use JDK1.6.X and maven 2.0.9 or higher. To view an integration test in action execute, "mvn install" from the root level of the project. The result should be something like:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.welflex.order.IntegrationTest
log4j:WARN No appenders could be found for logger (com.welflex.client.ProductClientImpl).
log4j:WARN Please initialize the log4j system properly.
Aug 28, 2009 8:59:55 PM org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
20:59:55 DEBUG - com.welflex.order.rest.ProductsResource.getProducts(44) | Getting Products in JSON Format
Number of Available Products:3
Product Id:663123
Product Id:9912123
Storing the order...
Aug 28, 2009 8:59:55 PM org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
20:59:56 DEBUG - com.welflex.order.rest.OrdersResource.createOrder(45) | Call to post an order:..........
Updating the order...
Order successfully
Retrieving the order...
Deleting the order..
20:59:56 DEBUG - com.welflex.order.rest.OrderResource.getOrder(86) | Requested order with id:5499731937442305515
20:59:56 DEBUG - com.welflex.order.rest.ApplicationStatusService.getStatus(21) | In Order Status Service :class org.restlet.resource.ResourceException
Expected Order to not be found. Look at the server stack we got back
com.welflex.exception.OrderNotFoundException: Error obtaining Order
 at com.welflex.client.OrderClientImpl.getOrder(OrderClientImpl.java:60)
 at com.welflex.order.IntegrationTest.testLifeCycle(IntegrationTest.java:131)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
 at java.lang.reflect.Method.invoke(Method.java:597)
 at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)

More on Restlet 2.0 once its released. For now, maybe I should probably look toward cleaning up the JaxbConverter and submitting it to Restlet or better still head in the search of kindred spirits, if you know what I mean ;-)

Documentating Software, my 2c

Documentation....Aaargh..why in heavens name would anyone waste time on it? The following few words are a window into my experiences and thoughts on documentation in IT organizations.

Sadly, most organizations cannot or will not afford, the luxury of an additional resource who is dedicated to just the task of documentation, i,e., a technical writer. So as developers and architects where does that leave us ? I have heard statements such as:

• "Your code should be clear enough such that documenting the same is redundant"
• "Documentation is not much value unless its kept up to date, bad or expired documentation is worse than no documentation"
• "It is more important that the code has enough test coverage. The tests if good, document the code use"
• "We do not have the time, it is lowest on our priority list. We got bigger fish to fry."
  

Don't get me wrong, I am not in total disagreement regarding the above. Neither am I a subject matter expert on documentation. I am someone who often finds myself in the middle of this.

What I find myself questioning is:

• If the code is so self documenting or clear, and if you find people keep asking the same question over an over, is it really so self documenting ?
  
• If one created documentation, clearly it was considered important. If important, then it would need to updated with the same love when created.
• And please, there is always time...:-)
  

I have often entered an enterprise only to find that there is no Enterprise level documentation, stale or current. On the other hand, I have walked into an Enterprise only to be handed a thick set of pages to wade over to acclimatize myself. Neither worked for me, what I was seeking as an entrant was to see the enterprise from the 2000 feet level and then drill down as appropriate.

I am a very visual person, many a time a figure to me means more than a 1000 words. If there is a diagram that describes the stack the company uses, the applications, the services, even if partially wrong, immediately gives me a description of the architecture and/or project. That is a big win as far as acclimatising me to the environment.

When different folk ask the same question to me many times over, to me, that is another clearly indication of a lack of documentation. If a topic is constantly questioned, it deserves a place IMHO for documentation such that the next time the question arises, I point the person posing the question to the documentation instead of the repeated sermon that only I hold the copyrights to. DRY (Don't repeat yourself) does not apply to code alone...

Documentation is often created with great detail and appears as a big win, however if it starts becoming stale, then it loses its value and significance and becomes more of a "White Elephant". A conversation I had with someone in the past:

Someone: "Can you help me with this?"
Me: "Sure, did you check the documentation on the same?"
Someone: "Yeah but that is so outdated, so I don't bother looking at it any more"

The above conversation totally supports the argument that bad or documentation that are stale are definite deterrents to readers.

The following represent a few directions regarding documentation that I will try to adhere to:

1. Enterprise or Project Documentation or "The BIG Picture documentation": From an Enterprise architect perspective, providing a high level description/diagram of the entire enterprise and how it all fits or works together can be valuable. The documentation does not have to drill into the microscopic details but instead provide a more a high level view that helps an entrant understand how all the parts of an enterprise fit together. From the perspective of a team lead/lead developer, it would be beneficial if documentation were provided regarding the boundaries and dependencies of a particular project. There is no need for details of a database schema, however there is need to indicate that the application talks to a particular schema on a particular database . These documents might even be able to help detect problems in the architecture, as now, you are looking at "The Big Picture" From previous experience, I also feel that documenting "Big" decisions that affect a project can have great value. With time, we tend to forget the reasons for choosing a direction. A place to consult back as to "Why?" can prove valuable.
   
2. Stack Documentation: A diagram and/or description indicating the various technology stack in play. For example, "JBoss - App Server, Spring Framework, HP UX - 10.2 deployment OS, ..". This might be encompassed within the scope of point no.1. Again, stacks are rarely changed, and if they are, they probabaly require immediate attention.
3. Repeated Query Documentation: If any question posed to a developer or architect is being asked more than once, regardless of its longevity deserves due diligence in the documentation space. In lawyer terms, it would classify as "Asked and answered" :-). If you enjoy the interaction and enjoy repeating the same answer, well then more power to you , please disregard the documentation angle.
4. Redundant Documentation : Code that contains redundant or unnecessary documentation (i.e., code is self explanatory) serves no purpose. I violate this sometimes and need to stop. For example, getters and setter of a POJO (Plain Ole Java object) rarely requires documenting if the method name is clear. Institute naming standards that are enforced. Don't bother with class or schema diagrams, any developer worth his salt will be able to easily find free tools to reverse engineer classes or database schema's and be able to get the most current versions of the same.
   
5. Documenting Complex Code: Code that is unavoidably complex, deserve either strong documentation or better still refactoring.
6. Job security as it applies to documentation: Job security is not improved by keeping the details in your head. People are smart and folks that tend to do that will not last in a smart organization. Organizations lacking strong leaders, well get strong and smart leaders :-)
   
7. When to document and how much ? : Things that are in constant flux are hard to document, wait for stabilization. However, do follow up after stabilization occurs. Make sure your code works first, then comes the documentation. If you have good documentation but code that is broken, thats not a win. In the same vein, I would go further to say, Unit Tests and Integration Tests are more valuable than documentation. Place yourself in the position of a person who will utilize the documentation, make the content detailed enough for your intended audience. For example, in an enterprise level document, do not bother listing individual Web Services or Database Schemas, it is enough to state that Web Services communicate with databases and or other Web Services. The detail in the documentation should also be only that much that you can afford to keep current moving forward. Finally piggy backing on this point, broken windows or no windows from the top can send a bad message. What I mean is, "If an architect or lead themselves do not document, then it becomes a hard sell to a developer that they need to document"
   
8. Detail and Artistry of Documentation: Reiterating, "Make it right before making it pretty". A straw diagram caught on a white board via a cell phone camera during the design, if posted on a WIKI is often as effective as a colorful work of art. High ceremony documentation is almost too much and often unnecessary. Concise and to the point documentation works in most cases. People are smart, they often like a start. Its like being provided a map into Yellowstone regarding the sights to see. One does not require information on the concentration of sulfur at each site, just where the sites are located, we'll figure out the sulfur content at each site :-)
   
9. Tools for documentation: Use a tool that works for collaborative authoring such as a WIKI. Almost any software development project IMO deserves a WIKI or equivalent presence. Media WIKI and such are free to use, so a smart person would avail such tools. Use technology where applicable.
10. Collaborative Documentation: Documentation on IT projects should not belong to a single owner but be considered a collaborative effort. In other words, every member of the team should feel that they can add or correct toward the documentation. If a person comes asking a question and you give an answer, if you ask the person to document the same as you feels its valuable and is missing, you might be surprised as to how gladly they are willing to contribute. On the same note when you find documentation that is outdated and more importantly wrong, deprecate the documentation at the very top if you do not have the time to make it current.
    
11. How to get started, documentation: As an entrant into a new IT environment, one often needs to get set up and be ready to get productive. A road map documentation that provides a user of "What, Why, Where, Who" can be quite useful.
12. Selling documentation to the business: Getting business to account for time slots for code testing itself can be a daunting task in some organizations, getting a time slot in the project budget for documentation can be harder sale. However, as a lead or person making an estimate of a project, it is important to bake in the same. Documentation is money, however you need to sell the same and its value to the business. Lack of critical documentation is something that can come back to haunt you. "Protect your skin and those of your kinsmen" :-)
    

I am sure I have only touch the tip of what needs documentation. I cannot claim the title of a "Documentation nazi" and/or authority, neither am I claiming a "One set of rules fits all". People have probably written books on the same.....Easy to preach, hard to do, but hey, trying is not so hard, is it, and that is motive, trying ;-)...

Scaling in the JMS World...in particular Topics

Scalability and Availability in JMS. For the former, as load on the messaging system increases, one would want to be able to service the increased messages without performance degradation. For the latter, one should ensure the reduction if not elimination of single point of failures, i.e., ensuring system is available for use.

Most JMS Providers have some sort of mechanism for addressing the above. This BLOG is however only going to concentrate on Oracle WebLogic.

WebLogic has an interesting way in which JMS Destinations (Queues and Topics) are scaled via Distributed or Virtual Destinations. A Virtual Destination can be considered a sort of router/load balancer that distributes messages it receives over to actual or physical destinations. Each physical destination would reside in separate WebLogic instances on potentially separate hardware. See figure below where a virtual destination "test_queue" has physical members test_queue_1, test_queue_2 and test_queue_3 on separate WebLogic instances:



As an example of Virtual Queues in action, in the conceptual figure shown below, Producers send messages to the Virtual Queue which then directs the message to one of the physical destinations



P : Producers
VQ: Virtual Queue
PQ-X: Physical Queue, where X is the number of the Queue
C: Consumer
M: Message

In the case of a Virtual Topic, messages sent to the Virtual Topic are sprayed on to each and every physical member as shown in the figure below where VT and PT-X represent Virutal and Physical topics respectively:


Scaling Queues:
From the figure shown regarding Queues, it is pretty simple to follow how the system scales and that it will continue to service new messages and consume them should one of the servers in the messaging cluster fails. A message will be consumed at most by one of many consumers successfully. Adding Consumers to the end of the physical members allow the architecture to process messages rapidly, i.e., Competing Consumer Design Pattern.

Now, what if one of the servers having physical members went down, for example VQ-1 went down? From the perspective of availability, messages are still flowing and getting consumed from the other available member, i.e., VQ-2. What about the messages that were in the physical member whose server went down? One has multiple options. The first and most straight forward is to restart the server and re-connect the consumers to drain the queue. WebLogic allows for auto-service migration in the event of failure. This is another feature that can assist in recovering from failure. Queues readily tend to present themselves for scaling. One note on the same, if message ordering is put into play, all bets are off :-)..more on the same on a subsequent blog.

Scaling Topics:
Topics unlike Queues present an interesting challenge. In the case of a topic, a message is delivered to all listeners of the topic and each listener is typically different than its peer in how it handles the message. For example a message posted to an order topic, might have a consumer that processes the order while another one looks at inventory calculations:

Clearly we cannot have two consumers that process the same order and it is redundant to have duplicate consumers perform the same inventory calculations.

Messaging systems have the concept of a Durable Subscriber, i.e., a Message listener who registers with the topic such that they are delivered messages sent to the topic even when the listener is not connected. Message providers store the message on behalf of the durable subscriber and forward the same when the subscriber comes back on line.

In WebLogic one cannot connect to a durable subscriber using a distributed (virtual) destination JNDI. One has to explicitly register with a physical member. This is one gripe that I have with the WebLogic solution. Adding to the problem, should one connect to one physical member and if that member goes down, attempting to establish the durable subscription on other available members are treated like totally new subscriptions. In consolation, if the server goes down that contains durable subscriptions, then messages will be stored on other physical members of the virtual topic on behalf of the durable subscribers, and when the server is restored, messages get forwarded to the durable subscribers when they comes back on line as shown below:


Now, as each subscriber of the topic has a unique agenda from the other subscribers, the fact that it is the sole entity that services the message, the same represents a single point of failure and a proves a scalability inhibitor as it cannot concurrently process multiple messages.

So how can one scale a topic so that a message on a given type of subscriber can be rapidly processed? One direction is have the consumer of a Topic post the message onto a Queue which can then be processed by competing consumers. One way this can be achieved is to have a Message Driven Bean withing the container whose sole purpose is to transfer the message from the topic to a queue for those consumers that need to scale. An example of the same is shown below:


In the above shown example, as one consumer of the topic needs processing to rapidly occur of messages received, an MDB (registered as a durable subscriber of the topic) is used to transfer every message onto a Queue which is then consumed by competing consumers. As other consumers, DC-1 and DC-2 are not in need of scaling, they remain durable single subscribers.

The above solution works to scale the durable subscriber of a topic. However, in WebLogic when using distributed topics the durable MDB still represents a single point of failure.

A question to ask at this point is, "Do you really need a topic for the use case at hand?" Topics are great IMO when the subscribers tend to be dynamic in nature or do not need to be durable, an example of the former is a stock quote topic and an example of the latter is a event listener that requires notification of an event.

When the number of subscribers are a known or finite in amount, the use of a queue coupled with the MDB can easily achieve a scalable solution as shown below where the MDB transfers messages on the in-queue to subsequent out-queues where competing consumers can scale and consume the messages:

Transfer MDB:
It is pretty trivial to create the Transfer MDB. One would typically like to re-use the Transfer MDB Java Code with only configuration changes for different applications. The same can easily achieved via separate projects where target destination can be provided via @Resource. A simple transfer MDB is shown below:

---

@MessageDriven
@TransactionAttribute(TransactionAttributeType.REQUIRED)

public class MessageRouterMdb implements MessageListener {

  /**
   * Map this jms connection factory to the actual one in the weblogic-ejb-jar.xml deployment
   * descriptor of the ejb project. The Connection factory to be used will be the one that is
   * specific to this mdb.
   */
  @Resource(name = "jms/connectionFactory")

  private ConnectionFactory connectionFactory;

  /**
   * Injected in Message Driven Context
   */
  @Resource
  private MessageDrivenContext mdc;

  /**
   * Define a comma separated list of destinations JNDI names in the ejb-jar.xml via an environment
   * entry.
   */
  @Resource
  private String targetDestinations;

  // Initialized
  private List<Destination> targetJmsDestinations;

  private MessageProducer messageProducer;

  private Connection connection;

  private Session session;

  @PostConstruct
  public void init() {
    try {

      InitialContext ctx = new InitialContext();
      connection = connectionFactory.createConnection();

      session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

      messageProducer = session.createProducer(null);
      String[] targetDestArray = StringUtils.split(targetDestinations, ",");

      targetJmsDestinations = new ArrayList<Destination>();

      for (String targetDest : targetDestArray) {

        targetJmsDestinations.add((Destination) ctx.lookup(targetDest));

      }

      connection.start();
    }
    catch (Exception e) {     
      throw new RuntimeException("Error Initializing Message Router Mdb", e);

    }
  }

  public void onMessage(Message message) {

    try {
      // Loop through target destinations and send message to each destination
      for (Destination dest : targetJmsDestinations) {

        messageProducer.send(dest, message);
      }
    }

    catch (Exception e) {
      mdc.setRollbackOnly();

    }
  }
  
  @PreDestroy
  public void cleanup() {

    // Close open jms resources here...
  }
}

---

The CODE for the above MDB and an example Maven Project that uses WebLogic is available for download HERE. Note that the code is not specific to WebLogic and can really be used in any container, for example JBoss.

Conclusion:

Scaling topics via an MDB and target queues just felt like an obvious pattern to me. WebLogic, like many JMS Providers has the concept of a Message Bridge. Initial investigation led me to believe that a message bridge more serves the purpose of transferring a message from one provider to to another rather than serving to scale a Topic and distributed messages to multiple destinations.

There may be cases as mentioned where the number of subscribers are not known. In such a case, a topic might work and it becomes the responsibility of the subscribers to scale on their end, for example multi-threading upon receiving the message.

What really would be nice if the JMS specification had the concept of "Durable Competing Consumers" where a Durable subscriber could register multiple instances of itself such that a message to a topic would be delivered to at most it or one of its clones :-). I believe there are some Messaging providers who permit such a pattern via customization that extend the JMS specifications.

Before bidding adieu, this has been one of my most figurative blogs, all Ye artists, look at the diagrams with a forgiving eye, "I am a programmer not an artist!" :-)

Ending the drought...

Its been a month since I posted anything on my BLOG, cannot let May end with a drought. It has been rather a trying month for me, the reasons I am saving for a later melodramatic post ;-)...

Right now I am very happy. At the company I currently work at, we released out a JMS architecture with minimal hiccups. It was very gratifying to witness the entire team involved work and mesh so well in order to make the architecture work. Everyone from Architects, Developers, System Administrators, Testers worked so well together. I must state that my level of confidence in the team and players involved was so much that I did not even have to be there during the go live in person. Put it in a word, my current organization is so much about "WE" and not "ME". That is a motto that works IMHO. Guess my current organization is very talented in its selection process of the people they want on board.

I truly enjoyed working with all the folks on the various facets involving the deployment of the architecture, frameworks, deployment, projects etc. We suffered a few hiccups after the deploy as well but the messaging solution itself was 100% solid. In fact in my own retarded way, I appreciated the fact that the failures occurred as they served to demonstrate that the failure path in fact worked ;-)

Personally, I have experienced considerable growth in the areas of JMS Messaging. However, every once so often I again am humbled by what I don't know or did not anticipate.

The project as a whole was like our baby that finally saw the light..and danced on...I hope it keeps dancing :-))))))))))) Even if does not, I believe we have a team strong enough to put it on crutches to make it so...

I can't wait for further adoption of the architecture...so thrilled, had a blast...

More BLOG's coming up soon, the little GREY cells were on vacation (not dead, just vacationing), you should see something from them soon....