RESTful Representation with Google Protocol Buffers and Jersey

When working with a RESTful system, one has the option of consuming different types of representations of the same resource. That is one of the beauties of a RESTful system IMO.

In many cases, especially from the SOAP Based services stack, XML has been the representation type of choice for information interchange.  XML was never really intended to be  high performance representation and when working with Java and XML, one often sees performance penalties experienced with mashalling/un-marshalling the payload and size of transfer that are rather undesirable.

There have been other formats that have gained popularity such as JSON which work really well with Java Script and Ajax. For those desiring a comparison between selecting JSON or XML, its only a google search away.

That said, both the above mentioned formats are not binary formats. There are many binary format options to available for users. One of these is google protocol buffers which the focus of this blog.

Why am I blogging about Protcol buffers now? Well, recently I saw a presentation on how Orbitz shifted from using JINI to RESTful services with Google Protocol Buffers as their representation type to success.  Protocol Buffers allowed them to meet their performance needs, versioning needs and language/platform compatibility needs really well.  Since then, I had to try the same :-)

Ted Neward has a very nice article about XML and his take on Protocol buffers where he digs into binary formats, pro-cons etc. I recommend a read of his posting.

Regarding performance metrics of using Java Externalizable, google protocol buffers, XML, JSON, Thrift, Avro etc look at the thrift-protobuf comparison google code page for more details. In addition, Wondering Around by Eishay Smith is a great read on the comparisons.

So all I have here is working example of using Protocol Buffers with Jersey and Spring. Like it my previous examples, I am using the same orders/product model.

I start with definitions of the contract. One defines the messages exchanged in  .proto files. Wow, YAIDL (Yet Another Interface Definition Language) here. True, but contracts need to be exchanged and there has to be a way to do the same especially when dealing with platform/language neutrality and B2B exchanges. I must say that I found the IDL rather easy to understand and use with my limited understanding so far.  One of the beauties of protocol buffers is their considerations for backward compatibility of contracts.  There is an eclipse plugin is available for editing .proto files as well at http://code.google.com/p/protoclipse.  With that said, I have two .proto files, one for Order definition and a second for the Products as shown below:
Orders.proto

package orders;

option java_package = "com.welflex.orders.proto.dto";
option java_outer_classname= "OrderProtos";
option optimize_for = SPEED;

message LineItem {
 optional int64 id =1;
 required int64 itemId = 2;
 required string itemName = 3;
 required int32 quantity = 4;
}

message Order {
   optional int64 id = 1;
   repeated LineItem lineItems = 2;   
}

Products.proto

package products;

option java_package = "com.welflex.products.proto.dto";
option java_outer_classname= "ProductProtos";
option optimize_for = SPEED;

message Product {
   required int32 id = 1;
   required string name = 2;
   required string description = 3;   
}

message ProductList {
   repeated Product productDto = 1;   
}

I have defined the above files in the common module at src/main/protobuf and when the maven compiler runs, it will generate equivalent Java Code based of the proto files which can then be used to create messages from consuming java code. The plugin is basically executing the "protoc" compiler to do the same. One can choose to create equivalent C ++ code if required or Python etc etc. However, the same is beyond the scope of this BLOG. With the above definition, OrderProtos.java is generated during the maven build at target/generated-sources/protoc/com/welflex/orders/proto/dto/OrderProtos.java. In the file, you will find the individual message which extend com.google.protos.GeneratedMessage. These objects are shared by the client and service code.

The generated java code uses the Builder Pattern with method chaining to make it really easy to set the necessary properties and build the protocol buffer message. For example, the Order Message can be built as shown below:

OrderProtos.Order order = OrderProtos.Order.newBuilder().setId(12313L)
     .addLineItem(OrderProtos.LineItem.newBuilder().setId(8913L)
                    .setItemId(123).setItemName("Foo Bar").setQuantity(21).build()).build();

For getting the Web Service to work with Jersey, based of another blog I mention later, I defined a custom Provider for marshalling/un-marshalling the Message. What amazes me is the ease of providing custom providers in JAX-RS. Big fan here :-). Message Body Reader and Message Body Writer classes are shown below that assist with the marshalling:

@Provider
@Component
@Consumes(AlternateMediaType.APPLICATION_XPROTOBUF)
public class ProtobufMessageReader implements MessageBodyReader<Message> {
  public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return Message.class.isAssignableFrom(type);
  }

  public Message readFrom(Class<Message> type, Type genericType, Annotation[] annotations,
    MediaType mediaType, MultivaluedMap<String, String> httpHeaders, InputStream entityStream) throws IOException,
    WebApplicationException {
    try {
      Method newBuilder = type.getMethod("newBuilder");
      GeneratedMessage.Builder<?> builder = (GeneratedMessage.Builder<?>) newBuilder.invoke(type);
      return builder.mergeFrom(entityStream).build();
    }
    catch (Exception e) {
      throw new WebApplicationException(e);
    }
  }
}
@Provider
@Component
@Produces(AlternateMediaType.APPLICATION_XPROTOBUF)
public class ProtobufMessageWriter implements MessageBodyWriter<Message> {
  public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return Message.class.isAssignableFrom(type);
  }

  public long getSize(Message m, Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return m.getSerializedSize();
  }

  public void writeTo(Message m, Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException,
    WebApplicationException {
    entityStream.write(m.toByteArray());
  }
}

With the above complete, the rest of the code is pretty similar to what I have done in previous BLOGS and therefore am not mentioning the same again. We now have the necessary artifacts to exchange the Protocol Buffer messages over HTTP.

The steps to get this example working are as follows:
1. Download the code from HERE
2. Install the maven plugin and thus Protocol Buffers:
>svn co http://protobuf.googlecode.com/svn/branches/maven-plugin/tools/maven-plugin
>cd maven-plugin
>wget -O pom.xml 'http://protobuf.googlecode.com/issues/attachment?aid=8860476605163151855&name=pom.xml'
>mvn install

If the above does not work, you might want to try looking at the Stack Overflow Posting where I got this from.
3. Execute "mvn install" from the root level of the project to see an integration test that will run the life cycle from client to server using Protocol buffers and not XML or JSON :-)

This example is highly inspired by the  fantastic maven example by Sam Pullara at Java Rants on integrating Jersey, Protocol Buffers and Maven. My example is of course tailored to any readers visiting this site ;-) The Products resource returns formats of JSON, XML and Protocol buffers for those interested in trying the same out.

Clearly, one has multiple choices regarding their representation types, the beauty of REST lies in the fact that one does not have to choose one over the other but allow for coexistence.  There are many factors that one would consider when choosing the format of their representation, some of the things I can think of in no particular order of importance,

• Performance - Marshalling/Un-Marshalling and transport footprint
• Integration with different platforms/different languages
• Testability and visibility - binary formats hide this
• Versioning of services to ensure backward compatibility
• B2B integration

I wonder whether there will be an effort to support annotations of Java Objects so that they may be transformed into .proto files, ala, JAXB Annotations in the future. So where next, simple, need to look at Avro and Thrift ;-)