Search This Blog

Friday, March 5, 2010

RESTful Representation with Google Protocol Buffers and Jersey

When working with a RESTful system, one has the option of consuming different types of representations of the same resource. That is one of the beauties of a RESTful system IMO.

In many cases, especially from the SOAP Based services stack, XML has been the representation type of choice for information interchange.  XML was never really intended to be  high performance representation and when working with Java and XML, one often sees performance penalties experienced with mashalling/un-marshalling the payload and size of transfer that are rather undesirable.

There have been other formats that have gained popularity such as JSON which work really well with Java Script and Ajax. For those desiring a comparison between selecting JSON or XML, its only a google search away.

That said, both the above mentioned formats are not binary formats. There are many binary format options to available for users. One of these is google protocol buffers which the focus of this blog.

Why am I blogging about Protcol buffers now? Well, recently I saw a presentation on how Orbitz shifted from using JINI to RESTful services with Google Protocol Buffers as their representation type to success.  Protocol Buffers allowed them to meet their performance needs, versioning needs and language/platform compatibility needs really well.  Since then, I had to try the same :-)

Ted Neward has a very nice article about XML and his take on Protocol buffers where he digs into binary formats, pro-cons etc. I recommend a read of his posting.

Regarding performance metrics of using Java Externalizable, google protocol buffers, XML, JSON, Thrift, Avro etc look at the thrift-protobuf comparison google code page for more details. In addition, Wondering Around by Eishay Smith is a great read on the comparisons.

So all I have here is working example of using Protocol Buffers with Jersey and Spring. Like it my previous examples, I am using the same orders/product model.

I start with definitions of the contract. One defines the messages exchanged in  .proto files. Wow, YAIDL (Yet Another Interface Definition Language) here. True, but contracts need to be exchanged and there has to be a way to do the same especially when dealing with platform/language neutrality and B2B exchanges. I must say that I found the IDL rather easy to understand and use with my limited understanding so far.  One of the beauties of protocol buffers is their considerations for backward compatibility of contracts.  There is an eclipse plugin is available for editing .proto files as well at http://code.google.com/p/protoclipse.  With that said, I have two .proto files, one for Order definition and a second for the Products as shown below:
Orders.proto
package orders;

option java_package = "com.welflex.orders.proto.dto";
option java_outer_classname= "OrderProtos";
option optimize_for = SPEED;

message LineItem {
 optional int64 id =1;
 required int64 itemId = 2;
 required string itemName = 3;
 required int32 quantity = 4;
}

message Order {
   optional int64 id = 1;
   repeated LineItem lineItems = 2;   
}
Products.proto
package products;

option java_package = "com.welflex.products.proto.dto";
option java_outer_classname= "ProductProtos";
option optimize_for = SPEED;

message Product {
   required int32 id = 1;
   required string name = 2;
   required string description = 3;   
}

message ProductList {
   repeated Product productDto = 1;   
}
I have defined the above files in the common module at src/main/protobuf and when the maven compiler runs, it will generate equivalent Java Code based of the proto files which can then be used to create messages from consuming java code. The plugin is basically executing the "protoc" compiler to do the same. One can choose to create equivalent C ++ code if required or Python etc etc. However, the same is beyond the scope of this BLOG. With the above definition, OrderProtos.java is generated during the maven build at target/generated-sources/protoc/com/welflex/orders/proto/dto/OrderProtos.java. In the file, you will find the individual message which extend com.google.protos.GeneratedMessage. These objects are shared by the client and service code.

The generated java code uses the Builder Pattern with method chaining to make it really easy to set the necessary properties and build the protocol buffer message. For example, the Order Message can be built as shown below:
OrderProtos.Order order = OrderProtos.Order.newBuilder().setId(12313L)
     .addLineItem(OrderProtos.LineItem.newBuilder().setId(8913L)
                    .setItemId(123).setItemName("Foo Bar").setQuantity(21).build()).build();

For getting the Web Service to work with Jersey, based of another blog I mention later, I defined a custom Provider for marshalling/un-marshalling the Message. What amazes me is the ease of providing custom providers in JAX-RS. Big fan here :-). Message Body Reader and Message Body Writer classes are shown below that assist with the marshalling:
@Provider
@Component
@Consumes(AlternateMediaType.APPLICATION_XPROTOBUF)
public class ProtobufMessageReader implements MessageBodyReader<Message> {
  public boolean isReadable(Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return Message.class.isAssignableFrom(type);
  }

  public Message readFrom(Class<Message> type, Type genericType, Annotation[] annotations,
    MediaType mediaType, MultivaluedMap<String, String> httpHeaders, InputStream entityStream) throws IOException,
    WebApplicationException {
    try {
      Method newBuilder = type.getMethod("newBuilder");
      GeneratedMessage.Builder<?> builder = (GeneratedMessage.Builder<?>) newBuilder.invoke(type);
      return builder.mergeFrom(entityStream).build();
    }
    catch (Exception e) {
      throw new WebApplicationException(e);
    }
  }
}
@Provider
@Component
@Produces(AlternateMediaType.APPLICATION_XPROTOBUF)
public class ProtobufMessageWriter implements MessageBodyWriter<Message> {
  public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return Message.class.isAssignableFrom(type);
  }

  public long getSize(Message m, Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType) {
    return m.getSerializedSize();
  }

  public void writeTo(Message m, Class<?> type, Type genericType, Annotation[] annotations,
    MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException,
    WebApplicationException {
    entityStream.write(m.toByteArray());
  }
}
With the above complete, the rest of the code is pretty similar to what I have done in previous BLOGS and therefore am not mentioning the same again. We now have the necessary artifacts to exchange the Protocol Buffer messages over HTTP.

The steps to get this example working are as follows:
1. Download the code from HERE
2. Install the maven plugin and thus Protocol Buffers:
>svn co http://protobuf.googlecode.com/svn/branches/maven-plugin/tools/maven-plugin
>cd maven-plugin
>wget -O pom.xml 'http://protobuf.googlecode.com/issues/attachment?aid=8860476605163151855&name=pom.xml'
>mvn install

If the above does not work, you might want to try looking at the Stack Overflow Posting where I got this from.
3. Execute "mvn install" from the root level of the project to see an integration test that will run the life cycle from client to server using Protocol buffers and not XML or JSON :-)

This example is highly inspired by the  fantastic maven example by Sam Pullara at Java Rants on integrating Jersey, Protocol Buffers and Maven. My example is of course tailored to any readers visiting this site ;-) The Products resource returns formats of JSON, XML and Protocol buffers for those interested in trying the same out.

Clearly, one has multiple choices regarding their representation types, the beauty of REST lies in the fact that one does not have to choose one over the other but allow for coexistence.  There are many factors that one would consider when choosing the format of their representation, some of the things I can think of in no particular order of importance,
  • Performance - Marshalling/Un-Marshalling and transport footprint
  • Integration with different platforms/different languages
  • Testability and visibility - binary formats hide this
  • Versioning of services to ensure backward compatibility
  • B2B integration
I wonder whether there will be an effort to support annotations of Java Objects so that they may be transformed into .proto files, ala, JAXB Annotations in the future. So where next, simple, need to look at Avro and Thrift ;-)

11 comments:

Anonymous said...

Hi Sanjay,
Thanks for the article.

However, I keep getting error like this:

SEVERE: A message body reader for Java type, class com.example.tutorial.ProfileRequestProto$ProfileRequest, and MIME media type, application/x-protobuf, was not found

I posted the full detail in stackoverflow, but it has got no reply. (or maybe yet)

http://stackoverflow.com/questions/3190329/protocol-buffers-mime-problem

Do you know what is causing this?
Any help will be appreciate it.

Thanks

Sanjay Acharya said...

It appears that your MessageBodyReader and/or MessageBodyWriter have not been properly registered with your JAX-RS provider. Download the example and compare your code with the provided source. A couple of things, make sure you have annotated your writer/reader class with the right annotation and make sure the same is being picked up by the implementation

Anonymous said...

Apparently I need to update jersey jar file. It solves the problem.
Thanks

wz said...

HI,
How can I pass a meaningful error message back to client? I implemented the ExceptionMapper with detailed error message. using the following code:
return Response.status(Status.INTERNAL_SERVER_ERROR).entity(e.getMessage()).build();
But how can the writer to pass the message back to client?

Sanjay Acharya said...

Did you set the Media Type on the response you are building?

wz said...

I think the correct question is how do server pass back a meaningful error message? Client can get the status code to be 500, where to set the error message in this case? Client expects a protobuf object which can not be returned in this case. What is the best practice? One way I can think of is to put the message in the http header. Or so, another question. What to return for POST? only status code? I am using Jersey client, it has the following implementation in the WebResrouce,

private T handle(Class c, ClientRequest ro) throws UniformInterfaceException {
setProperties(ro);
ClientResponse r = getHeadHandler().handle(ro);

if (c == ClientResponse.class) return c.cast(r);

if (r.getStatus() < 300) return r.getEntity(c);

throw new UniformInterfaceException(r,
ro.getPropertyAsFeature(ClientConfig.PROPERTY_BUFFER_RESPONSE_ENTITY_ON_EXCEPTION, true));
}

Looks like, it tries to get the entity always.

Thanks very much for your advice.

Sanjay Acharya said...

Lets see if I understand. In your exception mapper, if you set the entity and define the media type to be application/x-protobuf the Protobuf message body writer should marshall the entity to the client.
On the client, make sure you register the message body reader. After which, you can do somethign like the following:
ClientResponse r= client.post(ClientResponse.class);
if (r.getStatus() == Success) {
r.getEntity(FooObject.class);
} else {
..whatever
}
Above code is not compiler tested so you will need to make it work with actual methods.

wz said...

For the success case, I can get the entity back no problem. My problem is the exception case (invalid user input). I will pass the status code back (500 or something I can define). But how can I pass the detailed message back to client? If it's the JSON response, it's easy. But for protobuf, need to have entity class in order to unmarshall the response. Just wondering what is the best practice for exception case?
Also for POST, what's normally passed back to client? Status with 200 is good enough?

Sanjay Acharya said...

If you have the entity to unmarshall a response in the event of a success, why wouldn't you also have the error entity to unmarshall in the event of an error?

Regarding Error Codes and response types, http://shop.oreilly.com/product/9780596529260.do
is a great reference.

wz said...

Makes sense. I will add the response code + error message as part of the proto message.

total12 said...

RESTful system, one has the option of consuming different types of representations of the same resource. Click here