Search This Blog

Loading...

Tuesday, December 8, 2009

Google Chrome for Linux Rocks

Google Chrome for Linux Beta is available. I am currently trying the same on my 64 bit Ubuntu. It is sweet to say the least, pages render so much faster for me when compared to my Firefox installation. I do however keep seeing a message saying the 64 bit flash plugin crashed, although I am able to view You tube without a problem.

The Chrome Browser was able to import my setting from Firefox with the only clause that firefox not be running at the time of the import.

Have high hopes for this browser now that they have support for plugins. I just need a decent delicious plugin right now. A Google Chrome fan, next I want to try out Virtual Box VM and see if I can play with Google Chrome OS. I am only trying Virtual Box due to certain installation pains that I am having with VMWare that I need to spend some time fixing :-)..

Tuesday, November 17, 2009

Quite Interesting, need to get my own quote someday

The 25 Best Quotes from Tech History. Think of my colleagues at work will enjoy the one on JINI. My favourite quote of course lies with HAL.

http://www.computerworld.com/s/article/9141008/25_best_quotes_from_tech_history

Saturday, October 31, 2009

Ubuntu Karmic and Windows 7

Did two upgrades this weekend. Upgraded a laptop from Vista to Windows 7 and a desktop from Ubuntu Jaunty to Karmic, both 64 bit. Pleased with both upgrades so far. Vista was one of MS biggest lows as far as their OS version goes, Windows 7 seems more peppier in general so far.

I have been very pleased with Ubuntu ever since I installed the same. Have previously been a user of Mandriva, Fedora and Suse. Found mandriva comprehensive in its software offering but not very stable. I feel that Suse may be great to compete with Redhat on the server market but is no where as user friendly as Ubuntu. What I like about Ubuntu is that its pretty solid, convenient and just so easy to use. It detects the need for propriety drivers and installs in a breeze. Have dual monitor support without any xorg.conf tweaking. Printer, web cam all worked out of the box. Have it now running Skype as well without a problem.

After installing Ubuntu, I followed the following BLOG on a TODO list that really helped.
Anyway, got rid of the standard 1948 created Gnome menu in favour of AWN and now my desktop rocks.

A note for those installing eclipse 64 bit. One will find that clicking on some buttons does not work and one has use the keyboard actions to activate the buttons. The way around this problem is to set the following "export GDK_NATIVE_WINDOWS=true" prior to launching eclipse.

On the Windows 7 front, I prefer using Chrome over firefox or IE as its so much more faster on many sites that I frequent. The upgrade from Dell from Vista to Win 7 was pretty simple. Cannot watch youtube on 64 bit IE yet due to 64 flash plugin not available (no so on my Ubuntu though :-)) Don't get me wrong, I don't have anything against Win 7 (except $$$) but when compared with what FREE brings, I find it difficult to justify the extra $$$. I however do need to a Win machine due to the fact that there is no software for the iphone of any value on Linux and that there are sites that I go to such as CWTV to watch videos requiring Win or Mac. Another seller for the Win 7 is my few PC games that I have. BTW, I did not buy Win 7, its a free upgrade option.

The Mrs also is happy with Ubuntu...and as long as she is happy, I am as well :-)

Friday, October 16, 2009

Exceptions in RESTful HTTP with Restlet and JAXB

REST, or rather, RESTful HTTP and how exceptions can be handled therein is what this posting is about.

For those of us who have worked with SOAP and faults, we are familiar with how exceptions are handled. SOAP Stacks upon encountering an exception will throw a SOAP fault which is then marshalled into an exception on the consumer end of the service. SOAP faults in a way provide a protocol over standard HTTP.

When working with REST we have the concept of Resources and their Representations. An exception in a RESTful architecture is really a tuple of an HTTP Status Code and a Representation that describes the same.

The HTTP Status code in themselves are quite indicative, delving a bit into the same:
  • 1XX Series - Informational - Request received and continuing processing
  • 2XX Success - Action received, understood and accepted
  • 3XX Redirection - Further action required to complete the request
  • 4XX Client Error - Request is badly formatter or the server cannot process the request
  • 5XX Server Error - Server failure on a a potentially valid request
When working with architectures that have Java on both ends of a Service, shying away from Exception handling is IMHO a self imposed handicap. In a RESTful system, one would typically have an expected representation of a resource and in the event of exceptions, have them represented as alternate results along with the appropriate error codes. The JAX-RS specification has the concept of a WebServiceException.

What I am trying to do in this BLOG is a demonstrate how exception handling can be accomplished with a framework like Restlet. I am not demonstrating a specification by any means or a wrapper protocol. For the sake of this demonstration, I will be using Restlet and JAXB. In particular, the example will rely on the XmlAdapters of JAXB extensively coupled with the Java Reflection API and Annotations. The XmlAdapters serve to translate to and from Exceptions to JAXB Representions

The scope of this example is as follows:

a. Exceptions supported are of a particular type and its subclasses only:
Exceptions are restricted to WebServiceException and its sub classes. In addition, the WebServiceException is a RuntimeException. No checked exceptions. Any exceptions thrown by a Restlet Resource that are not an instance of WebServiceException will be converted to the same prior to being marshalled. Any WebServiceException marshalled, must have a JAXB Object associated with it.


@XmlJavaTypeAdapter(WebServiceExceptionAdapter.class)
public class WebServiceException extends RuntimeException {
// Http Status code
private int status;
}


// This class extends WebServiceException but has the same body as
// WebServiceException. For this reason, there is no special XmlAdapter required.
public class OrderException extends WebServiceException {
..
}

b. XML as the media type for Transferring Exceptions:
XML is the sole media type supported for exceptions in this example. Note that it is quite trivial to change the same to accommodate additional media types like JSON etc if required. Every Exception's representation is wrapped around with a WebServiceExceptionWrapperDto. The same is to allow for sub typing. Note that the targetException will be marshalled by an appropriate XmlAdapter at Runtime.

@XmlType(name = "ExceptionWrapper", propOrder = { "targetException" })
@XmlRootElement
public class WebServiceExceptionWrapperDto<E extends WebServiceException> {
@XmlElement(name = "targetException", required = true)
public E targetException;
}

c. Rich Exceptions:
Exceptions that can be richer, i.e., more than just message, cause and stack. For example validation messages, codes, inner objects etc. A simple example of the same is shown below where the OrderValidationException contains a description and a custom validation error code. The same could easily have been a complex object if required:


@XmlJavaTypeAdapter(OrderValidationExceptionAdapter.class)
public class OrderValidationException extends WebServiceException {
// A code representing the error
private int validationCode;

// A description indicating the cause of the validation error
private String validationMessage;
...
}


// Note that the OrderValidationException Adapter is behaving like a Template method pattern
// Base marshalling to an exception is handled by the parent class with the child only providing
// specific additions. The methods marshall() and unmarshall() are implemented in the parent class
// and will populate the base properties of the Exception and representation accordingly
public class OrderValidationExceptionAdapter extends
AbstractExceptionXmlAdapter<OrderValidationExceptionDto,OrderValidationException> {

@Override
public OrderValidationException toException(OrderValidationException created,
OrderValidationExceptionDto dto) throws Exception {

created.setValidationCode(dto.validationCode);
created.setValidationMessage(dto.validationMessage);
return created;
}

@Override
public OrderValidationExceptionDto toRepresentation(OrderValidationExceptionDto created,
OrderValidationException t) throws Exception {

created.validationCode = t.getValidationCode();
created.validationMessage = t.getValidationMessage();
return created;
}

d. Control over which parts of an Exception are marshalled:
Provide server stack, cause and message to client. At the same time, ability to prevent certain exceptions from transmitting stack. One might or might not want to marshall server stack in certain cases. The same will be accomplished via an annotation that resides on an Exception thrown. Note that the same can be enhanced for further control. In the example shown below, throwing of the BarException will not result in the marshalling of the stack or cause:

/**
* Foo Exception when thrown should not include server stack and any cause
*/
@ExceptionMarshallingParams(includeStack=false, marshallCause=false)
public class BarException extends WebServiceException {
}


e. Simple mechanism to throw exceptions on the Client:
Clients are aware of the Exceptions that can be expected from a Service and can expect them to be thrown appropriately. For example:

public OrderDto createOrder(OrderDto order) throws OrderValidationException, OrderException {
JaxbRepresentation<OrderDto> orderRep = new JaxbRepresentation<OrderDto>(order);
Response response = client.post(baseUri + ORDERS_RESOURCE_URI, orderRep);
if (response.getStatus().isSuccess()) {
orderRep = new JaxbRepresentation<OrderDto>(response.getEntity(), OrderDto.class);
try {
return orderRep.getObject();
}
catch (IOException e) {
throw new OrderException(e.getMessage(), e, response.getStatus());
}
}

// Not a Response we expected..throw one of the following exceptions expected instead
throw ExceptionUtil
.getWebServiceException(response.getEntity(), OrderValidationException.class, OrderException.class);
}


public class ExceptionUtil {
...
public static WebServiceException getWebServiceException(Representation xmlRep,
Class... expectedExceptions) {
StringBuilder packages = new StringBuilder(20);
packages.append(WebServiceException.class.getPackage().getName());

for (Class clazz : expectedExceptions) {
String ctxPath = getContextPath(clazz);
packages.append(":").append(ctxPath);
}

JaxbRepresentation<WebServiceExceptionWrapperDto<WebServiceException>> wsRep
= new JaxbRepresentation<WebServiceExceptionWrapperDto<WebServiceException>>(
xmlRep, packages.toString());
try {
return wsRep.getObject().targetException;
}
catch (IOException e) {
throw new RuntimeException("Error unmarshalling exception", e);
}
}
...
}

Note that it would have been nice if we could specify the exceptions that a method could throw using generics. That feature however is restricted to a single exception type and thus accommodating checked exceptions are hard.

f. Resource Classes throw Exceptions on failed code states:
Resource classes will be able to throw any type of WebServiceException and have it marshalled to the client. The Resource classes must however ensure they populate the Exception with the appropriate Http Status Code. For example, an OrderResource on create might throw an OrderNotValidException or an OrderException when a POST operation is invoked on it. At the same time, a GET on the OrderResource might throw a OrderNotFoundException.

public class OrderResource extends ServerResource {
@Post("xml")
public OrderDto createOrder(OrderDto dto) {
try {
dto = persistOrder(dto);
setStatus(Status.SUCCESS_CREATED);
return dto;
} catch (OrderValidationException ove) {
ove.setStatus(Status.CLIENT_ERROR_BAD_REQUEST);
throw ove;
}
catch (OrderException e) {
getResponse().setStatus(Status.SERVER_ERROR_INTERNAL);
throw e;
}
}
@Get("xml")
public OrderDto getOrder() {
....
Order order = null;
try {
order = orderService.getOrder(orderId);
}
catch (OrderNotFoundException nfe) {
nfe.setStatus(Status.CLIENT_ERROR_NOT_FOUND);
throw nfe;
}
return (OrderDto) beanMapper.map(order, OrderDto.class);
}
....
}


public class HealthResource extends ServerResource {
private static int counter = 0;
@Get
public String isHealthy() {
// Simply to prove a point....
switch (counter) {
case 0:
counter++;
// Throw a WebServiceException
throw new WebServiceException("System is unhealthy", Status.CLIENT_ERROR_UNAUTHORIZED);

case 1:
counter++;
// throw a RuntimeExeception..this will be transparently marshalled
throw new RuntimeException("Some unexpected exception thrown as a Runtime");
}

getResponse().setStatus(Status.SUCCESS_OK);
return "OK YEAH!";
}
}

g. Marshalling of exceptions occurs via a Restlet Status Service:

A custom extension of the Restlet Status Service is responsible for marshalling all thrown Exceptions. The same is shown below:

public class JaxbExceptionStatusService extends StatusService {
....
public Status getStatus(Throwable throwable, Request request, Response response) {
Throwable cause = throwable.getCause();
WebServiceException exception;
String contextPath = ExceptionUtil.BASE_EXCEPTION_PACKAGE;

if (cause == null || !(cause instanceof WebServiceException)) {
// If not a WebService Exception then throw a WebService Exception
exception = new WebServiceException(..., Status.SERVER_ERROR_INTERNAL);
} else {
exception = (WebServiceException) cause;
String clazzContext = ExceptionUtil.getContextPath(cause.getClass());

if (!contextPath.equals(cause.getClass().getPackage().getName())
&& !clazzContext.equals(contextPath)) {
contextPath = contextPath + ":" + clazzContext;
}

if (exception.getStatus() == 0) {
exception.setStatus(Status.SERVER_ERROR_INTERNAL);
}
}

// Create the Exception Wrapper
WebServiceExceptionWrapperDto<WebServiceException> exceptionWrapper = new WebServiceExceptionWrapperDto<WebServiceException>();
exceptionWrapper.targetException = exception;

JaxbRepresentation<WebServiceExceptionWrapperDto<?>> rep = new JaxbRepresentation<WebServiceExceptionWrapperDto<?>>(
exceptionWrapper);

rep.setContextPath(contextPath);
response.setEntity(rep);
return Status.valueOf(exception.getStatus());
}
}

I have included herewith a Maven example that will demonstrate the client, service and exception management in use. The maven module titled "exceptionnmodule" contains the exception handling code. The integration will demonstrate different exceptions that are thrown by the server and re-thrown by the client code. In particular, you can see the the following abilities of the framework:

a. Ability to throw a simple RuntimeException from the Service and see it marshalled/unmarshalled as WebServiceException
b. Ability to throw an instance of WebServiceException
c. Ability to throw an exception that is a subtype of WebServiceException but without any addition body. For example, the OrderException.
d. Ability to throw a sub type of the WebServiceException that has additional body elements. For example, OrderValidationException.

One can witness the server stack, cause, HTTP Status, specific exception types. Simply run a "mvn install" from the top level project.

Although the example uses Restlet 2.0, the framework has no dependencies on the same. There might be many areas for improvement in the concept and I would like to hear about them if a reader of this blog has ideas. Also, in this example, I have used code from a posting by Ian Robertson on Artima regarding Reflecting Generics.

Finally, I don't know how this BLOG will format. I am trying something new, I hope it woiks!

Download the Code HERE!

Wednesday, August 26, 2009

Restlet 2.0 example using Spring, Maven, Annotations and a Custom Converter

Its been some time since I posted regarding REST. Restlet 2.0 is on its way, I figured its time I updated the example I had created some time ago to a Restlet 2.0 milestone release. If you are continuing to read this BLOG, I would recommend a read of the Restlet 1.1 BLOG I mentioned prior to reading this one.

So whats new with Restlet 2.0? A read of the Resource API refactoring page is recommended.

One of the major features that I liked about JAX-RS over the core Restlet API while working with the former is that it is annotation driven versus class driven. Automatic content negotiation and the calling of the appropriate method in your Server Resource class for different Content-Types is really best left to the container. A method intended to serve content should not have if/else blocks for different content, something one would need to do in Restlet 1.1. Restlet 2.0 seems to take the same into consideration by providing the "best of both worlds"

There were major changes that occurred on Restlet version 1.2 that the team at Restlet decided to change the release to a 2.0. Further details of the change can be obtained at http://blog.noelios.com/2009/05/27/restlet-2-0-m3-released/

Restlet 2.0 has a major change in the way Resources are viewed. In particular a split into a ClientResource and a ServerResource. Details of the same can be viewed on the Restlet Resource API refactoring WIKI page. In addition, there has been a refactoring of certain core classes to different packages. I also noticed that the Restlet API and Restlet engine are shipped as a single jar org.restlet.jar. I could not find the com.neolios packages any more. Finally, there is a jse and j2ee edition of Restlet available.

So, eager to try out the Restlet 2.0 annotations, I reworked my example to use Restlet 2.0 while simplifying the Spring integration present.

Compared to my previous example, I have reduced my Spring applicationContext.xml to be more thinner by using a more annotated approach.



<context:component-scan base-package="com.welflex">
</context:component-scan>

<!-- Spring Application. Note there are no mapping of resources here -->
<bean id="application" class="org.restlet.Application">
<property name="name" value="orderApplication"></property>
<property name="root" ref="root" />
<!-- Added to handle Exceptions centrally -->
<property name="statusService" ref="applicationStatusService"/>
</bean>

<bean id="root" name="router" class="com.welflex.order.rest.SpringBeanRouter"/>
<bean id="beanMapper"
class="net.sf.dozer.util.mapping.DozerBeanMapper">
<property name="mappingFiles">
<list>
<value>dozerBeanMapping.xml</value>
</list>
</property>
</bean>



The OrdersResource has changed to:



@Component(value = "/orders")
@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrdersResource extends ServerResource {
......
@Post
public JaxbRepresentation<OrderDTO> createOrder(Representation rep) {
...
return new JaxbRepresentation<OrderDTO>(MediaType.APPLICATION_XML, dto);
}

}



Note that the OrderResource class now extends ServerResource unlike in my previous example which extended the Resource class. Also note that the mapping of the resources to paths are defined in the OrderResource class itself unlike in the xml file in my previous example. The POST method, called createOrder() accepts and returns XML. The return type is a JaxbRepresentation. If we desired, we could have another method in the code that accepts JSON content and returns JSON content that is titled createJSONRepresentation(). Depending on the content type requested, the appropriate method would be called. This is much better than having an if/else block in an acceptRepresentation(Variant v) method.

Take a look at the Products Resource, that returns a JSON representation when a GET method is invoked. Note that this method will only return JSON content.



@Component(value = "/products")

@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class ProductsResource extends ServerResource {

......
@Get
public JsonRepresentation getProducts() {
.....
return new JsonRepresentation(jsonString);

}
}



Pretty nice, I would say. To spoil the party, one of the things I liked about the JAXRS implementations was that I could annotate a Resource with the media type it supported and if it were a JAXB object it would automatically marshall the XML, if it were JSON, it would automatically marshal the same as well.

The same can be accomplished in Restlet by using a Converter Service that converts a java object to and from a media type. There are plans for adding JAXB converters amongst others in 2.0 M5. In short, by using a Converter Service, I would like the createOrder() method to have the following signature, i.e., method returns a JAXB Object, however the container knows how to marshal that into XML back to the caller:



@Post("xml":"xml")
public OrderDTO createOrder(OrderDTO inOrder) {

return outOrderDTO;
}



So what about the ClientResource class? How does that work? Well with Restlet 2.0, one can continue to use the Restlet Client class or use the ClientResource class. The ClientResource is not thread safe. An example of how the ClientResource can be used is shown below in the OrderClientImpl class:



public class OrderClientImpl implements OrderClient {

private static final String ORDERS_RESOURCE_URI = "/orders";
private final String baseUri;

public OrderClientImpl(String baseUri) {
this.baseUri = baseUri;

}

public OrderDTO createOrder(OrderDTO order) throws IOException {

ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI);

try {
return ordersResource.post(order, OrderDTO.class);

}
catch (ResourceException e) {
throw new IOException(e);

}
}

public void updateOrder(OrderDTO order) throws OrderException {

JaxbRepresentation<OrderDTO> orderCmd = new JaxbRepresentation<OrderDTO>(
MediaType.APPLICATION_XML, order);

ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI + "/"

+ String.valueOf(order.getOrderId()));
try {

ordersResource.put(orderCmd);
}
catch (ResourceException e) {

throw new OrderException("Order Update Failed", e);
}
}

public OrderDTO getOrder(Long orderId) throws OrderNotFoundException, IOException {

....
}

public void deleteOrder(Long orderId) throws OrderException {

....
}
}



I quite like the use of ClientResource versus the Restlet Client as I feel its more RESTful, i.e., talking to a Resource using HTTP verbs. However, again we witness the JAXB representation creeping in as we saw with the service due to the lack of a converter.

I decided to create my own Converter for JAXB to see if I can get it to work and simplify the code base. One of the things I wanted to add just for my pleasure is a way to propagate server stack traces to the client. This is NOT what one would typically include in a representation of a resource as one would not want clients getting details of the working of a server, however, as this is my playground, anything goes ;-). After some tinkering, the following is my Jaxb Converter, it assumes that all JAXB root level objects have the annotation @XmlRootElement:



public class CustomXmlConverter extends XmlConverter {

......
@Override
public List<Class<?>> getObjectClasses(Variant source) {

List<Class<?>> result = super.getObjectClasses(source);

result = addObjectClass(result, JaxbRepresentation.class);
result = addObjectClass(result, Object.class);

return result;
}

@Override
public List<VariantInfo> getVariants(Class<?> source) {

List<VariantInfo> result = super.getVariants(source);

if (source.getAnnotation(XmlRootElement.class) != null

|| source.isAssignableFrom(JaxbRepresentation.class)) {
result = addVariant(result, VARIANT_APPLICATION_ALL_XML);

result = addVariant(result, VARIANT_APPLICATION_XML);
result = addVariant(result, VARIANT_TEXT_XML);

}

return result;
}
@SuppressWarnings("unchecked")

@Override
public <T> T toObject(Representation source, Class<T> target, UniformResource resource) throws IOException {
.....
if (target.getAnnotation(XmlRootElement.class) != null) {
return new JaxbRepresentation<T>(source, target).getObject();

}

throw new IllegalStateException("Should not have got here");
}

@Override
public Representation toRepresentation(Object source, Variant target, UniformResource resource) throws IOException {

Representation result = null;
if (source.getClass().getAnnotation(XmlRootElement.class) != null) {

result = new JaxbRepresentation<Object>(source);
}

else {
result = super.toRepresentation(source, target, resource);

}

return result;
}
.....
}




Great, so we have a Converter, how do we make Restlet use this converter for XML. In my common maven module (shared between client and web), I define a file in META-INF/services/org.restlet.engine.converter.ConverterHelper, and in that file, I have a single entry defining my custom converter. Now when the client and service code use the common library, they will use the Custom Converter I have defined for JAXB.
Thus, to illustrate the change, my new resources for Orders and Order are shown below:



@Component(value = "/orders")

@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrdersResource extends ServerResource {
.....
@Post("xml")

public OrderDTO createOrder(OrderDTO dto) {
....
return createdOrderDto;

}
...
}

@Component(value = "/orders/{id}")

@Scope(value = BeanDefinition.SCOPE_PROTOTYPE)
public class OrderResource extends ServerResource {

@Put(value = "xml")
public void updateOrder(OrderDTO orderDTO) {

try {
String id = (String) getRequest().getAttributes().get("id");

orderDTO.setOrderId(Long.parseLong(id));
persistOrder(orderDTO);

}
catch (RuntimeException e) {
getResponse().setStatus(Status.CLIENT_ERROR_BAD_REQUEST);

throw new OrderException("Failed to update an order", e);
}
}
.....
}



We are no longer returning Representations in the above code. Quite similar to JAXRS, wouldn't you say ;-)? On the Client end, the OrderClientImpl no longer deals with Representations but only OrderDTOs as shown below:



public class OrderClientImpl implements OrderClient {

....

public OrderDTO createOrder(OrderDTO order) throws OrderException {

ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI);


try {
return ordersResource.post(order, OrderDTO.class);

}
catch (ResourceException e) {
throw new OrderException("Error Creating an Order", e);

}
}

public void updateOrder(OrderDTO order) throws OrderException {

ClientResource ordersResource = new ClientResource(baseUri + ORDERS_RESOURCE_URI + "/"

+ String.valueOf(order.getOrderId()));
try {

ordersResource.put(order);
}
catch (ResourceException e) {

throw new OrderException("Order Update Failed", e);
}
}
}




I like the simplicity of the above and cannot wait to see the release of Restlet 2.0 that has the additional converters built in.

For those interested, I have a maven project example available for download. The project provided uses the custom JAXB converter. To execute the project, one would need to use JDK1.6.X and maven 2.0.9 or higher. To view an integration test in action execute, "mvn install" from the root level of the project. The result should be something like:



-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.welflex.order.IntegrationTest
log4j:WARN No appenders could be found for logger (com.welflex.client.ProductClientImpl).
log4j:WARN Please initialize the log4j system properly.
Aug 28, 2009 8:59:55 PM org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
20:59:55 DEBUG - com.welflex.order.rest.ProductsResource.getProducts(44) | Getting Products in JSON Format
Number of Available Products:3
Product Id:663123
Product Id:9912123
Storing the order...
Aug 28, 2009 8:59:55 PM org.restlet.engine.http.StreamClientHelper start
INFO: Starting the HTTP client
20:59:56 DEBUG - com.welflex.order.rest.OrdersResource.createOrder(45) | Call to post an order:..........
Updating the order...
Order successfully
Retrieving the order...
Deleting the order..
20:59:56 DEBUG - com.welflex.order.rest.OrderResource.getOrder(86) | Requested order with id:5499731937442305515
20:59:56 DEBUG - com.welflex.order.rest.ApplicationStatusService.getStatus(21) | In Order Status Service :class org.restlet.resource.ResourceException
Expected Order to not be found. Look at the server stack we got back
com.welflex.exception.OrderNotFoundException: Error obtaining Order
at com.welflex.client.OrderClientImpl.getOrder(OrderClientImpl.java:60)
at com.welflex.order.IntegrationTest.testLifeCycle(IntegrationTest.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:59)




More on Restlet 2.0 once its released. For now, maybe I should probably look toward cleaning up the JaxbConverter and submitting it to Restlet or better still head in the search of kindred spirits, if you know what I mean ;-)

Tuesday, August 11, 2009

Documentating Software, my 2c

Documentation....Aaargh..why in heavens name would anyone waste time on it? The following few words are a window into my experiences and thoughts on documentation in IT organizations.

Sadly, most organizations cannot or will not afford, the luxury of an additional resource who is dedicated to just the task of documentation, i,e., a technical writer. So as developers and architects where does that leave us ? I have heard statements such as:
  • "Your code should be clear enough such that documenting the same is redundant"
  • "Documentation is not much value unless its kept up to date, bad or expired documentation is worse than no documentation"
  • "It is more important that the code has enough test coverage. The tests if good, document the code use"
  • "We do not have the time, it is lowest on our priority list. We got bigger fish to fry."
Don't get me wrong, I am not in total disagreement regarding the above. Neither am I a subject matter expert on documentation. I am someone who often finds myself in the middle of this.

What I find myself questioning is:
  • If the code is so self documenting or clear, and if you find people keep asking the same question over an over, is it really so self documenting ?
  • If one created documentation, clearly it was considered important. If important, then it would need to updated with the same love when created.
  • And please, there is always time...:-)
I have often entered an enterprise only to find that there is no Enterprise level documentation, stale or current. On the other hand, I have walked into an Enterprise only to be handed a thick set of pages to wade over to acclimatize myself. Neither worked for me, what I was seeking as an entrant was to see the enterprise from the 2000 feet level and then drill down as appropriate.

I am a very visual person, many a time a figure to me means more than a 1000 words. If there is a diagram that describes the stack the company uses, the applications, the services, even if partially wrong, immediately gives me a description of the architecture and/or project. That is a big win as far as acclimatising me to the environment.

When different folk ask the same question to me many times over, to me, that is another clearly indication of a lack of documentation. If a topic is constantly questioned, it deserves a place IMHO for documentation such that the next time the question arises, I point the person posing the question to the documentation instead of the repeated sermon that only I hold the copyrights to. DRY (Don't repeat yourself) does not apply to code alone...

Documentation is often created with great detail and appears as a big win, however if it starts becoming stale, then it loses its value and significance and becomes more of a "White Elephant". A conversation I had with someone in the past:

Someone: "Can you help me with this?"
Me: "Sure, did you check the documentation on the same?"
Someone: "Yeah but that is so outdated, so I don't bother looking at it any more"

The above conversation totally supports the argument that bad or documentation that are stale are definite deterrents to readers.

The following represent a few directions regarding documentation that I will try to adhere to:
  1. Enterprise or Project Documentation or "The BIG Picture documentation": From an Enterprise architect perspective, providing a high level description/diagram of the entire enterprise and how it all fits or works together can be valuable. The documentation does not have to drill into the microscopic details but instead provide a more a high level view that helps an entrant understand how all the parts of an enterprise fit together. From the perspective of a team lead/lead developer, it would be beneficial if documentation were provided regarding the boundaries and dependencies of a particular project. There is no need for details of a database schema, however there is need to indicate that the application talks to a particular schema on a particular database . These documents might even be able to help detect problems in the architecture, as now, you are looking at "The Big Picture" From previous experience, I also feel that documenting "Big" decisions that affect a project can have great value. With time, we tend to forget the reasons for choosing a direction. A place to consult back as to "Why?" can prove valuable.
  2. Stack Documentation: A diagram and/or description indicating the various technology stack in play. For example, "JBoss - App Server, Spring Framework, HP UX - 10.2 deployment OS, ..". This might be encompassed within the scope of point no.1. Again, stacks are rarely changed, and if they are, they probabaly require immediate attention.
  3. Repeated Query Documentation: If any question posed to a developer or architect is being asked more than once, regardless of its longevity deserves due diligence in the documentation space. In lawyer terms, it would classify as "Asked and answered" :-). If you enjoy the interaction and enjoy repeating the same answer, well then more power to you , please disregard the documentation angle.
  4. Redundant Documentation : Code that contains redundant or unnecessary documentation (i.e., code is self explanatory) serves no purpose. I violate this sometimes and need to stop. For example, getters and setter of a POJO (Plain Ole Java object) rarely requires documenting if the method name is clear. Institute naming standards that are enforced. Don't bother with class or schema diagrams, any developer worth his salt will be able to easily find free tools to reverse engineer classes or database schema's and be able to get the most current versions of the same.
  5. Documenting Complex Code: Code that is unavoidably complex, deserve either strong documentation or better still refactoring.
  6. Job security as it applies to documentation: Job security is not improved by keeping the details in your head. People are smart and folks that tend to do that will not last in a smart organization. Organizations lacking strong leaders, well get strong and smart leaders :-)
  7. When to document and how much ? : Things that are in constant flux are hard to document, wait for stabilization. However, do follow up after stabilization occurs. Make sure your code works first, then comes the documentation. If you have good documentation but code that is broken, thats not a win. In the same vein, I would go further to say, Unit Tests and Integration Tests are more valuable than documentation. Place yourself in the position of a person who will utilize the documentation, make the content detailed enough for your intended audience. For example, in an enterprise level document, do not bother listing individual Web Services or Database Schemas, it is enough to state that Web Services communicate with databases and or other Web Services. The detail in the documentation should also be only that much that you can afford to keep current moving forward. Finally piggy backing on this point, broken windows or no windows from the top can send a bad message. What I mean is, "If an architect or lead themselves do not document, then it becomes a hard sell to a developer that they need to document"
  8. Detail and Artistry of Documentation: Reiterating, "Make it right before making it pretty". A straw diagram caught on a white board via a cell phone camera during the design, if posted on a WIKI is often as effective as a colorful work of art. High ceremony documentation is almost too much and often unnecessary. Concise and to the point documentation works in most cases. People are smart, they often like a start. Its like being provided a map into Yellowstone regarding the sights to see. One does not require information on the concentration of sulfur at each site, just where the sites are located, we'll figure out the sulfur content at each site :-)
  9. Tools for documentation: Use a tool that works for collaborative authoring such as a WIKI. Almost any software development project IMO deserves a WIKI or equivalent presence. Media WIKI and such are free to use, so a smart person would avail such tools. Use technology where applicable.
  10. Collaborative Documentation: Documentation on IT projects should not belong to a single owner but be considered a collaborative effort. In other words, every member of the team should feel that they can add or correct toward the documentation. If a person comes asking a question and you give an answer, if you ask the person to document the same as you feels its valuable and is missing, you might be surprised as to how gladly they are willing to contribute. On the same note when you find documentation that is outdated and more importantly wrong, deprecate the documentation at the very top if you do not have the time to make it current.
  11. How to get started, documentation: As an entrant into a new IT environment, one often needs to get set up and be ready to get productive. A road map documentation that provides a user of "What, Why, Where, Who" can be quite useful.
  12. Selling documentation to the business: Getting business to account for time slots for code testing itself can be a daunting task in some organizations, getting a time slot in the project budget for documentation can be harder sale. However, as a lead or person making an estimate of a project, it is important to bake in the same. Documentation is money, however you need to sell the same and its value to the business. Lack of critical documentation is something that can come back to haunt you. "Protect your skin and those of your kinsmen" :-)
I am sure I have only touch the tip of what needs documentation. I cannot claim the title of a "Documentation nazi" and/or authority, neither am I claiming a "One set of rules fits all". People have probably written books on the same.....Easy to preach, hard to do, but hey, trying is not so hard, is it, and that is motive, trying ;-)...

Tuesday, June 23, 2009

Scaling in the JMS World...in particular Topics

Scalability and Availability in JMS. For the former, as load on the messaging system increases, one would want to be able to service the increased messages without performance degradation. For the latter, one should ensure the reduction if not elimination of single point of failures, i.e., ensuring system is available for use.

Most JMS Providers have some sort of mechanism for addressing the above. This BLOG is however only going to concentrate on Oracle WebLogic.

WebLogic has an interesting way in which JMS Destinations (Queues and Topics) are scaled via Distributed or Virtual Destinations. A Virtual Destination can be considered a sort of router/load balancer that distributes messages it receives over to actual or physical destinations. Each physical destination would reside in separate WebLogic instances on potentially separate hardware. See figure below where a virtual destination "test_queue" has physical members test_queue_1, test_queue_2 and test_queue_3 on separate WebLogic instances:



As an example of Virtual Queues in action, in the conceptual figure shown below, Producers send messages to the Virtual Queue which then directs the message to one of the physical destinations



P : Producers
VQ: Virtual Queue
PQ-X: Physical Queue, where X is the number of the Queue
C: Consumer
M: Message

In the case of a Virtual Topic, messages sent to the Virtual Topic are sprayed on to each and every physical member as shown in the figure below where VT and PT-X represent Virutal and Physical topics respectively:


Scaling Queues:
From the figure shown regarding Queues, it is pretty simple to follow how the system scales and that it will continue to service new messages and consume them should one of the servers in the messaging cluster fails. A message will be consumed at most by one of many consumers successfully. Adding Consumers to the end of the physical members allow the architecture to process messages rapidly, i.e., Competing Consumer Design Pattern.

Now, what if one of the servers having physical members went down, for example VQ-1 went down? From the perspective of availability, messages are still flowing and getting consumed from the other available member, i.e., VQ-2. What about the messages that were in the physical member whose server went down? One has multiple options. The first and most straight forward is to restart the server and re-connect the consumers to drain the queue. WebLogic allows for auto-service migration in the event of failure. This is another feature that can assist in recovering from failure. Queues readily tend to present themselves for scaling. One note on the same, if message ordering is put into play, all bets are off :-)..more on the same on a subsequent blog.

Scaling Topics:
Topics unlike Queues present an interesting challenge. In the case of a topic, a message is delivered to all listeners of the topic and each listener is typically different than its peer in how it handles the message. For example a message posted to an order topic, might have a consumer that processes the order while another one looks at inventory calculations:


Clearly we cannot have two consumers that process the same order and it is redundant to have duplicate consumers perform the same inventory calculations.

Messaging systems have the concept of a Durable Subscriber, i.e., a Message listener who registers with the topic such that they are delivered messages sent to the topic even when the listener is not connected. Message providers store the message on behalf of the durable subscriber and forward the same when the subscriber comes back on line.

In WebLogic one cannot connect to a durable subscriber using a distributed (virtual) destination JNDI. One has to explicitly register with a physical member. This is one gripe that I have with the WebLogic solution. Adding to the problem, should one connect to one physical member and if that member goes down, attempting to establish the durable subscription on other available members are treated like totally new subscriptions. In consolation, if the server goes down that contains durable subscriptions, then messages will be stored on other physical members of the virtual topic on behalf of the durable subscribers, and when the server is restored, messages get forwarded to the durable subscribers when they comes back on line as shown below:


Now, as each subscriber of the topic has a unique agenda from the other subscribers, the fact that it is the sole entity that services the message, the same represents a single point of failure and a proves a scalability inhibitor as it cannot concurrently process multiple messages.

So how can one scale a topic so that a message on a given type of subscriber can be rapidly processed? One direction is have the consumer of a Topic post the message onto a Queue which can then be processed by competing consumers. One way this can be achieved is to have a Message Driven Bean withing the container whose sole purpose is to transfer the message from the topic to a queue for those consumers that need to scale. An example of the same is shown below:


In the above shown example, as one consumer of the topic needs processing to rapidly occur of messages received, an MDB (registered as a durable subscriber of the topic) is used to transfer every message onto a Queue which is then consumed by competing consumers. As other consumers, DC-1 and DC-2 are not in need of scaling, they remain durable single subscribers.

The above solution works to scale the durable subscriber of a topic. However, in WebLogic when using distributed topics the durable MDB still represents a single point of failure.

A question to ask at this point is, "Do you really need a topic for the use case at hand?" Topics are great IMO when the subscribers tend to be dynamic in nature or do not need to be durable, an example of the former is a stock quote topic and an example of the latter is a event listener that requires notification of an event.

When the number of subscribers are a known or finite in amount, the use of a queue coupled with the MDB can easily achieve a scalable solution as shown below where the MDB transfers messages on the in-queue to subsequent out-queues where competing consumers can scale and consume the messages:



Transfer MDB:
It is pretty trivial to create the Transfer MDB. One would typically like to re-use the Transfer MDB Java Code with only configuration changes for different applications. The same can easily achieved via separate projects where target destination can be provided via @Resource. A simple transfer MDB is shown below:



@MessageDriven
@TransactionAttribute(TransactionAttributeType.REQUIRED)

public class MessageRouterMdb implements MessageListener {

/**
* Map this jms connection factory to the actual one in the weblogic-ejb-jar.xml deployment
* descriptor of the ejb project. The Connection factory to be used will be the one that is
* specific to this mdb.
*/

@Resource(name = "jms/connectionFactory")

private ConnectionFactory connectionFactory;

/**
* Injected in Message Driven Context
*/

@Resource
private MessageDrivenContext mdc;

/**
* Define a comma separated list of destinations JNDI names in the ejb-jar.xml via an environment
* entry.
*/

@Resource
private String targetDestinations;

// Initialized
private List<Destination> targetJmsDestinations;

private MessageProducer messageProducer;

private Connection connection;

private Session session;

@PostConstruct
public void init() {
try {

InitialContext ctx = new InitialContext();
connection = connectionFactory.createConnection();

session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

messageProducer = session.createProducer(null);
String[] targetDestArray = StringUtils.split(targetDestinations, ",");

targetJmsDestinations = new ArrayList<Destination>();

for (String targetDest : targetDestArray) {

targetJmsDestinations.add((Destination) ctx.lookup(targetDest));

}

connection.start();
}
catch (Exception e) {
throw new RuntimeException("Error Initializing Message Router Mdb", e);

}
}

public void onMessage(Message message) {

try {
// Loop through target destinations and send message to each destination
for (Destination dest : targetJmsDestinations) {

messageProducer.send(dest, message);
}
}

catch (Exception e) {
mdc.setRollbackOnly();

}
}

@PreDestroy
public void cleanup() {

// Close open jms resources here...
}
}




The CODE for the above MDB and an example Maven Project that uses WebLogic is available for download HERE. Note that the code is not specific to WebLogic and can really be used in any container, for example JBoss.

Conclusion:

Scaling topics via an MDB and target queues just felt like an obvious pattern to me. WebLogic, like many JMS Providers has the concept of a Message Bridge. Initial investigation led me to believe that a message bridge more serves the purpose of transferring a message from one provider to to another rather than serving to scale a Topic and distributed messages to multiple destinations.

There may be cases as mentioned where the number of subscribers are not known. In such a case, a topic might work and it becomes the responsibility of the subscribers to scale on their end, for example multi-threading upon receiving the message.

What really would be nice if the JMS specification had the concept of "Durable Competing Consumers" where a Durable subscriber could register multiple instances of itself such that a message to a topic would be delivered to at most it or one of its clones :-). I believe there are some Messaging providers who permit such a pattern via customization that extend the JMS specifications.

Before bidding adieu, this has been one of my most figurative blogs, all Ye artists, look at the diagrams with a forgiving eye, "I am a programmer not an artist!" :-)

Friday, May 22, 2009

Ending the drought...

Its been a month since I posted anything on my BLOG, cannot let May end with a drought. It has been rather a trying month for me, the reasons I am saving for a later melodramatic post ;-)...

Right now I am very happy. At the company I currently work at, we released out a JMS architecture with minimal hiccups. It was very gratifying to witness the entire team involved work and mesh so well in order to make the architecture work. Everyone from Architects, Developers, System Administrators, Testers worked so well together. I must state that my level of confidence in the team and players involved was so much that I did not even have to be there during the go live in person. Put it in a word, my current organization is so much about "WE" and not "ME". That is a motto that works IMHO. Guess my current organization is very talented in its selection process of the people they want on board.

I truly enjoyed working with all the folks on the various facets involving the deployment of the architecture, frameworks, deployment, projects etc. We suffered a few hiccups after the deploy as well but the messaging solution itself was 100% solid. In fact in my own retarded way, I appreciated the fact that the failures occurred as they served to demonstrate that the failure path in fact worked ;-)

Personally, I have experienced considerable growth in the areas of JMS Messaging. However, every once so often I again am humbled by what I don't know or did not anticipate.

The project as a whole was like our baby that finally saw the light..and danced on...I hope it keeps dancing :-))))))))))) Even if does not, I believe we have a team strong enough to put it on crutches to make it so...

I can't wait for further adoption of the architecture...so thrilled, had a blast...

More BLOG's coming up soon, the little GREY cells were on vacation (not dead, just vacationing), you should see something from them soon....

Friday, April 24, 2009

Browsing Weblogic JMS Distrubuted Queues

On of the interesting features of Weblogic is that when javax.jms.QueueBrowsers are created, they are pinned to a particular physical member of a distributed destination. What that means is that a QueueBrowser created for a distributed destination will only be able to browse messages on the member it gets pinned to and not the whole queue.

One can try to create X number of queue browsers based of the number of servers in the provider URL, for example, "t3://localhost:8002, localhost:8003", and hope that no two queue browsers will pin to the same destination, i.e., at the mercy of Weblogic round robin if possible. Even if we do manage to do that, one still needs to iterate over X separate enumerations in order to manage the message. Consider an integration test where one sends a message to a distributed destination and would have to create X queue browsers to determine whether or not the message is present.

What would be really nice is if the Weblogic runtime gave us a QueueBrowser that provided an enumeration of enumerations of the physical members of a distributed queue. That however is not present. So, I decided to create a simple example that would help me with the same. At the same time, I am interested in understanding the JMX capabilities of Weblogic.

The example will create a QueueBrowser and obtain an enumeration from each of the physical members of the distributed destination and provide back an enumeration of the respective enumerations. In the case of uniformly created distributed destinations, the JNDI name of the physical members will be of the format JMSServerName@JNDIName. If one knows the JMSServerName thats great. However, one should not have to rely on the same as JMSServers might be added/removed. It would be nice if the distributed member JNDI names could be located on the fly by using Weblogic's JMX capabilities and thats what I intend to do. Note that the enumerations obtained in Weblogic are snapshots of the current state of the queue and are not dynamic, i.e., new messages added being dynamically reflected in the enumeration. Finally, I am providing a java.util.Enumeration for the JMS messages instead of an java.util.Iterator as Enumeration does not have a "remove" method and I hate putting an "UnsupportedOperationException" in the Iterator and also I would like to be true to the JMS API.

So first obtaining the JNDI name of physical members of a distributed queue. Weblogic JMX is a tree for objects. Once we obtain the ServerRuntimeMBean information, we can obtain information about the JMSRuntimeMBeans and then the different JMSServers and finally the JNDI names of the JMS Servers as shown below:



public class WeblogicMBeanHelper {
private final MBeanServerConnection connection;

private final JMXConnector connector;

private final ObjectName service;

public WeblogicMBeanHelper(String url, String userName, String password) {

try {
service = new ObjectName(
"com.bea:Name=DomainRuntimeService,Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean");

}
catch (MalformedObjectNameException e) {
throw new AssertionError(e.getMessage());

}
......
......
JMXServiceURL serviceURL;

try {
serviceURL = new JMXServiceURL("t3", hostName, Integer.valueOf(port), jndiroot + mserver);

Hashtable<String, String> h = new Hashtable<String, String>();

h.put(Context.SECURITY_PRINCIPAL, userName);
h.put(Context.SECURITY_CREDENTIALS, password);

h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
connector = JMXConnectorFactory.connect(serviceURL, h);

connection = connector.getMBeanServerConnection();
}
catch (Exception e) {

throw new RuntimeException(e);
}
}

public Iterable<String> getDistributedMemberJndiNames(String distributedDestJndiName) {
Iterable<String> serverNames = getJmsServerNames();

Set<String> distributedDestNames = new TreeSet<String>();

for (String serverName : serverNames) {
distributedDestNames.add(serverName + "@" + distributedDestJndiName);

}

return distributedDestNames;
}

public Iterable<String> getJmsServerNames() {

.....
}

public Iterable<ObjectName> getJMSServers() {

....
}

public Iterable<ObjectName> getJMSRuntimes() {

....
}

public List<ObjectName> getServerRuntimeMBeans() {

try {
return Arrays.asList((ObjectName[]) connection.getAttribute(service, "ServerRuntimes"));

}
catch (Exception e) {
throw new RuntimeException("Error obtaining Server Runtime Information", e);

}
}

public void close() {

// Close the connector
....
}
}





public class WeblogicMBeanHelper {
private final MBeanServerConnection connection;

private final JMXConnector connector;

private final ObjectName service;

public WeblogicMBeanHelper(String url, String userName, String password) {

try {
service = new ObjectName(
"com.bea:Name=DomainRuntimeService,Type=weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean");

}
catch (MalformedObjectNameException e) {
throw new AssertionError(e.getMessage());

}
......
......
JMXServiceURL serviceURL;

try {
serviceURL = new JMXServiceURL("t3", hostName, Integer.valueOf(port), jndiroot + mserver);

Hashtable<String, String> h = new Hashtable<String, String>();

h.put(Context.SECURITY_PRINCIPAL, userName);
h.put(Context.SECURITY_CREDENTIALS, password);

h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
connector = JMXConnectorFactory.connect(serviceURL, h);

connection = connector.getMBeanServerConnection();
}
catch (Exception e) {

throw new RuntimeException(e);
}
}

public Iterable<String> getDistributedMemberJndiNames(String distributedDestJndiName) {
Iterable<String> serverNames = getJmsServerNames();

Set<String> distributedDestNames = new TreeSet<String>();

for (String serverName : serverNames) {
distributedDestNames.add(serverName + "@" + distributedDestJndiName);

}

return distributedDestNames;
}

public Iterable<String> getJmsServerNames() {

.....
}

public Iterable<ObjectName> getJMSServers() {

....
}

public Iterable<ObjectName> getJMSRuntimes() {

....
}

public List<ObjectName> getServerRuntimeMBeans() {

try {
return Arrays.asList((ObjectName[]) connection.getAttribute(service, "ServerRuntimes"));

}
catch (Exception e) {
throw new RuntimeException("Error obtaining Server Runtime Information", e);

}
}

public void close() {

// Close the connector
....
}
}




Now that we have a way of obtaining the distributed members, the following represents the QueueBrowser that enumerates over the different members and consolidating the same into a single enumeration. I have a second enumeration that provides information regarding the JMServer that hosts the given message:



public class DistribuedQueueBrowser {
...
private final Iterable<String> queueNames;

public DistribuedQueueBrowser(String adminUrl, String providerUrl,
String distributedDestinationName, String userName, String password) throws Exception {

ctx = getInitialContext(providerUrl, userName, password);
WeblogicMBeanHelper helper = null;

try {
helper = new WeblogicMBeanHelper(adminUrl, userName, password);

queueNames = helper.getDistributedMemberJndiNames(distributedDestinationName);
}
finally {

if (helper != null) {
helper.close();

}
}

// Set up connection/session..
....
}

private InitialContext getInitialContext(String providerUrl, String userName, String password) throws Exception {

....
}

// Retrun an Enumeration of Messages only
public Enumeration<Message> getEnumeration() throws JMSException, NamingException {

return new JmsMessageEnumeration(getMessageEnumeratorMap());
}

@SuppressWarnings("unchecked") private Map<String, Enumeration<Message>> getMessageEnumeratorMap() throws JMSException,
NamingException {

Map<String, Enumeration<Message>> serverMessageMap = new HashMap<String, Enumeration<Message>>();

for (String queueName : queueNames) {
String serverDq[] = StringUtils.split(queueName, "@");

Queue queue = (Queue) ctx.lookup(queueName);

javax.jms.QueueBrowser qb = session.createBrowser(queue);

serverMessageMap.put(serverDq[0], qb.getEnumeration());

}

return serverMessageMap;
}

// Return an enumeration of ServerLocatedMessage that contains the
// the jms server that houses the message

public Enumeration<ServerLocatedMessage> getServerLocatedEnumeration() throws JMSException,
NamingException {

return new ServerLocatedMessageEnumeration(getMessageEnumeratorMap());
}

public static class ServerLocatedMessage {
private final Message message;

private final String jmsServerName;

public ServerLocatedMessage(String jmsServerName, Message message) {

this.message = message;
this.jmsServerName = jmsServerName;

}
...
}

private static abstract class AbstractMessageEnumeration<T> implements Enumeration<T> {

Map.Entry<String, Enumeration<Message>> current;
private Enumeration<Message> currMessageEnumer;

private final Iterator<Map.Entry<String, Enumeration<Message>>> iterator;

public AbstractMessageEnumeration(Map<String, Enumeration<Message>> map) {

iterator = map.entrySet().iterator();
current = iterator.hasNext()

? iterator.next()
: null;
currMessageEnumer = current != null

? current.getValue()
: new Enumeration<Message>() {

public boolean hasMoreElements() {
return false;

}

public Message nextElement() {
throw new NoSuchElementException();

}
};
}

Enumeration<Message> getEnumeration() {

if (current == null || currMessageEnumer.hasMoreElements()) {

return currMessageEnumer;
}

while (iterator.hasNext()) {

current = iterator.next();
currMessageEnumer = current.getValue();

if (currMessageEnumer.hasMoreElements()) {
return currMessageEnumer;

}
}

return currMessageEnumer;
}

public boolean hasMoreElements() {
return getEnumeration().hasMoreElements();

}
}

// Wraps the JMS Message within a ServerLocatedMessage object
// which contains the server name along with the message
private static class ServerLocatedMessageEnumeration extends

AbstractMessageEnumeration<ServerLocatedMessage> {

public ServerLocatedMessageEnumeration(Map<String, Enumeration<Message>> map) {

super(map);
}

public ServerLocatedMessage nextElement() {

Message message = getEnumeration().nextElement();
return new ServerLocatedMessage(current.getKey(), message);

}
}

private static class JmsMessageEnumeration extends AbstractMessageEnumeration<Message> {

public JmsMessageEnumeration(Map<String, Enumeration<Message>> map) {
super(map);

}

public Message nextElement() {
return getEnumeration().nextElement();

}
}

public void close() {

......
}
}



Using this QueueBrowser is rather straight forward and can be done via:



// Note that the first argument is the admin url and the second is the
// managed server url.
DistribuedQueueBrowser qb = new DistribuedQueueBrowser("t3://localhost:7001",
"t3://localhost:8001",
"test_queue", "weblogic", "weblogic");

Enumeration<Message> i = qb.getEnumeration();
while (i.hasMoreElements()) {

Message m = i.nextElement();
System.out.println("Message:" + m);

}

Enumeration<ServerLocatedMessage> sli = qb.getServerLocatedEnumeration();

while (sli.hasMoreElements()) {
ServerLocatedMessage m = sli.nextElement();

System.out.println(m);
}
}



The code for the same is available HERE. The zip file contains only the JAVA files. One would need to compile the same with weblogic's client jar's and then use them. Let me know if you have problems running the code or if you have any tips regarding the code.
Later...

Saturday, April 18, 2009

Aspect J Maven and Logging

At my earlier job, I had looked at Aspect Oriented programming with Spring. I used the same for transactions. I wanted to use it for Logging method enter/exits as well but abandoned the same as one would need to use Spring's native AspectJ weaving. I was not ready to take that on at the time. Another option was to use AspectJ compile-time weaving, again, not something I had the time for and simply settled on Transactional support with Spring Aspects.

So what am I upto? I wanted to play with AspectJ and how compile time weaving on maven project would help me eliminate the logging.

Enter and exit of methods are quite useful, especially when debugging. Another use case of debugging would be to trace the method arguments passed to a method. In addition, performance of profiling method executions has its benefits. In an earlier blog post I had investigated the use of Java Instrumentation to assist with profiling.

So I start by creating a logging Aspect. If the logging level is set as DEBUG, it will LOG entry and exit of a method. If the level is set as TRACE, then the time for method execution is also logged along with the method arguments to the method. The following represents the Aspect:



import org.aspectj.lang.*;
import org.apache.log4j.*;

public aspect LoggingAspect {

Logger LOG = Logger.getLogger("trace");

// Execute on all methods except this Logging Aspect

pointcut traceMethods()
: execution(* *.*(..)) && !within(LoggingAspect);

// Around each method do the following
Object around() : traceMethods() {

Signature sig = thisJoinPointStaticPart.getSignature();

String baseEnterMessage = "Enter [" + sig.getDeclaringType().getName() + "." + sig.getName() + "] ";


if (LOG.isTraceEnabled()) {
Object[] args = thisJoinPoint.getArgs();


if (args.length > 0) {
StringBuilder argsBuilder = new StringBuilder(20);

argsBuilder.append(baseEnterMessage).append(" Args :");

for (int i = 0; i < args.length; i++) {

argsBuilder.append("{Index [" + i + "], Value [" + args[i] + "]}");

if (i != args.length - 1) {

argsBuilder.append(",");
}
}
argsBuilder.append("}");


LOG.trace(argsBuilder.toString());
}
else {

LOG.trace(baseEnterMessage);
}
}
else if (LOG.isDebugEnabled()) {

// Print base debug without arguments
LOG.debug(baseEnterMessage);
}

long start = System.currentTimeMillis();


try {
Object result = proceed();
return result;

} finally {
String baseMessage = "Exiting [" + sig.getDeclaringType().getName() + "." + sig.getName() + "].";


if (LOG.isTraceEnabled()) {
// If trace is enabled, print the time for execution.
long end = System.currentTimeMillis();

LOG.trace(baseMessage + "Completed in:" + (end - start) + " millis");

} else if (LOG.isDebugEnabled()) {
LOG.debug(baseMessage);

}
}
}



Another nice use of Aspects is to be able to enforce compile time standard checks. Maybe Find Bugs has a way to already do this. Not sure. Anyway, say for example, we want to detect when there are System.out.println() calls in the code or maybe the code in question is using some Class/Framework that we do not approve of, with Aspects and compile time checking there is a nice way out. I quite liked reading the blog by Jacob Matute on the compile time enforcement of standards. What I envision is the inclusion of a simple jar in project that can/will enforce the standards one desires. I think this might be preferable over depending on IDE based configuration of plugins.

What about method invocations where checks are performed? For example in many methods we repeat code that checks if an argument is null and throws some exception? What if one could declare an annotation on the method and an aspect check for the annotation to handle the same for you? The code becomes cleaner IMO. I think Java 7 will have this feature. Till then one can create an @NotNull annotation on a parameter and have aspects handle the exception throwing part for you across your entire code base. Andy Clement's blog discusses the same very nicely.

As I used maven for all my projects, there is a plugin from codehaus, for integrating AspectJ into maven.

For the sake of demonstration, I have a simple, fictional and potentially farcical objects that would undergo the tests of the Aspects:



public class Order {
private final Long id;

private final String userName;
private final Random random = new Random();


public Order(Long id, String userName) {
this.id = id;

this.userName = userName;
}

public String getUserName() { return userName; }


public int getTotalCost(int range) {
System.out.println("Calculating Cost...");

try {
Thread.sleep(2000L);
} catch (InterruptedException e) {}

return getRandom(range);
}

private int getRandom(int range) { return random.nextInt(range);}

}



An even trivial use of the Order object which is not even a unit test is shown below:



@Test
public void testOrderModel() {

Order order = new Order(23L, "Foo");

System.out.println("Order UserName:" + order.getUserName());

System.out.println("Order Total cost:" + order.getTotalCost(113));

System.out.println(Util.concatName("Sanjay", "Acharya"));

}



When I run "mvn install", I notice the following depending on the logging level I have setup in my log4j.properties, i.e., DEBUG or TRACE:



-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running com.welflex.model.ModelTest
Enter [com.welflex.model.Order.getUserName]
Exiting [com.welflex.model.Order.getUserName].Completed in:0 millis
Order UserName:Foo
Enter [com.welflex.model.Order.getTotalCost] Args :{Index [0], Value [113]}}
Calculating Cost...
Enter [com.welflex.model.Order.getRandom] Args :{Index [0], Value [113]}}
Exiting [com.welflex.model.Order.getRandom].Completed in:0 millis
Exiting [com.welflex.model.Order.getTotalCost].Completed in:2000 millis
Order Total cost:40
Enter [com.welflex.model.Util.concatName] Args :{Index [0], Value [Sanjay]},{Index [1], Value [Acharya]}}
Exiting [com.welflex.model.Util.concatName].Completed in:1000 millis
SanjayAcharya
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.12 sec



Parting Thoughts:
I think Aspects are rather powerful. There is plugin support for Aspects as well. For cross cutting concerns, use of Aspects is the way to go IMHO. Finally, although this is a maven project using JDK 1.5, there is no reason why this applies to maven only or cannot be used on previous versions of the JDK.

A maven project that demonstrates the above is available for download HERE
. One problem that I faced was getting the maven-aspectj plugin to install correctly. If you do install the same, you might want to add the codehaus repository to the pom.xml to ensure that the plugin is downloaded. To execute the project, from your command line, issue a "mvn install". There is a log4j.properties file in the project whose logging level you can tweak to see differences.

Enjoy :-)!

Saturday, April 11, 2009

Weblogic JMS Standalone Multi-threaded Client Security

I have been building a helper library that provides some convenience functionalities and all of a sudden it started failing with authentication exceptions. Not on obtaining Jms artifacts but during the send of a message. Jeez! One part of me says let go, deal with it on Monday, the other part is hungry to find out why. No prizes for guessing which part of me won :-)

The story so far:

Weblogic client runtime stores credentials in a ThreadLocal of the Thread that creates an InitialContext to obtain a jms resource, i.e., equivalent of saying the thread that has the credentials can execute privileged operations on certain JMS resource but as other threads do not have the credentials and thus, they cannot. Weblogic's security will obtain the information from the ThreadLocal and use the credentials therein for send/receive operations.

Now, in a Jms environment, one would like to create multiple javax.jms.Session, javax.jms.MessageProducer and use multi-threading where appropriate, sharing the same JMS connection. Think of a Servlet container that has multiple threads. If Jms artifacts (Connection factories, Destinations) are obtained only once using a Singleton by one of the Container's thread's, then it is only this thread that can send/receive messages from a JMS destination if the destination has security enabled on the same. The operation of creating a Context places a javax.security.auth.Subject in the ThreadLocal of the thread creating the Context. If the context is closed, the Subject is removed.

Consider the following example where the jms artifacts are obtained in a single thread of execution and a message is sent in the same thread:



public void sameThreadCreateSend() throws Exception {

InitialContext ctx = getInitialContext();

ConnectionFactory connectionFactory = (ConnectionFactory) ctx

.lookup("Connectionfactory");
Destination destination = (Destination) ctx.lookup("test_queue");

// Closing the context causes the send authentication exception
// ctx.close();

Connection connection = connectionFactory.createConnection();

connection.start();
Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

MessageProducer producer = session.createProducer(destination);

// Fails to send message as closing of the context removes the security information

// from the Thread local. If the context is kept open, then it works great.
producer.send(session.createTextMessage());
System.out.println("Sent message when send was called on thread that created context...");

}





In the above example, the message gets sent successfully, What about a case where the Jms artifacts are created on one thread and a thread created from the creator thread tries to send a message? Will the permissions get propagated to the child thread? Look at the code below:



public void testInheritableThreadSend() throws Exception {

InitialContext ctx = getInitialContext();

// Current Thread has the security information
final ConnectionFactory connectionFactory = (ConnectionFactory) ctx

.lookup("connectionfactory");
final Destination destination = (Destination) ctx.lookup("test_queue");

// This thread will have the security as well due to Inheritable thread local
Thread t = new Thread() {

@Override public void run() {
try {

final Connection connection = connectionFactory.createConnection();
connection.start();

Session session = connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

MessageProducer producer = session.createProducer(destination);
producer.send(session.createTextMessage());

System.out.println("Sent message using inheritable thread local...");
}
catch (Exception e) {

e.printStackTrace();
}
}
};

t.start();
t.join();
}




Yes the above code sent a message successfully as the credentials in the parent are propagated to the child thread via Inheritable thread local.

Now what about a case where the creation of the Jms artifacts are created on a thread that is not the parent of the thread that sends the message using those artifacts? Look at the sample below:



public void createOnSeparateThread() throws Exception {

// First thread opens a context and had the credentials
ResourceCreator c = new ResourceCreator();

c.start();
c.join();

// Second thread is not a child of the first, thus does not have the security
NonCreatorThreadRunner n = new NonCreatorThreadRunner(c);

n.start();
n.join();
}

private class ResourceCreator extends Thread {
InitialContext ctx;

Destination destination;

Connection connection;

@Override public void run() {

try {
ctx = Example.this.getInitialContext();

connection = ((ConnectionFactory) ctx.lookup("connectionfactory"))

.createConnection();
destination = (Destination) ctx.lookup("test_queue");

}
catch (Exception e) {
e.printStackTrace();

}
}
}

private class NonCreatorThreadRunner extends Thread {

private final ResourceCreator resourceCreator;

public NonCreatorThreadRunner(ResourceCreator resourceCreator) {

this.resourceCreator = resourceCreator;
}

@Override public void run() {

try {
resourceCreator.connection.start();
Session session = resourceCreator.connection.createSession(false,
Session.AUTO_ACKNOWLEDGE);

MessageProducer producer = session.createProducer(resourceCreator.destination);

producer.send(session.createTextMessage());
System.out.println("Sent message when context was created on a separate thread...huh ???");

}
catch (Exception e) {
e.printStackTrace();

}
}
}




So the above send operation failed. Why? Because the second thread does not have the security credentials in its ThreadLocal. Now what if we could use the security information from the first thread on the second thread? We can, using weblogic security classes and using the javax.security.auth.Subject to execute a javax.security.PrivilidgedAction.




private class SubjectHoldingResourceCreator extends ResourceCreator {

Subject subject;

// Obtain the Subject from the thread
@Override public void run() {

super.run();
subject = weblogic.security.Security.getCurrentSubject();

}
}

private class SubjectUsingNonCreatorThreadRunner extends Thread {

private final SubjectHoldingResourceCreator creator;
public SubjectUsingNonCreatorThreadRunner(SubjectHoldingResourceCreator creator) {

this.creator = creator;
}

@Override public void run() {

try {
final Session session = creator.connection.createSession(false, Session.AUTO_ACKNOWLEDGE);

final MessageProducer producer = session.createProducer(creator.destination);

// Execute a secure action using the Subject from the previous thread.
weblogic.security.Security.runAs(creator.subject, new PrivilegedAction<Void>() {

public Void run() {
try {
producer.send(session.createTextMessage());

System.out.println("Sent using Subject created on separate thread....");
}
catch (JMSException e) {

e.printStackTrace();
}
return null;

}
});
} catch (Exception e) {

e.printStackTrace();
}
}
}

public void testCreationOnSeparateUsingSubject() throws Exception {
SubjectHoldingResourceCreator c = new SubjectHoldingResourceCreator();

c.start();
c.join();
SubjectUsingNonCreatorThreadRunner r = new SubjectUsingNonCreatorThreadRunner(c);

r.start();
r.join();
}




The above example worked and sent the message. I have seen others facing a similar issue when using Spring classes such as JmsTemplate and DefaultMessageListenerContainer with Weblogic where this security issue bites them. Some folks have taken the route of initializing the Context before every send/receive of a message. This is clearly not optimal as the creation of an InitialContext is not cheap and can lead to degradation of performance.

My recommendation is to use the Proxy Pattern and have proxied versions of javax.jms.MessageProducer(s) and javax.jms.MessageConsumer(s) which ensure that calls to send/receive are executed within a PrivilidgedAction.
Look at my posting on Weblogic Jms Template to see how Proxies can be used.

The javax.security.auth.Subject can be obtained once in a multi-threaded environment for a particular user authenticator (userName/pass) and then be held for use by all the threads in the environment.

public Subject createSingleSubject(String providerUrl, String userName, String passsword) {

Subject subject = new Subject();
// Weblogic env class

Environment env = new Environment();
env.setProviderUrl(providerUrl);

env.setSecurityPrincipal(userName);
env.setSecurityCredentials(password);

try {
// Weblogic Authenticate class will populate and Seal the subject
Authenticate.authenticate(env, subject);

return subject;
}
catch (LoginException e) {

throw new RuntimeException("Unable to Authenticate User", e);
}
catch (Exception e) {

throw new RuntimeException("Error authenticating user", e);
}
}



If the above method is invoked once to set up a Subject, then the same Subject can be used by Proxies to invoke PrivilidgedActions.

One thing I don't like is that on closing of the obtained Context, the ThreadLocal object is not cleared of all allocated permission related objects. It is only the Subject that is detached. I am looking for a way prevent the caching of security information in the ThreadLocal by WebLogic.

Some Links:
1. Spring JIRA
2. Weblogic Security Documentation

Wednesday, April 1, 2009

Liking Emma and FindBugs Eclipse Plugins

I have found myself liking two helpers as I continue to develop, Code Coverage and Finding Bugs. I usually use plugins in maven reporting to help me find issues and coverage information after issuing a "mvn site".

I however prefer to catch the same without having to generate the maven site though and in my IDE. I see two nice eclipse plugins for the same. For the Code Coverage part, I quite like the Emma Eclipse plugin which allows me to run my unit tests in "Coverage" mode and helps me determine the amount of code coverage my unit tests are providing. The coloring is pretty good, green for fully covered code, red for missed code and yellow for partially covered code. Trying to be green...

I have also started to use the Find Bugs eclipse plugin. Professional "Hara Kiri" here but I often get pleasantly surprised by what the utility finds for me :-). It would have been nice if the "Problems" tab in Eclipse showed the issues found by "Find Bugs" via a different icon. Also I am refraining from showing results of either plugin here for obvious reasons :-))))

Monday, March 30, 2009

Java Closure(s) but no cigar?

It has now been a year since I have been working with Java 5+ features. During development, I have welcomed auto-boxing, the concise for loop, vargs, enums and Generics. In particular the last feature has been rather appreciated as type safe programming makes life for a developer so much easier. That said, I will also state that there have been many times that I have been troubled by the wild cards in generics and erasure. However, I find the take aways I get from generics far out weigh the few times I need to stress my brain with wild cards and erasure.

Like Generics, one equally important feature that the Java language has been wanting is Closures. A JSR for closures resulted in multiple proposals. Notably,
  1. Full Closures by the BGGA proposal
  2. CICE Simplified inner classes and
  3. FCM First Class methods.
As a note a poll taken to determine which proposal Java developers would welcome led to, BGGA taking the lead. Interestingly enough, a large percentage of Java developers felt that they do not want Closures in Java. My intuition tells me that a large percentage of the latter voters are saying "I do not want BGGA closures in Java" rather than "I do not want Closures at all".

A famous debate between Neal Gafter who has come forth as the major spokesperson for the BGGA closures proposal and Josh Bloch, a supporter of the CICE proposal has ensued via presentations and blogs. The same can be googled for.

A bit about my stance, I definitely see the power of Closures. However as the saying from Spiderman, the movie goes, "With great power comes great responsibility." I want the power that Closures offer in Java, and I want an implementation that I can understand and be productive with. I want intuitive compilation errors that will immediately alert me to the problem without straining the few @deprecated Grey cells that are still struggling to function in my brain.

I want conciseness, but not at the cost of readability and maintainability. In short, I want power with clarity and brevity. One cannot always get everything they want, sad but true, however, my stance is that if I can benefit considerably most of the time, I do not mind the pain for a little of the time. Think of it like a marriage ;-)))..luckily, the wife does not read my technical blogs lol! A better parallel; (Gain Factor >>>> Pain Factor == Adoption)

One can read about what a Closure is, what a Lambda expression is and I will not spend time on the same. Primary reason being, I cannot understand the same myself :-(

What I want to do in this Blog is to understand Closures by BGGA and try for myself how hard or how easy is it to work with them. I do not use an IDE but revert to using emacs and the command line for this exercise.

I downloaded the BGGA propopal and ran some of the examples from the Java Closures Tutorial so very well explained by Zdeněk Troníček. Then I tried to apply the same to cases I have encountered before or am working with now to see for myself how easy/difficult the proposal is and problems I faced, if any.

Function Types:
I think function types would be a great addition to the language. I understand Mr.Bloch's concern that they might tend to lead to an exotic style of programming, however, I feel exotic styles of programming can even be developed with the simplest of constructs should developers choose to. It is possible for a psychotic developer as myself to take the simplest of problems and provide the most convoluted but workable solutions for the same. If I were introduced to the book Java Puzzlers prior to learning Java, I might have fled the scene. Sure, there will be puzzling cases but thats what code reviews, documentation, etc are all for after all. I am not sure I buy the argument, that "BGGA closures lend themselves to an exotic programming style". One can easily be exotic with the simplest of language constructs IMHO.

Lets look at a couple of examples, I have an eachEntry() method that runs through an java.lang.Iterable and executes the block provided. I also a have a simple filter() that can be used to filter an Iterable which delegates to the eachEntry().



public static <T> Collection<T> filter(Iterable<T> input,
{T=>boolean} filterBlock) {
Collection<T> filteredItems = new ArrayList<T>();

eachEntry(input, {T item =>
if (filterBlock.invoke(item)) filteredItems.add(item);});


return matched;
}

public static for <T, throws E extends Exception>
void eachEntry(Iterable<T> items, {T => void throws E} block) throws E {

for (T item : items) {
block.invoke(item);

}
}



An example usage of the above where Employee objects are filtered:



// Static import the utility functions, filter and eachEntry
import static com.welflex.util.Utils.*;

List<Employee> emps = ...// Get employees

// A closure that uses the Function type of eachEntry to dump out the
// information about each employee
eachEntry(emps, {Employee e =>
System.out.println("Id:" + e.getId()

+ ", Manager:"
+ e.isManager()
+ ", Salary:" + e.getSalary());});

// Filter out only employees who are managers
Collection<Employee> managers = filter(emps, {Employee e => e.isManager()});

// Filter out only managers who earn gt 60000 using a filter of filters
Collection<Employee> mgrsGt = filter(filter(emps, {Employee e => e.isManager()}), {Employee e => e.getSalary() > 60000);

// The above one re-written
mgrsGt = filter(emps, {Employee e = > e.isManager() && e.getSalary() > 60000});



I find the above pretty concise and powerful. I also find the same rather easy to read as well. So no complaints from me there. For those reading, hey I never claimed I was invincible as far as programming goes, close to invincible, yes ;-)

Now the negative, some things I could however not understand,

a. for construct did not work for me:
"for" construct did not work for me. I might not have done it correctly, but the eachEntry() that uses the "for" construct compiled just fine. Compiling the following should have worked, I am not sure what I am missing:



for eachEntry(Employee e : emps) {

System.out.println("Id:" + e.getId());

}



A compilation error leading to:
"Filter.java:20: method forEach in class Filter cannot be applied to given types
required: java.lang.Iterable
,javax.lang.function.OO,java.lang.Class
found: java.util.Collection,{T => void},java.lang.Class
forEach(orig, {T item => if (block.invoke(item)) matched.add(item);},"

The above was not very instructive for me in addressing the issue :-(

b. Static Inner Class when used in the example caused compiler error:

When my Employee class was a static inner class, I ended up with the following exception when I tried to compile the examples class, leading me take it the Employee class to its own separate compilation unit:

exception has occurred in the compiler (1.6.0_11). Please file a bug at the Java Developer Connection (http://java.sun.com/webapps/bugreport) after checking the Bug Parade for duplicates. Include your program and the following diagnostic in your report. Thank you. java.lang.NullPointerException at com.sun.tools.javac.comp.Resolve.isAccessible(Resolve.java:140)

I would like to also provide another example where I am using an ARM block to obtain a JMS Object from JNDI and subsequently closing the context. Note that I am not using actual javax.naming.Context classes but have instead modeled my own for the sake of demonstration. Without closures and using anonymous classes, I have the following:



private <T> T executeContextTask(ContextTask<T> task) throws LookupException {

Context ctx = null;
try {
ctx = new ContextImpl();
return task.executeWithContext(ctx);

} finally {
if (ctx != null) {

try {
ctx.close();
} catch (Exception e) {

//log this
}
}
}
}

private static interface ContextTask<T> {

public T executeWithContext(Context ctx) throws LookupException;

}

public ConnectionFactory getConnectionFactory(final String name)
throws LookupException {

return executeContextTask(new ContextTask<ConnectionFactory>() {
public ConnectionFactory executeWithContext(Context ctx)
throws LookupException {

ConnectionFactory c = (ConnectionFactory) ctx.lookup(name);

// Do something with Connection Factory before returning
return c;
}
});

}
....
public Destination getDestination(final String name)
throws LookupException {

... // Similar to above..except returns a Destination..//
}




The same above example when done using BGGA closures is simplified to:



private static <T, throws E extends LookupException>
T withContext({Context=>T throws E} contextTaskBlock) throws E {

Context ctx = null;
try {
ctx = new ContextImpl();

return contextTaskBlock.invoke(ctx);
} finally {

if (ctx != null) {
try {

ctx.close();
} catch (Exception e) {

// Log error
}
}
}

// Note name is no longer final, neither is there a class called
// Context Task and the anonymous class has been replaced by the
// closure.

public ConnectionFactory getConnectionFactory(String name)
throws LookupException {

return withContext({Context ctx =>
ConnectionFactory c = (ConnectionFactory) ctx.lookup(name); c});

}

.. Similar code for Destination...



Closure Conversion:
I quite like Closure Conversion. For example, I have an interface called Read that reads in an Employee Record. Assigning a closure to the interface type we have the below:

I am liking the conciseness presented by the above over writing an anonymous inner class + method definition + body (although the conversion is translated to the same).



interface Read {
public Employee read(String criteria);

}

Read readRecord = {String criteria =>
System.out.println("Criteria:" + criteria);

Random r = new Random();
new Employee(r.nextInt(5),
r.nextBoolean(),
r.nextInt(10000))};


public void closureConversion() {
Employee e = readRecord.read("select * from Emp where id=\"20\"");

System.out.println("Employee Read:" + e);
}




Arrays:
Arrays have boggle my mind in Generics, Arrays continue to boggle my mind with BGGA closures.



// Will not compile
{ => int} closures[] = new { => int}[10];

// Will not compile
{=>int} closures[] = {{=>5}, {=>6}};

// Compiles
List<{=> int}> list = new ArrayList<{=>int}>();

// But no way of
{=>int} closures[] = list.toArray({=>int} closures[0]);




Non local returns:
When returning from a Closure, should the "return" result in returning from the method itself? That is not very intuitive to me. I am of the opinion that returns from a closure, should only be local and not result in exiting the method. That is room for accidental programing with disastrous results.

Consider the following Code, what do you thing will be the result of the invocation?



static boolean testBooleanLocal(boolean input) {

{boolean =>boolean} closure = {boolean args=>

args
};

boolean result = !closure.invoke(input);

System.out.println("Test Boolean Local Result: " + result);
return result;

}

static boolean testBooleanNonLocal(boolean input) {

{boolean ==> boolean} closure = { boolean arg ==>

return arg;
};

boolean result = !closure.invoke(input);

System.out.println("Test Boolean NonLocal Result:" + result);
return result;

}

public static void main(String args[]) {

System.out.println("Local:" + testBooleanLocal(true));

System.out.println("Non Local:" + testBooleanNonLocal(true));

}



The answer is:



Test Boolean Local Result: false
Local:false
Non Local:true



What just happened to the second println()? All because I had a ==> versus a => along with a return statement?

The Syntax:
Without the assistance of an IDE, I had a few growing pains where I was accidentally doing "= >" instead of "=>", i.e, a space in between. Surely any IDE ready to support BGGA will catch the same. What about "==> "versus "=>"? Will an IDE be able to warn regarding the difference? I sure hope so :-)

Running the Examples:
As always, I am sharing my playground. For the ones interested, I tried the same using:
Download the ...., install the same and then set your PATH to include the existing JDK 1.6. For example,
export PATH=/.../jdk1.6.12/bin:$PATH. Then compile the provided code using the "javac" from the BGGA proposal, for example, "$BGGA_HOME/bin/javac -classpath .:com/welflex/utils:com/welflex/model:com/welflex/example *.java" and run the same using the similar syntax.

Parting Thoughts:
Sadness that Java does not have closures. Visual Basic developers are benefitting and thriving with the same, something I heard. Closures are a construct that has been around since ages, Java compromised at the time of inception via Anonymous classes rather then deal with the complexity of Closures, maybe a good decision for the time. However, time is now neigh for change. If Java has to survive, Closures need to enter the language. One cannot wait between JDK7 to JDK8 without knowing closures will or will not find a place in Java. Java folk are tending to gravitate toward other languages like Scala, Clojure etc. IMHO we need to stop this abandonement. My rationale for the same is (based purely on sentiments), I love Java, its been the reason for my bread and butter over the years, I love developing with it and I want to continue to develop with it. More importantly, its represents Coffee, I cannot wake up without my shot of the same. Scala or Clojure doesn't inspire me as much, maybe an alternate VM language called Nicotine could cut it and spark my interest ;-)

BGGA is not perfect. Sadly, nothing ever is. In my own humble opinion, the warring factions of Gafter/Bloch need to join forces to improve the compiler warnings, syntax and any other ambiguities in the features of BGGA present and look toward introducing it as part of JDK 8.X. Come on guys, you are super smart chaps (Phd's don't come easy) that have collectively led to the creation/enhancement of this great language, make my life easy and do not let me leave my favorite language for some pretenders to the throne :-(. If not, I propose that Gosling take the place of a dictator and get this in..I will welcome the same! Maybe IBM buying Sun will change things... Got to cut em beers when I post, for all you know I might have made a case for "No closures ever in Java via my misadventures :-)", lol!


Sample Code that I have used can be downloaded from here.

Resources:

Monday, March 2, 2009

An Example of Caching with REST using Jersey JAX-RS

One of the constraints/benefits of a RESTful architecture is the use of Cache's where possible. REST architecture gains from the use of cache's by reducing network bandwidth and unnecessary I/O. In short, caching of information has a direct impact on the scalability of the RESTful architecture.

Service Side Cache:
When a request is made to retrieve a data set, if the data set information does not change over a fixed duration as determined via non-functional requirements, then caching the data set on the Server side has the benefit of not having to suffer for example, database I/O for every client request. Network bandwidth utilization from Client to Server is of course suffered with a Server only cache.

Client Side Cache:
HTTP has a very cool construct in terms of information that a server can provide to a client saying cache or do not cache the payload provided by the server via HTTP Header attributes. In addition, client's can also utilize Conditional GET's to only obtain payload if the data has changed on the server. With Conditional GET one could also only return the changes that have occurred and a Client could easily compute the delta and update its local cached copy of the data.

I wonder how many organizations utilize these features provided by HTTP. The Server side cache can easily be accommodated via using Squid or some other tool.

On the Client side, now thats a bit of discussion. As part of HTTP response, a server can let the client know whether or not to Cache the provided representation via the "Expires" HTTP header attribute. As an example, when a Product Client or Web Browser, requests a list of Products from the server, if the server knows that the Product data will not be refreshed for some time, it can inform the client to cache the payload representing Products for the duration until expiration. What I like about this control is that the Server and not the Client is instructing the duration for which the data is valid and can be cached. Client's in a Client-Server environment that decide to cache data based of non-functional requirements is a bad way to cache IMO. The duration logic should IMO emanate from the service or server.

Using the "Expires" header the server can tell the client to cache the data till a particular date or provide a time duration to cache. The former can be a problem if the client and server clocks are not in sync. For example, the server tells the client to cache a piece of data till Dec 20th, 2012. However, the Client clock is 10 mins behind the server. So although the data on the client has not expired, the server data has. For this reason, setting a duration for expiration via a time duration such as 10 mins will allow both Client/Server to be in sync regarding expiration of the cache.

What about a case when caching is recommended on the client but there is a certain amount of volatility involved with the data. For example, lets say we have an OrderClient that GETs order information about a particular order from the server. The Order information could potentially be updated by a user subsequently, for example, adding a new line item to the Order. In such a case one could avail the Conditional GET features of HTTP to obtain new payload only if the data cached by the Client is stale. The server determines whether the data has changed between the last time the client requested the payload and either provides the entire data or responds back with a HTTP status of 304, indicating UN-Modified payload. The client in turn can in turn as a result of a 304 returned from the server, respond the consumer with the data it has previously cached. This reduces the amount of data transferred between client and server and thus alleviates network bandwidth utilization. Conditional HTTP GET can be availed using either Etags or Last-Modified header attributes.

As an example of the above, let us look at a Jersey, JAX-RS example. In the example, we have two clients, a ProductClient that obtains information about Products and an OrderClient used to manage the life cycle of an Order. The Product Client will cache the Products until the time has come to re-fetch the products due to expiration while the OrderClient will cache the payload obtained an issue a Conditional GET to only obtain the payload if the data has changed on the server since its last request.

The ProductsResource as shown below for the sake of demonstration, sets the Products to expire 3 seconds after its invocation:
@GET @Produces("application/json") public Response getProducts() {

...
ProductListDto productListDto = new ProductListDto(productDtos);
Response.ResponseBuilder response = Response.ok(productListDto).type(MediaType.APPLICATION_JSON);

// Expires 3 seconds from now..this would be ideally based
// of some pre-determined non-functional requirement.
Date expirationDate = new Date(System.currentTimeMillis() + 3000);

response.expires(expirationDate);

return response.build();

}


The OrderResource on the other hand based of an etag determines if the order has been modified since the last GET request by the client and returns back a status of 304 or the entire order body as shown below:

@GET
@Produces("application/xml")

public Response getOrder(@Context HttpHeaders hh, @Context Request request) throws OrderNotFoundException {

Order order = orderService.getOrder(orderId);

LOG.debug("Checking if there an Etag and whether there is a change in the order...");

EntityTag etag = computeEtagForOrder(order);
Response.ResponseBuilder responseBuilder = request.evaluatePreconditions(etag);

if (responseBuilder != null) {
// Etag match

LOG.debug("Order has not changed..returning unmodified response code");
return responseBuilder.build();

}

LOG.debug("Returning full Order to the Client");
OrderDto orderDto = (OrderDto) beanMapper.map(order, OrderDto.class);

responseBuilder = Response.ok(orderDto).tag(etag);

return responseBuilder.build();
}



From the Perspective of the ProductClient, it looks to see whether the cached data has expired before issuing a new request to the server as shown below:

public ProductListDto getProducts() {

// Key into the cache
String path = resource.getURI().getPath();

CacheEntry entry = CacheManager.get(path);
ProductListDto productList = null;


if (entry != null) {
LOG.debug("Product Entry in cache is not null...checking expiration date..");

Date cacheTillDate = entry.getCacheTillDate();
Date now = new Date();

if (now.before(cacheTillDate)) {
LOG.debug("Product List is not stale..using cached value");

productList = (ProductListDto) entry.getObject();
} else {

LOG.debug("Product List is stale..will request server for new Product List..");
}
}

if (productList == null) {

LOG.debug("Fetching Product List from Service...");
ClientResponse response = resource.accept(MediaType.APPLICATION_JSON).get(ClientResponse.class);

if (response.getResponseStatus().equals(Status.OK)) {

productList = response.getEntity(ProductListDto.class);
String cacheDate = response.getMetadata().getFirst("Expires");

if (cacheDate != null) {
Date ccDate;

try {
ccDate = DATE_FORMAT.parse(cacheDate);

entry = new CacheEntry(productList, null, ccDate);

CacheManager.cache(path, entry);
}
catch (ParseException e) {

LOG.error("Error Parsing returned cache date..no caching will occur", e);
}
}

} else {
throw new RuntimeException("Error Getting Products....");

}
}

return productList;
}

The Order Client on the other hand uses the etag and sends that as part of every request to the server as shown below:

public OrderDto getOrder(Long orderId) throws OrderNotFoundException, IOException {
    try {
String path = resource.path(orderId.toString()).getURI().getPath();

CacheEntry entry = CacheManager.get(path);
Builder wr = resource.path(orderId.toString()).accept(MediaType.APPLICATION_XML);

if (entry != null && entry.getEtag() != null) {

// Set the etag
wr.header("If-None-Match", entry.getEtag().getValue());

}

ClientResponse response = wr.get(ClientResponse.class);

if (response.getResponseStatus().equals(Status.NOT_MODIFIED)) {

LOG.debug("Order has not been modified..returning Cached Order...");
return (OrderDto) entry.getObject();

}
else if (response.getResponseStatus().equals(Status.OK)) {

LOG.debug("Obtained full Order from Service...Caching it..");
OrderDto dto = response.getEntity(OrderDto.class);

CacheManager.cache(path, new CacheEntry(dto, response.getEntityTag(), null));

return dto;
}
else {
LOG.debug("Order not found on server...removing from cache");

CacheManager.remove(path);
throw new UniformInterfaceException(response);

}
}
catch (UniformInterfaceException e) {

if (e.getResponse().getStatus() == Status.NOT_FOUND.getStatusCode()) {

throw new OrderNotFoundException(e.getResponse().getEntity(String.class));

}
throw new RuntimeException(e);
}

}


Seeing the above in action, for the Products; we obtain Products in the first call, this should result in caching of the same, the second request executed immediately after the first should obtain the cached Products. Sleeping for sometime will allow the data to become stale and a subsequent request should re-fetch the data. The logs when the tests are run look like:

1 Request, cache Products:
20:37:33 DEBUG - com.welflex.client.CacheManager.cache(14) | Caching Object with key [/IntegrationTest/products]

2. Request, Product Cache still good:
20:37:33 DEBUG - com.welflex.client.CacheManager.get(19) | Getting Object from Cache for Key:/IntegrationTest/products
20:37:33 DEBUG - com.welflex.client.ProductClientImpl.getProducts(49) | Product Entry in cache is not null...checking cache till date
20:37:33 DEBUG - com.welflex.client.ProductClientImpl.getProducts(54) | Product List is not stale..using cached value

3. Request, Products have expired:
20:37:43 DEBUG - com.welflex.client.CacheManager.get(19) | Getting Object from Cache for Key:/IntegrationTest/products
20:37:43 DEBUG - com.welflex.client.CacheManager.get(21) | Object in Cache for Key [/IntegrationTest/products] is :com.welflex.client.CacheEntry@1bf3d87
20:37:43 DEBUG - com.welflex.client.ProductClientImpl.getProducts(49) | Product Entry in cache is not null...checking cache till date
20:37:43 DEBUG - com.welflex.client.ProductClientImpl.getProducts(57) | Product List is stale..will request server for new Product List..
20:37:43 DEBUG - com.welflex.client.ProductClientImpl.getProducts(62) | Fetching Product List from Service...


From the Order Client Perspective, the first request to obtain the order results in the Order being cached with the etag. When a subsequent request is sent, the server only responds back with a status of 304, i.e, un-modified and then the Order Client responds back with the cached copy. After this second request, the order is updated and the etag is no longer valid therefore a subsequent GET of the order results in the full order being fetched and re-cached as shown below:

1. First time Order is retreived, Order is cached:
Retrieving the order...
22:33:13 DEBUG - com.welflex.client.CacheManager.get(19) | Getting Object from Cache for Key:/IntegrationTest/orders/3443274629940897628
22:33:13 DEBUG - com.welflex.client.OrderClientImpl.getOrder(68) | Obtained full Order from Service...Caching it..

2. Second Request, Order not changed on Server, 304 returned to Client:
22:33:13 DEBUG - com.welflex.order.rest.resource.OrderResource.getOrder(68) | Order Resource 22:33:13 DEBUG - com.welflex.order.rest.resource.OrderResource.getOrder(79) | Order has not changed..returning unmodified response code
22:33:13 DEBUG - com.welflex.client.OrderClientImpl.getOrder(64) | Order has not been modified..returning Cached Order...

3. Issue a PUT to update the Order, thus changing it:
Updating the order...
22:33:13 DEBUG - com.welflex.order.rest.resource.OrderResource.updateOrder(106) | Enter Update Order, Id=3443274629940897628

4. Retrieve the Order the etag should no longer be valid:
Retrieving the order..should not obtained cached copy...
22:33:13 DEBUG - com.welflex.order.rest.resource.OrderResource.getOrder(73) | Checking if there an Etag and whether there is a change in the order...
22:33:13 DEBUG - com.welflex.order.rest.resource.OrderResource.getOrder(83) | Returning full Order to the Client
22:33:13 DEBUG - com.welflex.client.OrderClientImpl.getOrder(68) | Obtained full Order from Service...Caching it..


Attached HERE is the sample Maven Jersey JAX-RS sample that will allow you to witness the above. The caching implemented is primitive at best and the attached code is only an EXAMPLE. One could easily delegate the caching to some caching framework such as ehcache, oscache, jcs etc. One can even potentially get more exotic and think of Aspects that will intercept calls to GET and transparently provide the caching.

To execute the example, from the command line, at the root of the project, execute a "mvn install". Note that one needs JDK5.X+ in order to execute the code.

Caching is a very critical feature of REST. Without using the same is like saying one is doing RES without the T. As always, if a reader of this blog has any comments, I'd appreciate the same. If I am wrong, I would like comments on the same as that will help me improve..or else forever hold ur breath :-) If you cannot run the example, ping me...

Sub-Resources with Jersey,Spring, jax-rs

A simple example on how to use Sub-Resources with Jersey and Spring.

For the sake of discussion, consider a Resource /orders. One would typically POST to the resource to create an Order, update the Order at the Resource /orders/{id}, GET the Resource from the URI /orders/{id} and also delete the order at the Resource /orders/{id}.

One can therefore state that /orders/{id} is a Sub-Resource of /orders.

In jax-rs, sub-resources are obtained from the parent resource. A parent resource will not define a HTTP method for the sub-resource, but instead simple have a @Path annotation indicating a sub-resource.

For example,


@Path("/orders")
@Component

@Scope("singleton")
public class OrdersResource {
@Context

private ResourceContext resourceContext;

@Path("/{id}")
public OrderResource subResource(@PathParam("id") String id) {

OrderResource resource = resourceContext.getResource(OrderResource.class);

resource.setOrderId(Long.parseLong(id));

return resource;

}
...
}



In the above example, we have an OrdersResource that services the URI pattern of /orders. Calls to /orders/{id} are delegated to a sub-resource of the OrdersResource called OrderResource as shown below:


@PerRequest
@Component
@Scope("prototype")

public class OrderResource {
private Long orderId;
// Jersey injected

@Context
private ResourceContext resourceContext;

// Spring injected
@Autowired

private OrderService orderService;
@Autowired
private MapperIF beanMapper;

....
public void setOrderId(Long orderId) {
this.orderId = orderId;

}

// @Context are Jersey injected

@GET
@Produces("application/xml")

public Response getOrder(@Context HttpHeaders hh,
@Context Request request) { ... }

@Consumes("application/xml")
@PUT
public void updateOrder(@PathParam("id") String id, OrderDto orderDto) {

....
}

@DELETE
public void deleteOrder(@PathParam("id") String id) {

....
}

@Path("/lineItems")
public LineItemListDto getLineItems() throws OrderNotFoundException {

.....
}
}



Both the above mentioned Resources are beautifully managed by Spring and Jersey interoperability. Jersey Resources are injected by Jersey and Spring resources are injected by Spring auto-wiring. So cool!!!!

As shown above the OrderResource is following a Prototype pattern and is obtained via a called to resourceContext.getResource(OrderResource.class).

I quite like the fact that the OrderResource is declared as sub-resource of the OrdersResource and the creation of the Sub-Resource via the prototype pattern matches so beautifully within jax-rs. It becomes a trivial exercise to trace the flow of a uri-call from the parent resource to sub-resources.