Search This Blog

Loading...

Sunday, September 14, 2014

Dynamic Service Discovery with Apache Curator

Service Discovery is a core part of most Service Oriented Architectures in particular and Distributed Network Component Architectures in general.

Traditional usages to find the location of a service have been to use Local file system, Shared File System, Database or some 3rd party Service Registry that statically define service locations. In many cases, one knows exactly the number of services that will be used and their deployment locations thus leading to a statically defined registry as an ideal candidate for discovery. In larger scale systems statically defined locations can become harder to manage as -
  • Number of services in the ecosystem grow (micro-services)
  • Number of deployment 'environments' grow - Test Environment 1, Test Enviroment 2...etc
  • Elasticity scalability becomes desirable - More important with Cloud based environments
  • Multiple version support of services need to be supported
Enter the concept of Dynamic Service Registration and Discovery. Services on becoming available register themselves as a member ready to participate with some 'registry'. Service Consumers are notified of the availability of the new service and avail its usage. If the service terminates for whatever reason, the Service Consumer is notified of the same and can stop availing its usage. This is not a novel concept and has likely been addressed in different ways.

Any Dynamic Service Registration System presents the following challenges -
  • Availability - The Service Registry can become the SPOF (Single Point Of Failure) for an entire ecosystem
  • Reliability - Keeping the registry and thus the service consumers in sync with which service instance is currently available. A service might go down and if the registry and service consumer are not notified immediately, service calls might fail.
  • Generality - Registry that supports multiple operating systems and technologies.
Again, the above is solvable by different ways. The benefit we have is that there are a number of tools that intrinsically support the requirements of a dynamic registry such as ZooKeeper, Netflix Eureka, Linked In's Dynamic Discovery, Doozer etc.  Jason Wilder has an excellent BLOG on Open Source Service Discovery and a comparison of popular open source service registries.

I have been looking into ZooKeeper and thus will share a few of my findings on using it but before I do the same, I would like to take a moment to discuss Load Balancing.

In standard H/A architectures a Load Balancer serves to balance traffic across the different service instances as shown below -




The same works well but poses the following challenges -
  • A new Load Balancer for every service and every environment
  • An additional hop between a Service Consumer and Service
  • Adding/Removing services per need and time to do the same
  • Intelligent Routing and Supporting Service Degradation
A Dynamic Service Registry could help with the mentioned concerns  where different instances of Service C register themselves with ZooKeeper and Service A and Service B on obtaining locations of Service C execute calls directly to them  -




ZooKeeper's Ephermal nodes serve well to maintain registered service instances as an Ephermal node only exists as long the client is alive, so if a client dies, the Ephermal node is deleted.  It is possible to use ZooKeeper directly and develop code that will register services, discover registered service etc. However, there is a library called Apache Curator that was initially created by Netflix to make writing ZooKeeper based application easier and more reliable. It was then incorporated as an Apache Project. Curator provides 'recipes' to common use cases one would use ZooKeeper for, i.e., Leader Election, Queue, Semaphore, Distributed Locks etc. In addition to these 'recipes', it also provides a mechanism for Service Registration and Discovery.

The remainder of this BLOG will look at providing a simple example that demonstrates the use of Apache Curator in the context of a simple JAX-RS web service that gets registered and is subsequently dynamically discovered by clients. For the sake of the example, I will continue to use the Notes JAX-RS Web Service that I have used on previous posts.

Service Registration -

In the example domain, all services will be registered under the ZooKeeper path "/services".  When the Notes Service starts, it registers itself with ZooKeeper as an ephermal node in the path "/services/notes/XXXXXXXX", where XXX indicates the ephermal instance.

A simple class called ServiceRegistrar is responsible for registering the Notes Service:
public class ServiceRegistrarImpl implements ServiceRegistrar {
   ...
  private static final String BASE_PATH = "/services";
  private static final String SERVICE_NAME = "notes";  

  private final CuratorFramework client; // A Curator Client
  
  private ServiceDiscovery<InstanceDetails> serviceDiscovery;
  
  // This instance of the service
  private ServiceInstance<InstanceDetails> thisInstance;
  
  private final JsonInstanceSerializer<InstanceDetails> serializer; 
  
  public ServiceRegistrarImpl(CuratorFramework client, int servicePort) throws UnknownHostException, Exception {
    this.client = client;
    serializer = new JsonInstanceSerializer<>(InstanceDetails.class); // Payload Serializer
    
    UriSpec uriSpec = new UriSpec("{scheme}://{address}:{port}"); // Scheme, address and port
      
    thisInstance = ServiceInstance.<InstanceDetails>builder().name(SERVICE_NAME)
      .uriSpec(uriSpec)
      .address(InetAddress.getLocalHost().getHostAddress()) // Service information 
      .payload(new InstanceDetails()).port(servicePort) // Port and payload
      .build(); // this instance definition
  }

  @Override 
  public void registerService() throws Exception {
    serviceDiscovery = ServiceDiscoveryBuilder.builder(InstanceDetails.class)
        .client(client)
        .basePath(BASE_PATH).serializer(serializer).thisInstance(thisInstance)
        .build();
    serviceDiscovery.start(); // Registers this service
  }
  
  @Override 
  public void close() throws IOException {
    try {
      serviceDiscovery.close();
    }
    catch (Exception e) {
      LOG.info("Error Closing Discovery", e);
    }
  }
}
A Servlet Listener is defined whose responsibility is to start the registrar -
public class ServiceRegistrarListener implements ServletContextListener {
  private static final Logger LOG = Logger.getLogger(ServiceRegistrarListener.class);
  
  @Override
  public void contextDestroyed(ServletContextEvent sc) {  
  }

  @Override
  public void contextInitialized(ServletContextEvent sc) {
    try {
      WebApplicationContextUtils.getRequiredWebApplicationContext(sc.getServletContext())
      .getBean(ServiceRegistrar.class).registerService();
    }
    catch (Exception e) {
      LOG.error("Error Registering Service", e);
      throw new RuntimeException("Exception Registering Service", e);
    }   
  }
}

Service Discovery -
A Simple Discoverer class is shown below which uses the Curator library in obtaining instances of registered Notes Services. Curator provides a ServiceDiscovery class that is made aware of a Service Registry. A ServiceProvider is then used to obtain registered service instances for a particular service.
public class ServiceDiscoverer {
  private static final Logger LOG = Logger.getLogger(ServiceDiscovery.class);
  private static final String BASE_PATH = "/services";

  private final CuratorFramework curatorClient;

  private final ServiceDiscovery<InstanceDetails> serviceDiscovery;

  private final ServiceProvider<InstanceDetails> serviceProvider;

  private final JsonInstanceSerializer<InstanceDetails> serializer;

  private final List<Closeable> closeAbles = Lists.newArrayList();

  public ServiceDiscoverer(String zookeeperAddress, String serviceName) throws Exception {
    curatorClient = CuratorFrameworkFactory.newClient(zookeeperAddress,
      new ExponentialBackoffRetry(1000, 3)); // Ideally this would be injected

    serializer = new JsonInstanceSerializer<>(InstanceDetails.class); // Payload Serializer

    serviceDiscovery = ServiceDiscoveryBuilder.builder(InstanceDetails.class).client(curatorClient)
        .basePath(BASE_PATH).serializer(serializer).build(); // Service Discovery

    serviceProvider = serviceDiscovery.serviceProviderBuilder().serviceName(serviceName).build(); // Service Provider for a particular service
  }

  public void start() {
    try {
      curatorClient.start();
      closeAbles.add(curatorClient);
      serviceDiscovery.start();
      closeAbles.add(0, serviceDiscovery);
      serviceProvider.start();
      closeAbles.add(0, serviceProvider);
    }
    catch (Exception e) {
      throw new RuntimeException("Error starting Service Discoverer", e);
    }
  }

  public void close() {
    for (Closeable closeable : closeAbles) {
      // Close all
      ...
    }
  }
  
  public ServiceInstance<InstanceDetails> getServiceUrl() {
    try {
      return instance = serviceProvider.getInstance();
    }
    catch (Exception e) {
      throw new RuntimeException("Error obtaining service url", e);
    }
  }
}
The ServiceProvider is used in the Notes Service Client as follows:
public class NotesClientImpl implements NotesClient {
  private static final Logger LOG = Logger.getLogger(NotesClientImpl.class);

  public static final String SERVICE_NAME = "notes";

  private final Client webServiceClient; // JAX-RS Client

  private final ServiceDiscoverer serviceDiscoverer; // Service Disoverer

  /**
   * @param getNotesServerUrl() Server URI
   * @throws Exception
   */
  public NotesClientImpl(String zookeeperAddress) throws Exception {
    serviceDiscoverer = new ServiceDiscoverer(zookeeperAddress, SERVICE_NAME);
    serviceDiscoverer.start();

    ClientConfig config = new ClientConfig();
    webServiceClient = ClientBuilder.newClient(config);
  }
  

  @Override
  public NoteResult create(Note note) {
    ServiceInstance<InstanceDetails> instance = serviceDiscoverer.getServiceInstance();
    
    try {
      return webServiceClient
          .target(UriBuilder.fromUri(instance.buildUriSpec()).path("/notes").build())
          .request(MediaType.APPLICATION_XML)
          .post(Entity.entity(note, MediaType.APPLICATION_XML_TYPE), NoteResult.class);
    }
    catch (ProcessingException e) {
      // If a ProcessingException occurs, the current ServiceInstance is marked as down
      serviceDiscoverer.noteError(instance);
      throw e;
    }
  }
In the above code, a ServiceInstance is obtained from the ServiceDiscoverer. It is important to 'not' locally hold onto the instance but use it during the processing of a call and then let go of it. If a javax.ws.rs.ProcessingException occurs say due to a Service Instance dying abruptly during processing for the request, then, notifying Curator to not use that instance for a certain period of time is the recommended direction. There is a time out delay between when the service instance dies and ZooKeeper times out the ephermal node associated with the service. So unless the instance is marked as being in an error state on the service client, the ServiceDiscoverer would continue to provide the downed service instance for usage.

Integration Test -

A simple Integration Test has been shown where a number of Notes Server instances are started up and they register themselves with a test ZooKeeper instance. The Notes Client is able to round robin to each of those instances. Each of the servers are designed to randomly die after a period of time leading to them being un-registered .When a Notes Server Instance goes down, it will no longer be invoked as part of the test. There is a Notes Server started that runs for a long time and acts as a fall back for demonstration completeness.
public class SimpleIntegrationTest {
  private TestingServer testServer; // ZooKeeper Server

  private NotesServer longRunningNotesServer; // A Notes Server that does not die for the demo
  
  private static final int MAX_SERVERS = 10; // Max Servers to use
 
  @Before
  public void setUp() throws Exception {
    testServer = new TestingServer(9001);   // Start the Curator ZooKeeper Test Server
  }  

  @After
  public void tearDown() throws IOException {...}

  @Test
  public void integration() throws Exception {
    CyclicBarrier barrier = new CyclicBarrier(MAX_SERVERS + 1);
    
    for (Integer port : ports()) { // Start Instances of Notes Server on different ports
      new NotesServer(port, barrier).start(); // Start an Instance of the Notes Server at the provided port
      Thread.sleep(1000);
    }    
    
    longRunningNotesServer = new NotesServer(9210, 500000L, null); // Start a Long Running Notes Server
    longRunningNotesServer.start();
    
    barrier.await(); // Wait for all Servers start
    
    NotesClient notesClient = new NotesClientImpl(testServer.getConnectString());
    List<NoteResult> noteResults = Lists.newArrayList();
    
    int i = 0;
    int maxNotes = 10000;
    
    while (i < maxNotes) {
      try {
        Note note = new Note("" + i, "" + i);
        NoteResult result = notesClient.create(note);
        noteResults.add(result);      
        i++;
      }
      catch (Exception e) {
        e.printStackTrace(); // If an error occurs on one instance, try again
      }
    }   

    // Ensure that all Notes Servers have been utilized
    assertEquals(MAX_SERVERS +1, notesClient.getServiceInstancesUsed().size());
  }
      
  private static Set<Integer> ports() {
    // Provide a unique set of ports for running multiple Notes Server
  }
  ....
}
The example itself can be downloaded from HERE. Execute the project using:
>mvn install

Thoughts -
  • ZooKeeper is CP so in the event of Partition, writes can suffer.
  • ZooKeeper is SPOF of the ecosystem. Should have a contingency strategy.
    • Linked In's Dynamic Discovery Architecture addresses something like this but has a lot of moving parts
    • Simplicity of a Load Balancer makes one think YAGNI
      • Tools like Puppet Labs might make a lesser case for burden of standing up environments as well
  • Service Discovery allow clients to be 'smart'. Support for Service Degradation, alerting, and monitoring kind of go hand in hand.
  • Service Registration and De-Registration can be smartly built as well. For example, a Service could itself determine it's not healthy for whatever reason and de-registerer itself from the registry.
  • Supporting multiple environments where an ecosystem is self discoverable is simpler than having to provide overrides across different environments.
  • Managing  downed nodes with Curator needs to be handled carefully. Idempotence is your friend here.
  • Curator can definitely help with intelligent load balancing and different Load Balancing Strategies
    • Round Robin
    • Random
    • Custom one that involves Service Degradation
  • Curator provides a REST service that helps non-java clients wishing to participate in service registration and discovery.
  • If Netflix, Linked In and Amazon do something like this, it warrants looking into :-)


Friday, August 22, 2014

JVM Native memory leak

The Problem


There are times I learn something new of something that has been known since ages. Today was one of those times. Had a  Java Virtual Machine (JVM) whose heap usage looked great but the JVM process ended up growing native memory and finally died with an exception stating unable to allocate native memory. One such example might look like:
java.lang.OutOfMemoryError at java.util.zip.Inflater.init(Native Method) at java.util.zip.Inflater.(Inflater.java:101) at java.util.zip.ZipFile.getInflater(ZipFile.java:450) 

Or
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 184 bytes for AllocateHeap
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.

The application in question that manifested the above is a simple jersey web service that serves up and consumes JSON using JSON JAXB (Clue...). Initial interrogation yielded the following:
  • Thread usage is normal (-Xss settings has no benefit)
  •  –XXMaxDirectMemorySize - No difference
  • Heap usage is healthy
  • Heap dumps look healthy - no leaks
  • Perm Gen is looking good
  • Native file descriptors are within normal usage limits
  • java.net.Socket creation did not appear to cause issues
  • JDK version latest (1.7.X)
  • Happens on Virtual machines or stand alone boxes
Some Stack Overflow and other assistance:
  • Using the G1 Garbage collector
  • Or Setting Xms and Xmx to smaller values, i.e., lowering heap size
Right, so using G1 Garbage collector worked, or setting max heap to a lower value also worked? WTF!

What happened?


Creation of  a JAXB Context on every request/response. The application in question was not caching JAXB Contexts but was creating them on every request/response. Initial thought is, yeah, that is a performance problem, but why will it count towards native process memory growth? Consider the following benign code:


public class SimpleTest extends Thread { 
  public void run() {
    while (!Thread.currentThread().isInterrupted()) {
      JSONJAXBContext context = new JSONJAXBContext(SomeDTOClass.class); // From jersey-json.jar
    }
  }

  public static void main(String args[]) {
    new SimpleTest().start();
  }
} 

Well, the problem is that JAXB Contexts indirectly involve in the invoking of native memory allocations. How does it do it and what happens underneath? Well, that is an exercise for the reader to figure out and comment on this BLOG with findings ;-).

The root cause can be dialed down to the following issue posted: http://bugs.java.com/view_bug.do?bug_id=4797189

public class Bug {
  public static void main( String args[] ) {
    while ( true ) {
      /* If ANY of these two lines is not commented, the JVM runs out of memory */
      final Deflater deflater = new Deflater( 9, true );
      final Inflater inflater = new Inflater( true );
    }
 }
}

The same code that might work without native OOM:
public class Bug {
  public static void main( String args[] ) {
    int count = 0;
    while ( true ) {
      final Deflater deflater = new Deflater( 9, true );
      final Inflater inflater = new Inflater( true );
      i++;
      if (i % 100 == 0) {
       System.gc(); // Don't, please don't do this!
      }
    }
 }
}

As shown above, you might allocate 'small' objects in the JVM but underneath they might end up allocating native memory for their operations. The intention is noble, that when the objects are finalized, the native operating system memory will be de-allocated as well. However, the freeing of the native memory is contingent on the Garbage collector running and leading to finalization of these 'small' objects. Its a matter of relative pace, the native memory is growing far faster compared to JVM heap and the lack of Garbage collection occurring is what leads to running out of native space. When using the G1 collector or lowering max heap settings, garbage collection occurs more often, thus preventing the native heap from exploding over time.

Peace!

Sunday, April 20, 2014

jersey/jaxrs 2.X example using spring JavaConfig and spring security

Introduction


I had blogged previously on using jersey/JAX-RS 2.0 when it was in a pre-release. Since then Jersey/JAX-RS 2.0 has released and undergone a few versions. As I have recently been working with the jersey 2.7-spring integration and spring security and figured I'd share an example.

Requirements


As is quite standard with me, I like to 'manufacture' requirements. The application being developed is a Notes application that allows one to perform CRUD operations for simple notes. So, for the Web Service being created, I have the following requirements:
  1. No XML- You get type safety during refactoring
  2. Spring Dependency Injection with Java Config. The official jersey/spring3 example is very nice but does not demonstrate Java Config usage.
  3. Jersey Should manage Resource classes an have Spring service objects injected into them
  4. Security should be enabled via Spring Security. Only users with the ROLE 'notesuser' should be able to create a note.

The Project

Look Ma, no XML

This example will use Servlet 3.X specification, so we eliminate web.xml. There are many ways one can eliminate XML using Servlet 3.X, but this example focuses on the usage of javax.servlet.ServletContainerInitializer. With a ServletContainerInitializer one can programatically register Servlets, Filters and Listeners thus giving you type safety that can be useful during refactoring. As we are in Spring Land, rather than utilize ServletContainerInitializer, an extension of Spring's WebApplicationInitializer is used as shown below:
@Order(Ordered.HIGHEST_PRECEDENCE) 
public class NotesApplicationInitializer implements WebApplicationInitializer {

  private static final String SPRING_SECURITY_FILTER_NAME = "springSecurityFilterChain";
  
  @Override public void onStartup(ServletContext ctx) throws ServletException {
    // Listeners
    ctx.addListener(ContextLoaderListener.class);

    ctx.setInitParameter(ContextLoader.CONTEXT_CLASS_PARAM,
      AnnotationConfigWebApplicationContext.class.getName());
    ctx.setInitParameter(ContextLoader.CONFIG_LOCATION_PARAM, SpringConfig.class.getName());

    // Register Spring Security Filter chain
    ctx.addFilter(SPRING_SECURITY_FILTER_NAME, DelegatingFilterProxy.class)
        .addMappingForUrlPatterns(
          EnumSet.<DispatcherType> of(DispatcherType.REQUEST, DispatcherType.FORWARD), false, "/*");

    // Register Jersey 2.0 servlet
    ServletRegistration.Dynamic servletRegistration = ctx.addServlet("Notes",
      ServletContainer.class.getName());
    
    servletRegistration.addMapping("/*");
    servletRegistration.setLoadOnStartup(1);
    servletRegistration.setInitParameter("javax.ws.rs.Application", NotesApplication.class.getName());
  }
}
The WebApplicationInitializer does the following:
  1. Sets up a Spring ContextLoaderListener
  2. Tells the Context loader the location of the Spring Java Config that is to be used for Services
  3. Registers the Spring Security Filter chain
  4. Registers the Jersey Servlet Container by providing it the NotesApplication as the class to use for Resource and Provider management
Of note is the fact that the NotesApplicationInitializer has been set to highest precedence as that will ensure that it is executed before any other WebApplicationInitializer provided by accompanying jars. If for example, the SpringWebApplicationInitializer from jerseyspring3.jar gets loaded, then it attempts to find a spring  applicationContext.xml and will fail as our example does not use spring xml style bean definitions.

Spring Dependency Injection

In conjunction with the registration of the Spring Java Config in the WebApplicationInitializer shown above, the Java Config enables the Services and Spring Security filter chain via the following:
@Configuration
@EnableNotesService
@EnableNotesSecurity
public class SpringConfig {
}
I have not delved into the details of the @Enable annotations but they are in line with Spring's style to import dependencies. For the scope of this example, you can assume that @EnableNotesSecurity results in the importing of the notes Java services and @EnableNotesSecurity imports the security configuration.

Jersey Should Manage the Resource Classes

I wanted all my JAX-RS resources managed by Jersey and not Spring. I did not want to annotate my JAX-RS classes with @Component + Classpath-scanning and/or have them defined in a Java Config which would then result in Spring managing them. All the Resources and relevant providers are registered with Jersey by extending the javax.ws.rs.core.Application class as shown below:
public class NotesApplication extends Application {
  
  @Override
  public Set<Class<?>> getClasses() {
    return ImmutableSet.<Class<?>>of(NotesResource.class, 
      HealthResource.class,
      NoteNotFoundExceptionMapper.class, LoggingFilter.class, AccessDeniedExeptionMapper.class);
  }
}
Note that the 'service' java classes, such as NotesService.java (managed by Spring) are made available via dependency injection into the NotesResource via Jersey's Spring-HK2 bridge.

Security Should be enabled Via Spring Security

For this example,I have very simple Spring Security filter chain set up that ensures a POST to create a Note can only be done by a user who has the ROLE 'notesuser' provided in a HTTP header while making the call. It is trivial, but hey, this is an example :-) The goal was to demonstrate the use of a @PreAuthorize annotation on the create a Note and how it can be made to work when Jersey manages the resource and not spring. The Notes resource create method looks like the following:
@Path("/notes")
@Produces({ MediaType.APPLICATION_XML })
@Consumes({ MediaType.APPLICATION_XML })
public class NotesResource {
  // Jersey object injection
  private final UriInfo uriInfo;
  
  // Spring object injection
  private final NotesService notesService;
  
  // Note that UriInfo is obtained from Jersey but the NotesService is a spring dependency
  @Inject
  public NotesResource(@Context UriInfo uriInfo, NotesService notesService) {
    this.uriInfo = uriInfo;
    this.notesService = notesService;
  }
  
  @POST
  @Loggable
  @PreAuthorize("hasRole('notesuser')") // Pre Authorize role that allows only notesuser to create the note
  public Response create(Note note) {
    LOG.debug("Creating a note:" + note);
    NoteResult result = createNote(note);
    
    return Response.created(result.getLocation()).entity(result).build();
  }
  .....
}
As the NotesResource is managed by Jersey, the only way I could figure to enforce the @PreAuthorize annotation was via AspectJ weaving of the resource class. The same is accomplished by adding the aspectJ maven plugin which is used during the build process to enhance the JAX-RS Resource classes:
 <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>aspectj-maven-plugin</artifactId>
    <version>1.6</version>
    <configuration>
    ...
    <sources>
      <source>
        <basedir>${basedir}/src/main/java/com/welflex/notes/rest/resource</basedir>
        <includes>
           <include>**/*Resource.java</include>
        </includes>
      </source>
    </sources>
    ....
    <aspectLibraries>
      <aspectLibrary>
        <groupId>org.springframework.security</groupId>
          <artifactId>spring-security-aspects</artifactId>
      </aspectLibrary>
   </aspectLibraries>
   </configuration>
  <executions>
    <execution>
      <goals>       
        <goal>compile</goal>
      </goals>
    </execution>
  <executions>
  <dependencies>
    <dependency>
      <groupId>org.aspectj</groupId>
      <artifactId>aspectjrt</artifactId>
      <version>1.7.4</version>
    </dependency>
    <dependency>
      <groupId>org.aspectj</groupId>
      <artifactId>aspectjtools</artifactId>
      <version>1.7.4</version>
    </dependency>
  </dependencies>
<plugin>

Running the Example


The example is provided as a maven project. I am utilizing the concept of a maven 'bom' for the first time for managing the related jersey dependencies. Execute a 'mvn install' to see the example in action or execute a 'mvn jetty:run' on the web project to start the service. The example itself can be downloaded from here.

Client

The client will send the 'role' as a header parameter as shown below for creating a note:
  public NoteResult create(Note note, String role) {
    return webServiceClient.target(UriBuilder.fromUri(baseUri).path("/notes").build())
        .request(MediaType.APPLICATION_XML)
        .header("role", role)
        .post(Entity.entity(note, MediaType.APPLICATION_XML_TYPE), NoteResult.class);
  }

Integration Test

Integration test starts up a web container and executes the client operations. The following shows the spring-security role being honored:
  @Test(expected = ForbiddenException.class)
  public void roleAbsent() {
    Note note = new Note();
    note.setUserId("sacharya");
    note.setContent("Something to say");
    client.create(note, "Fake Role"); // Should throw a JAX-RS ForbiddenException
    fail("Should not have created a note as the role provided is not supported");
  }
Running the equivalent of the above with the role of 'notesuser' will allow the creation of the note. The life-cycle integration test in the attached example demonstrates the same succeeding. The example should be devoid of web.xml. If anyone coming across this blog has integrated Jersey/Spring/Spring-security in a different way, please share, would love to hear. That's all folks! Enjoy!

Thursday, February 21, 2013

Netflix Hystrix - It's Hysterical...err Hystrixcal?

Introduction:


In any distributed architecture, failure of services are inevitable. These could be due to network related issues, bugs in code, unexpected loads, failure of dependencies, what have you? When failure of a single service in a network starts affecting the availability of other services, then that service could become a single point of failure for the entire dependency graph associated with that service.  Hystrix is a library developed and subsequently outsourced by Netflix that is geared toward making a distributed architecture resilient while addressing some of the above concerns. 
Hystrix function as described by the creators "Hystrix is a library designed to control the interactions between these distributed services providing greater latency and fault tolerance. Hystrix does this by isolating points of access between the services, stopping cascading failures across them, and providing fallback options, all of which improve the system's overall resiliency."
I could talk more about Hystrix but reading their excellent WIKI is definitely the more DRY route. My goal in this BLOG is simply to have fun while providing you an example of Hystrix usage to play around with :-)


Fictional Case Study


Developers of company Acme had developed a GreetingService and had also provided a client for consuming the service. The client developed looked like:

public class GreetingJerseyClient implements GreetingClient {
  private final String baseUri;
  private final Client client;

  public GreetingJerseyClient(String baseUri) {
    this.baseUri = baseUri;
    client = ClientFactory.newClient();
  }

  /**
   * @return a Greeting in the specific language, for example "Hello" or "Hola"
   */
  @Override
  public String getGreeting(String languageCode) {
    ....
  }

  /**
   * @return a Greeting for the individual in the specfied language, for example "Hello Sanjay", "Hola Sanjay".
   * @throws InvalidNameException if null was provided as the language code.
   */
  @Override
  public String greet(String name, String languageCode) throws InvalidNameException {
    ...
  }
}

This GreetingService was a core service that was utilized by all their web applications and in some cases consumed by other services as well. The GreetingService itself was a beast that which on load tests showed requests could take upto 1 second to respond at times (probably full GC's kicking in). Accordingly the developers configured the read-timeout for the connections to be 1 second so that any request taking longer than 1 second would time out. Upon deployment all looked good for sometime but then they started encountering problems where the service started degrading, i.e., it was not the occasional request that took a second to respond but all requests in a particular time window that exhibited latency or failure. This resulted in the application container threads all blocking leading to a denial of service.

Bosses were mad, stake holders took to narcotics, stock values plummeted.


A postmortem of the incident was held. Cause of the problem was not singular but multiple things that happened, a perfect storm if you may, a bug was introduced that increased response times, their load balancer ran into unexpected issues, a data center technician spilled his drink on some servers of the cluster..phew. Results of the postmortem were:
  • To provide a default greeting for the getGreeting() call in the event the Greeting service were experiencing degradation.
  • Deploy the Greeting Service in a cloud provider and pay them only for usage ($$$$$) in the event of failure of their existent measly data center. Only the greet(name, languageCode) call would access that.
  • Provide a transparent means of switching to the cloud service in the event of degradation of the in house Greeting Service and switch back when the in home Greeting Service is responsive again.
  • Be able to track and monitor service degradation so the network operations folk could react quickly.
Tasked with the above, the development team decided that Hystrix would help address the above concern really well. As changing the API was not an option, the developers decided to provide a HystrixGreetingClient that would wrap the service calls with a HystrixCommand.

The following is the enhancement they made to provide a default greeting for the greet() call in the event of service degradation:
public class HystrixGreetingClient implements GreetingClient {
  private final GreetingClient client;

  public HystrixGreetingClient(GreetingClient client) {
  }

  @Override
  public String greet() {
    return new GetGreetingHystrixCommand().execute();
  }

 // Default Greeting Agreed Upon
  public static final String DEFAULT_GREETING = "Hello!";

  class GetGreetingHystrixCommand extends HystrixCommand<String> {

    public GetGreetingHystrixCommand() {
      super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("GreetingClient"));
    }

    @Override
    protected String run() throws Exception {
    // Attempt to obtain greeting from the service
      return client.getGreeting();
    }

    @Override
    protected String getFallback() {
      // Fallback provides a default greeting
      return DEFAULT_GREETING;
    }
  }
}

The GetGreetingCommand is a HystrixCommand  which will invoke the fallback greeting if Hystrix detected service degration.  For the greet(name, languageCode) operation, as the goal was to invoke the cloud deployed Greeting Service in the event of service degradation of their in-house version, the developers provided a fallback client that could be invoked as shown below:
public class HystrixGreetingClient implements GreetingClient {
  private final GreetingClient primary;
  private final GreetingClient fallback;

  /**
   * @param primary The primary client which communicates with the in-house data center
   * @param fallback The fallback client which communicates with the $$$$$ cloud provider
   */
  public HystrixGreetingClient(GreetingClient primary, GreetingClient fallback) {
  }

  @Override
  public String greet(String languageCode) {
    ....
  }
 
  @Override
  public String greet(String name, String languageCode) {
    try {
      return new GreetHystrixCommand(name, languageCode).execute();
    }
    catch (HystrixBadRequestException e) {
      if (e.getCause() instanceof InvalidNameException) {
        throw InvalidNameException.class.cast(e.getCause());
      }
      throw e;
    }
  }

  class GreetHystrixCommand extends HystrixCommand<String> {
    private final String name;
    private final String languageCode;

    public GreetHystrixCommand(String name, String languageCode) {
      super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("GreetingClient"))
        .andCommandPropertiesDefaults(HystrixCommandProperties.Setter().withExecutionIsolationThreadTimeoutInMilliseconds(500)));
      this.name = name;
      this.languageCode = languageCode;
    }

    @Override
    protected String run() throws Exception {
      try {
        // Attempt to invoke the primary service
        return primary.greet(name, languageCode);
      }
      catch (InvalidNameException e) {
        // Throw a HystrixBadRequestException as this is not a network or service issue
        throw new HystrixBadRequestException("Invalid Name", e);
      }
    }

    @Override
    protected String getFallback() {
      // If primary failed or was short circuited, invoke the secondary
      return new GreetHystrixFallbackCommand().execute();
    }

    /**
     * Fallback command designed to talk to the cloud provider service
     */
    class GreetHystrixFallbackCommand extends HystrixCommand<String> {

      public GreetHystrixFallbackCommand() {
        super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("GreetingClient"));
      }
      
      /**
       * There is no fallback for this call. Fail fast.
       */
      @Override
      protected String run() throws Exception {
        try {
          return fallback.greet(name, languageCode);
        }
        catch (InvalidNameException e) {
          // This is a user or programming error, not a network or service problem
          // so throw a HystrixBadRequestException
          throw new HystrixBadRequestException("Invalid Name", e);
        }
        catch (RuntimeException e) {
        // Demonstrating that all other exceptions are thrown
          throw e;
        }
      }
    }
  }
}
The call to greet(name, language) if experiencing service degradation will result in the fallback cloud provider equivalent being invoked. The cloud provider call is also wrapped in a HystrixCommand so that it also is subjected to SLA constraints. If the fall back provider cannot fulfill the request, then the there is no fallback for it and failure will occur fast. One thing to note is that if a name provided was invalid, for example null, the service would throw an InvalidNameException. The same should not contribute toward calling a fallback or short circuiting metrics. For this reason, a HystrixBadRequestException is thrown which will not result in the fallback being invoked as well as not contributing toward the failure metrics.

Results of their efforts


Once the new client was released every consumer adopted the same were able to have their applications be resilient to the failure of the dreaded Greeting Service :-)

Logs looked like:
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
.. More of the above
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.run(56) | Obtained Greeting from Service
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.getFallback(62) | Fallback default greeting is being provided
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.run(56) | Obtained Greeting from Service
INFO - com.welflex.example.client.HystrixGreetingClient$GetGreetingHystrixCommand.run(56) | Obtained Greeting from Service
....
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(82) | Obtaining greeting from Primary service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(82) | Obtaining greeting from Primary service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(82) | Obtaining greeting from Primary service
.... More of the above
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(107) | Obtaining greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(107) | Obtaining greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(84) | Obtained greeting from Primary service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(109) | Obtained greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(109) | Obtained greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(82) | Obtaining greeting from Primary service^M
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(109) | Obtained greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(109) | Obtained greeting from Secondary Service

ERROR - com.netflix.hystrix.HystrixCommand.executeCommand(812) | Error executing HystrixCommand
java.lang.RuntimeException: Unable to greet:500
        at com.welflex.example.client.GreetingJerseyClient.greet(GreetingJerseyClient.java:57)
        at com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(HystrixGreetingClient.java:83)
        at com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(HystrixGreetingClient.java:67)
        at com.netflix.hystrix.HystrixCommand.executeCommand(HystrixCommand.java:764)
        at com.netflix.hystrix.HystrixCommand.access$1400(HystrixCommand.java:81)
        at com.netflix.hystrix.HystrixCommand$2.call(HystrixCommand.java:706)
        at com.netflix.hystrix.strategy.concurrency.HystrixContextCallable.call(HystrixContextCallable.java:45)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(107) | Obtaining greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(107) | Obtaining greeting from Secondary Service
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand$GreetHystrixFallbackCommand.run(107) | Obtaining greeting from Secondary Service
..More of the above
INFO - com.welflex.example.client.HystrixGreetingClient$GreetHystrixCommand.run(84) | Obtained greeting from Primary service

The getGreeting() command falls back to the fall back greeting agreed upon when the primary service shows degradation. On the call to greet(), the cloud service is invoked when the primary service shows degradation and then reverts back to the primary service when it health improves.

Monitoring

For monitoring the company set up all web applications using the Greeting service to have the Hystrix Event Stream Servlet so that statistics and metrics could be obtained from each deployment of the application on how the command's were performing. They aggregated these metrics using Turbine and finally had a pretty Hystrix Dashboard at which the Network Operations folk spent hours admiring the animated graph.

An Example

As always, herewith is a maven example of the above GreetingWebService and Hystrix for download. It is a simple Jersey 2.0 application with a test that demonstrates fallback and fail fast. It does not exercise all the configuration that Hystrix provides, that's for you to have fun with. You can tweak the Rest resources to include further delays, tweak the Hystrix commands with different SLAs. Simply run the following command and witness a simple test demonstrating the functionality:
>mvn install 

Conclusion

Why would you want to adopt Hystrix? I ask why not? Do you believe your network to be infallible, your software bug free, your availability ready to be questioned? Yes there is a cost, a cost toward the extra code that one needs to wrap around network calls with Hystrix Commands but compared to the cost of your leaders taking to narcotics or more appropriately your site being non-responsive, it might be a small price to pay. Adoption is not easy IMHO. Care must be taken to understand the exact SLA parameters to configure a Hystrix Command as getting it wrong might hurt rather than alleviate a problem. For this reason a gradual adoption is the preferred route.

I have taken the liberty of a creating a HystrixClient that is only partially configurable. I do not believe this to be the right approach and feel that separate client that allows full configuration of the HystrixCommand by a service consumer is the correct approach.

As always, I might have got things wrong in which case, I would love to learn the error of my ways :-) In the end, remember its Hystrix not Hysterics as I often confused it to be. Enjoy!

All images linked with in this BLOG are NOT my own. I am simply referring to them for demonstration purposes. If owners have concerns, I will gladly remove them. 

Friday, October 5, 2012

JAX-RS 2.0 - Jersey 2.0 Preview - An Example

Introduction


JAX-RS 2.0 spec has reached public review status. Implementors of the API have been hard at work. One among them is Jersey. I have a vested interest in Jersey as I use it at work and therefore wanted to keep up to date on the changes in JAX-RS 2.0 and Jersey 2.0 and thus this BLOG. Jersey 2.0 is not backward compatible with Jersey 1.0 per their site and is instead a major re-write to correctly handle the new specification. When working with a previous REST provider, I found that when they created a 2.0 version of their product, they did not care to be backward "sensitive". I did not say compatible but sensitive :-). Backward sensitivity IMHO is a way to provide new functionality and refactor of an existing API in a way that provides a gradual migration path of consumers of the API and not requiring an all or nothing upgrade.

When backward incompatible changes are made to API's without changing package names, then binary incompatibilities surface when both versions of the API are required within the same class loader system. These incompatibilities can be easily absorbed by small scale adopters who can easily ensure all dependencies are upgraded simultaneously and only the latest version of the API is present. This requirement of simultaneous upgrade is however an arduous challenge for larger and more complex deployments whose development and deployment windows are very separated. Sure that one can look at OSGI or write their own class loaders if they should choose to but I am too dull headed for such extravagance.

In short what I want to ensure with Jersey 2.0 is the following use case:

There are three Services (REST services running within a Servlet container) written in Jersey 1.X. Each of the Services have multiple consumers but we are currently interested in the following where Service A calls Service B and Service C using their respective clients. A Service client is nothing more than a friendly API/library wrapping Jersey Client calls that can be distributed to different consumers of the service.

For the sake of discussion, let us consider that Service B is upgraded to Jersey 2.0 but Service C is not. As part of the upgrade, the maintainers of Service B provide a client to consume the service which is written in Jersey 2.0 as well. Will the Service clients of Service B and Service C be able to co-exist and function in Service A? Does the fact that Service A has not been upgraded to Jersey 2.0 cause problems for the clients of Service B and Service C to function in the container? In addition, let us say that Service A also gets upgraded to Jersey 2.0, then will the Service client for C that is using Jersey 1.X be able to function within Service A? Clearly, one option is to provide multiple clients of the service, one for Jersey 1.X and a second for Jersey 2.0. The same is achievable but does involve additional work of development and maintainence. So the question remains, whether the compatibility discussed will hold true or not? The BLOG and example provided herewith will evaluate the same.

JAX-RS 2.0 and what to expect


Client API

With earlier versions of the JAX-RS, the specification only accounted for a Server Side API for RESTful calls. The same led to different implementors of the Server API such as Jersey and RESTEasy independently developing a client side API (See my BLOG post on REST Clients). JAX-RS 2.0 will contain a Client API to support the server side specification. The JAX-RS 2.0 Client API is a major incorporation of the Jersey 1.X client API.

Asynchronous Support on Client and Server side

Client Side asynchronous support will allow a request to be dispatched in a non-blocking manner with results made available asynchronously. On the server side, long running requests that are I/O bound can be dispatched away, thus releasing the Application server thread to service other requests.

Validation

The lack of validation of data received from Headers, Body etc is something that I whined about in a previous BLOG. JAX-RS 2.0 will provide support for validation of request data.

Filters and Handlers

JAX-RS 2.0 specification accounts for Client and Server Filters. As in the case of the Client API, where implementer's provided their own versions of these filters for the Client and Server. The JAX-RS 2.0 specification absorbs the same into its API. For a more detailed understanding of changes in JAX-RS 2.0, a read of Arun Gupta's blog is recommended. This BLOG will be leaning on the Notes Web Service example that I had previously utilized where Notes can be created, updated and deleted via a client.

Jersey 2.0 and HK2



In previous version of my examples, I have used either Spring or Guice for injecting dependencies into Resources. At the time of this BLOG, I could not find the Spring or Guice support and have therefore used what Jersey 2.0 natively uses for its dependency injection and service location needs, i.e., HK2. HK2 is based on JSR 330 standard annotations. In the following example, a service binder utilizes HK2 and binds interface contracts to implementors as shown below. This binder which is defined in a "service" maven module is then used in the web maven module to enable injection of the NotesService:
public class ServiceBinder implements Binder {
  @Override
  public void bind(DynamicConfiguration config) {
    config.bind(BuilderHelper.link(NotesServiceImpl.class).to(NotesService.class).build());
  }
}

/**
 * Application class
 */
public class NotesApplication extends ResourceConfig {
  
  public NotesApplication() {
    super(NotesResource.class..);
    
    // Register different Binders
    addBinders(new ServiceBinder(), new JacksonBinder());
  }
}

/**
 * A Resource class injected with the Notes Service
 */
@Path("/notes")
public class NotesResource {
  @javax.inject.Inject
  private NotesService service; // Notes Service injection
  ...
}



Client



As previously mentioned, JAX-RS 2.0 supports a client API and Jersey provides an implementation of the same. At the time of writing this BLOG, I could not find hookins for Apache HTTP Client and therefore will be using the standard HttpClient for HTTP calls.

Client Creation

The process of configuring a client and creating the same is very similar to Jersey 1.X.
// Configuration for the Client
ClientConfig config = new ClientConfig();

// Add Providers
config.register(new JacksonJaxbJsonProvider());
config.register(new LoggingFilter());

// Instantiate the Client using the Client Factory
Client  webServiceClient = ClientFactory.newClient(config);

Executing REST Request

With Jersey 1.X, a typical POST to create a Note would have looked like-
NoteResult result = client.resource(UriBuilder.fromUri(baseUri).path("/notes").build())
        .accept(MediaType.APPLICATION_XML).entity(note,  MediaType.APPLICATION_XML).post(NoteResult.class);
The same request using the JAX-RS 2.0 client looks like:
NotesResult result = client.target(UriBuilder.fromUri(baseUri).path("/notes").build())
        .request(MediaType.APPLICATION_XML) // Request instead of accept
        .post(Entity.entity(note, MediaType.APPLICATION_XML_TYPE), NoteResult.class);
The primary difference seems to be where the Resource is identified as "target" rather than "resource". There is no "accepts" method but is replaced with "request" for media type. The Entity is built from a Entity Builder method which returns an instance of java.ws.rs.client.Entity. Overall the API seems quite flowing and does not seem to conflict with the Jersey 1.X client API.

Server Changes



One of the changes I wanted to work with is the Validation but I do not believe the same has been fully implemented as yet. Apart from the Async support mentioned earlier, one area of interest for me  is the Filters and Name Binding feature of JAX-RS.

Name binding and Filter

For those who have used Jersey's ContainerRequest and ContainerResponse Filters, the JAX-RS 2.0 equivalents should be quite familiar. An example of a simple ContainerRequestFilter that logs header attributes is shown below:
@Provider
@Loggable
public class LoggingFilter implements ContainerRequestFilter {
  private static final Logger LOG = Logger.getLogger(LoggingFilter.class);

  @Override
  public void filter(ContainerRequestContext requestContext) throws IOException {
    MultivaluedMap<String, String> map = requestContext.getHeaders();
    LOG.info(map);
  }
}
The annotation @Loggable shown above is a custom one that I created to demonstrate the Name Binding features of JAX-RS 2.0 where a Filter or Handler can be associated with a Resource class, method or an invocation on the client. In short this means one has finer grained control over which methods/resources the filter or handler is made available to. The annotation @Loggable is shown below:
@NameBinding
@Target({ ElementType.TYPE, ElementType.METHOD })
@Retention(value = RetentionPolicy.RUNTIME)
public @interface Loggable {
}
With the above, if we wish to apply the LoggingFilter only to a specific method of a Resource, one simply has to annotate the method with @Loggable:
@Path("/notes")
public class NotesResource {
  @POST
  @Loggable // Will be logged
  public Response create(Note note) {
   ...
  }

  @GET // No annotation @Loggable added
  public Note get(Long noteId) {
    ...
  }
If one wanted to apply the filter on a global level, then Global Binding is used where annotating the filter or handler with @GlobalBinding makes it applicable to all resources of an application.

Asynchronous HTTP



This is where its time for fun, at least for me :-). With Servlet 3.0, one can utilize NIO features and thus free service threads from blocking IO Tasks. Jersey supports suspending the incoming connection and resuming it at a later time. On the client side, the ability to dispatch requests asynchronously is a convenience.

Fictitious Requirements of the Notes Service
  • Notes Client should be able to submit a large number of Notes for creation. The Service should not block the JVM Server thread while it waits to submit but instead should background the same. Client should be notified of a Task ID asynchronously when it becomes available.
  • Notes Client should be able to obtain the status of Note creation asynchronously. The Service should not block the JVM Server thread while waiting for task completion. Upon obtaining the task result, the client should be notified of the same.
  • Notes Client should be able to register a listener that is notified of Note Creation events from the server

Posting a list of Notes for Creation

A list of Notes are submitted to the resource to asynchronously create them. The call itself is asynchronous with a Task ID being delivered to the InvocationCallback when available. One could accomplish the same via an Executor Service etc but this feature simplifies the operation.
 @Override
  public void submitAsync(List<Note> notes, InvocationCallback<Long> taskCallback) {
    webServiceClient.target(UriBuilder.fromUri(baseUri).path("/notes/async").build())
        .request(MediaType.TEXT_PLAIN).async()
        .post(Entity.entity(new NotesContainer(notes), MediaType.APPLICATION_XML), taskCallback);
  }
On the server side, the AsyncNotesResource handles the POST as shown below:
  @POST 
  public void createAsync(@Suspended final AsyncResponse response, final NotesContainer notes) {

    QUEUE_EXECUTOR.submit(new Runnable() {
      public void run() {
        try {
          // This task of submitting notes can take some time
          Long taskId = notesService.asyncCreate(notes.getNotes());
          LOG.info("Submitted Notes for Creation successfully, resuming client connection with TaskId:"
            + taskId);
          // Resume the connection
          response.resume(String.valueOf(taskId));
        }
        catch (InterruptedException e) {
          LOG.error("Waiting to submit notes creation interrupted", e);
          response.resume(e);
        }
      }
    });
  }
One can also obtain a java.util.concurrent.Future using the API if desired.

Obtaining Result of Task Processing Asynchronously

The InvocationCallback of the POST request will be provided a Task ID. Using this Task ID, an asynchronous GET request can be dispatched to obtain the results of the task. As the task might not be completed at the time of the request, the client issues the request in an asynchronous manner and is provided the results of the task when available through the InvocationCallback.
 public void getAsyncNoteCreationResult(Long taskId,
    InvocationCallback<NotesResultContainer> resultCallback) {
    webServiceClient.target(UriBuilder.fromUri(baseUri).path("/notes/async/" + taskId).build())
        .request(MediaType.APPLICATION_XML).async().get(resultCallback);
  }
The Server resource as shown below will not block the JVM server thread but will background the process of obtaining the Task results and the backgrounded process will resume the client connection with the task result when available. Also a time out is set on the request, if the server is not able to respond within the alloted period of time, then a 503 response code is sent back with a "Retry-After" header set to 60 sec.
@GET 
@Path("{taskId}") 
public void getTaskResult(@Suspended final AsyncResponse ar, @PathParam("taskId") final Long taskId) {

    LOG.info("Received Request to Obtain Task Result for Task:" + taskId);
    ar.setTimeout(10000L, TimeUnit.MILLISECONDS);
    ar.setTimeoutHandler(new TimeoutHandler() {
      
      @Override 
      public void handleTimeout(AsyncResponse asyncResponse) {   
        // Respond with 503 with Retry-After header set to 60 seconds
        // when the service could become available.     
        asyncResponse.cancel(60);
      }
    });
    QUEUE_EXECUTOR.submit(new Runnable() {

      @Override public void run() {
        try {
          // This call will block until results are available.
          List<Long> noteIds = notesService.getAysyncNoteCreationResult(taskId);

          NotesResultContainer resultContainer = new NotesResultContainer();
          ....
          LOG.info("Received Note Creation Result, resuming client connection");
          ar.resume(resultContainer);
        }
        catch (InterruptedException ex) {
          LOG.info("Interrupted Cancelling Request", ex);
          ar.cancel();
        }
      }
    });
  }
Server Sent Events

A Feature of Jersey, though I believe not to be really a feature of JAX-RS is the ability to listen to Server sent events. The same is accomplished by the client obtaining an instance of org.glassfish.jersey.media.sse.EventChannel from the Service which then receives events published by the server. The channel is kept alive until disconnected. The Client code to obtain the EventProcessor is shown below:
   WebTarget target = webServiceClient.target(UriBuilder.fromUri(baseUri)
            .path("/notes/async/serverSent").build());
   
   // Register a reader for the Event Processor
   target.configuration().register(EventProcessorReader.class);

   // Obtain an event Processor from the Service
   EventProcessor processor = target.request().get(EventProcessor.class);

   processor.process(new EventListener() {
     public void onEvent(InboundEvent inboundEvent) {
       Note inboundNote = inboundEvent.getData(Note.class);

       for (NoteListener listener : serverEventListeners) {
         listener.notify(inboundNote);
       }
      }
   });
On the service end, a thread is started that will dispatch Note creation events to the channel.
  @Produces(EventChannel.SERVER_SENT_EVENTS)
  @Path("serverSent")
  @GET
  public EventChannel serverEvents() {
    final EventChannel channel = new EventChannel();
    new EventChannelThread(channel, notesService).start();

    return seq;
  }
  
  public class EventChannelThread extends Thread {
     ...
     public void run() {
        // While loop jargon ommited for lack of real estate
        channel.write(new OutboundEvent.Builder().name("domain-progress")
                  .mediaType(MediaType.APPLICATION_XML_TYPE).data(Note.class, noteToDispatch).build());

     }
  }

Example Project



An example project can be downloaded from HERE.

As mentioned previously, the example uses the concept of CRUD as related to a Note. A multi-module maven project is provided herewith that demonstrates the Notes application as written with Jersey 1.X and Jersey 2.X. The Jersey 2.X version has some additional features such as ASYNC, Name Binding and Filter that are demonstrated. In addition, there is a compatibility-test project that evaluates the compatibility concerns that I had mentioned at the beginning of my BLOG. Please note that 2.0-SNAPSHOT of Jersey is being used. A brief description of the maven modules you will find in the example:
  • Notes-common: Contains Data transfer objects and common code used by the Jersey 1.X and Jersey 2.X applications
  • Notes-service: Contains Java classes that are functional services for managing the Notes.
  • Notes-jersey-1.X: Contains Client, Webapp and Integration test modules that demonstrate Jersey 1.X.
  • Notes-jersey-2.X: Contains Client, Webapp and Integration test modules that demonstrate Jersey 2.X features
In order to exercise the compatibility tests two test webapps are created, one using Jersey 1.X and the second using Jersey 2.0. The Notes-client's of Jersey 1.X and Jersey 2.X are exercised within both mentioned test webapps to check for compatibility problems.
The compatibility integration test invokes a resource "/compatibility" on a test web app that is developed using jersey. When the resource is invoked, both the Notes Jersey Clients are exercised to perform CRUD operations using RESTful calls to a Notes Web service Webapp. There are two versions of the test-jersey-webapp, one written in Jersey-1.X and the second in Jersey 2.0.

You can run the full suite of Integration tests merely by executing a "mvn install" from the root level of the project. If you wish to run individual tests, you can also drill down into the Notes-jersey-1.X and Notes-jersey-2.0 modules and execute the same command.

Compatibility Test Results



So can Jersey 2.0 and Jersey 1.X co-exist in the same class loader system? The answer seems to be "Yes" but with some effort. jersey-client-1.X.jar depends on jersey-core-1.X.jar. The latter bundles JAX-RS 1.X (javax.ws.rs.*) packages in it. jersey-client-2.X depends on javax.ws.rs.* packages that are part of JAX-RS 2.0 and therefore requires that JAX-RS API 2.0 is available.

If an application that is running Jersey 1.X wishes to utilize a Jersey 1.X and Jersey 2.X client to call other web services, it is best to upgrade the service in question. The reasoning for the same is best established by the following use case:

Consider the javax.ws.rs.core.UriBuilder from jersey-core-1.X.jar:

public abstract class UriBuilder {
 public abstract UriBuilder uri(URI uri) throws IllegalArgumentException;
}

An implementation by Jersey of the same is com.sun.jersey.api.uri.UriBuilderImpl which is as included part of jersey-core-1.X.jar.

Now the JAX-RS 2.0 version of UriBuilder has additional abstract methods one of which is shown below:

public abstract class UriBuilder {
 public abstract UriBuilder uri(URI uri) throws IllegalArgumentException;

 // New Method Added as part of JAX-RS 2.0
 public abstract UriBuilder uri(String uri) throws IllegalArgumentException;
}
An implementation of the same is org.glassfish.jersey.uri.internal.JerseyUriBuilder.

When a UriBuilder is loaded, it is important that the implementation that defines all the abstract methods of UriBuilder is loaded. If not, one would get see an exception like:
java.lang.AbstractMethodError: javax.ws.rs.core.UriBuilder.uri(Ljava/lang/String;)Ljavax/ws/rs/core/UriBuilder;
The same problem could exist for other JAX-RS 2.0 classes for which Jersey 1.X does not have concrete implementations. As the JAX-RS 2.0 API has to exist in the classpath, one needs to ensure that JAX-RS 1.X API classes are not included as well.

The jersey-core-1.X.jar includes javax.ws.rs.* packages from jax-rs 1.0 API. Re-bundling the jersey-core-1.X jar without these packages and including jersey 2.0 and jax-rs 2.0 API classes will always ensure that the API's have the latest and complete implementations.

In addition, the implementation of javax.ws.rs.ext.RuntimeDelegate is responsible for instantiating a version of UriBuilder. If the Jersey 1.X version of RuntimeDelegate is chosen at runtime, then it will attempt to load com.sun.jersey.apu.uri.UriBuilderImpl which will not have the abstract methods defined as part or JAX-RS 2.0's UriBuilder. For this reason, the org.glassfish.jersey.server.internal.RuntimeDelegateImpl must be chosen so that the JAX-RS 2.0 implementation of UriBuilder, i.e., org.glassfish.jersey.uri.internal.JerseyUriBuilder is always created. The same is accomplished by defining in META-INF/services a file java.ws.rs.ext.RuntimeDelegate that contains a single line, org.glassfish.jersey.uri.internal.JerseyUriBuilder.

The compatiblity-test web applications exercise the above mentioned route in addition to including a custom version of jersey-core-1.X.jar which is nothing more than a version that is stripped of javax.ws.rs.* java packages.

With the above changes, I found that my compatibility tests passed. Of course I am working under the assumption that the JAX-RS 2.0 API classes did not change method signatures in an incompatible way :-).

Conclusion



Overall I feel that the JAX-RS 2.0 implementation with the unification of the Client API is definitely a move in the right direction. Jersey 2.0-SNAPSHOT also seemed to work well from an API and implementation perspective. It is heartening to know that co-existence can be achieved using Jersey 1.X and Jersey 2.0.The Jersey folks have also stated that they will be providing a migration path for Jersey 1.X adopters which is really professional of them. Some work has already been done to that effect on their site.

On the Async features of JAX-RS 2.0, I see potential value on the server side with NIO. On the client side, I find the API to be a convenience. It is arguable that in a clustered stateless environment client side ASYNC might provide limited value.

I have not covered all JAX-RS 2.0 features but only those that have interested me. There are many changes and I might investigate the remaining come another day. Finally, if I have got some areas wrong or my understanding if flawed, I look forward to your corrections on the same.

UPDATE - An example of the released jersey 2/jax-rs 2.0 that uses Spring DI is now available