Maven Integration Testing of Spring Boot Cloud Netflix Eureka Services with Docker

Introduction

My previous blog was about using Spring Cloud Netflix and my experiences with it.  In this BLOG, I will share how one could perform a maven integration test that involves multiple Spring Boot services that use Netflix Eureka as a service registry. I have utilized a similar example as the one I used in my BLOG about Reactive Programming with Jersey and RxJava where a product is assembled from different JAX-RS microservices. For this BLOG though, each of the microservices are written in Spring Boot using Spring Cloud Netflix Eureka for service discovery. A service which I am calling the Product Gateway, assembles data from the independent product based microservices to hydrate a Product.

Product Gateway

A mention of the Product Gateway is warranted for completeness. The following represents the ProductGateway classes where the ObservableProductResource uses a ProductService which in turn utilizes the REST clients of the different services and RxJava to hydrate a Product.

The ObservableProductResource Spring Controller class is shown below which uses a DeferredResult for Asynchronous processing:

@RestController
public class ObservableProductResource {  
  @Inject
  private ProductService productService;
  
  @RequestMapping("/products/{productId}")
  public DeferredResult<Product> get(@PathVariable Long productId) {
    DeferredResult<Product> deferredResult = new DeferredResult<Product>();
    Observable<Product> productObservable = productService.getProduct(productId);

    productObservable.observeOn(Schedulers.io())
    .subscribe(productToSet -> deferredResult.setResult(productToSet), t1 -> deferredResult.setErrorResult(t1));
    
    return deferredResult;
  }
}

Each Micro Service client is declaratively created using Netflix Feign. As an example of one such client, the BaseProductClient, is shown below:

@FeignClient("baseproduct")
public interface BaseProductClient {
  @RequestMapping(method = RequestMethod.GET, value = "/baseProduct/{id}", consumes = "application/json")
  BaseProduct getBaseProduct(@PathVariable("id") Long id);
}

What does the Integration Test do?

The primary purpose is to test the actual end to end integration of the Product Gateway Service. As a maven integration test, the expectancy is that it would entail:

• Starting an instance of Eureka
• Starting Product related services registering them with Eureka
• Starting Product Gateway Service and registering it with Eureka
• Issuing a call to Product Gateway Service to obtain said Product
• Product Gateway Service discovering instances of Product microservices like Inventory, Reviews and Price from Eureka
• Product Gateway issuing calls to each of the services using RxJava and hydrating a Product
• Asserting the retrieval of the Product and shutting down the different services

The test itself would be a maven JUnit integration test.

As the services are bundled as JARs with embedded containers, they present a challenge to start up and tear down during an integration test. 

One option is to create equivalent WAR based artifacts for testing purposes only and use the maven-cargo plugin to deploy each of them under a separate context of the container and test the gateway. That however does mean creating a WAR that might never really be used apart from testing purposes.

Another option to start the different services is using the exec maven plugin and/or some flavor(hack) to launch external JVMs.

Yet another option is write custom class loader logic [to prevent stomping of properties and classes of individual microservices] and launch the different services in the same integration test JVM.

All these are options but what appealed to me was to use Docker containers to start each of these microservice JVMs and run the integration test.  So why Docker? Docker seems a natural fit to compose an application and distribute it across a development environment as a consistent artifact. The benefits during micro service based integration testing where one can simply pull in different docker images such as services, data stores etc of specific versions without dealing with environment based conflicts is what I find appealing.

Creating Docker Images 

As part of building each of the web services, it would be ideal to create a Docker image. There are many maven plugins out there to create Docker images [actually too many]. In the example, we have used the one from Spotify.  The building of the Docker image using the Spotify plugin for Spring Boot applications is nicely explained in the BLOG from spring.io, Spring Boot with Docker.

What I would see happening is that as part of the build process, the Docker image would be published to a docker repository which is internal to an organization and then made available for other consumers.

Integration Test

As part of the pre-integration test phase of maven, we would like to start up the Docker containers representing the different services. In order for the gateway container to work with the other service containers, we need to be able to link the Docker containers. I was not able to find a way to do that using the Spotify plugin. What I instead found myself doing is utilizing another maven plugin for Docker by Roland HuB which has much better documentation and more features. Shown below is the plugin configuration for the integration test.

<plugin>
  <groupId>org.jolokia</groupId>
  <artifactId>docker-maven-plugin</artifactId>
  <version>0.13.6</version>
  <configuration>
    <logDate>default</logDate>
    <autoPull>true</autoPull>
    <images>
      <image>
        <!-- Eureka Server -->
        <alias>eureka</alias>
        <name>docker/eureka</name>
        <run>
          <wait>
            <http>
              <url>http://localhost:8761</url>
              <method>GET</method>
              <status>200</status>
            </http>
            <time>30000</time>
          </wait>
          <log>
            <prefix>EEEE</prefix>
            <color>green</color> <!-- Color the output green -->
          </log>
          <ports>
            <port>8761:8761</port> <!-- Local to container port mapping -->
          </ports>
          <env>
            <eureka.instance.hostname>eureka</eureka.instance.hostname> <!-- Override host name property -->
          </env>
        </run>
      </image>
      <image>
        <alias>baseproduct</alias>
        <name>docker/baseproduct</name>
        <run>
          <wait>
            <http>
              <url>http://localhost:9090</url>
              <method>GET</method>
              <status>200</status>
            </http>
            <time>30000</time>
          </wait>
          <log>
            <prefix>EEEE</prefix>
            <color>blue</color>
          </log>
          <ports>
            <port>9090:9090</port>
          </ports>
          <links>
            <link>eureka</link> <!-- Link to Eureka Docker image -->
          </links>
          <env>
            <!-- Notice the system property overriding of the eureka service Url -->
            <eureka.client.serviceUrl.defaultZone>http://eureka:8761/eureka/</eureka.client.serviceUrl.defaultZone>
          </env>
        </run>
      </image>
      <!--....Other service containers like price, review, inventory-->
      <image>
        <alias>product-gateway</alias>
        <name>docker/product-gateway</name>
        <run>
          <wait>
            <http>
              <url>http://localhost:9094</url>
              <method>GET</method>
              <status>200</status>
            </http>
            <time>30000</time>
          </wait>
          <log>
            <prefix>EEEE</prefix>
            <color>blue</color>
          </log>
          <ports>
            <port>9094:9094</port>
          </ports>
          <links>
            <!-- Links to all other containers -->
            <link>eureka</link>
            <link>baseproduct</link>
            <link>price</link>
            <link>inventory</link>
            <link>review</link>
          </links>
          <env>
            <eureka.client.serviceUrl.defaultZone>http://eureka:8761/eureka/</eureka.client.serviceUrl.defaultZone>
            <!-- Setting this property to prefer ip address, else Integration will fail as it does not know host name of product-gateway container-->
            <eureka.instance.prefer-ip-address>true</eureka.instance.prefer-ip-address>
          </env>
        </run>
      </image>
    </images>
  </configuration>
  <executions>
    <execution>
      <id>start</id>
      <phase>pre-integration-test</phase>
      <goals>
        <goal>start</goal>
      </goals>
    </execution>
    <execution>
      <id>stop</id>
      <phase>post-integration-test</phase>
      <goals>
        <goal>stop</goal>
      </goals>
    </execution>
  </executions>
</plugin>

One of the nice features is the syntax color prefix of each containers messages, this gives one a sense of visual separation among the multitude of containers that are started. The Integration Test itself is shown below:

@RunWith(SpringJUnit4ClassRunner.class)
@ContextConfiguration(classes = { ProductGatewayIntegrationTest.IntegrationTestConfig.class })
public class ProductGatewayIntegrationTest {
  private static final Logger LOGGER = Logger.getLogger(ProductGatewayIntegrationTest.class);
  
  /**
   * A Feign Client to obtain a Product
   */
  @FeignClient("product-gateway")
  public static interface ProductClient {
    @RequestMapping(method = RequestMethod.GET, value = "/products/{productId}", consumes = "application/json")
    Product getProduct(@PathVariable("productId") Long productId);
  }

  @EnableFeignClients
  @EnableDiscoveryClient
  @EnableAutoConfiguration
  @ComponentScan
  @Configuration
  public static class IntegrationTestConfig {}

  // Ribbon Load Balancer Client used for testing to ensure an instance is available before invoking call
  @Autowired
  LoadBalancerClient loadBalancerClient;

  @Inject
  private ProductClient productClient;

  static final Long PRODUCT_ID = 9310301L;

  @Test(timeout = 30000)
  public void getProduct() throws InterruptedException {
    waitForGatewayDiscovery();
    Product product = productClient.getProduct(PRODUCT_ID);
    assertNotNull(product);
  }

  /**
   * Waits for the product gateway service to register with Eureka
   * and be available on the client.
   */
  private void waitForGatewayDiscovery() {
    while (!Thread.currentThread().isInterrupted()) {
      LOGGER.debug("Checking to see if an instance of product-gateway is available..");
      ServiceInstance choose = loadBalancerClient.choose("product-gateway");
      if (choose != null) {
        LOGGER.debug("An instance of product-gateway was found. Test can proceed.");
        break;
      }
      try {
        LOGGER.debug("Sleeping for a second waiting for service discovery to catch up");
        Thread.sleep(1000);
      }
      catch (InterruptedException e) {
        Thread.currentThread().interrupt();
      }
    }
  }
}

The test uses the LoadBalancerClient client from Ribbon to ensure an instance of 'product-gateway' can be discovered from Eureka prior to using the Product client to invoke the gateway service to obtain back a product.

Running the Example

The first thing you need to do is make sure you have Docker installed on your machine. Once you have Docker installed, clone the example from github (https://github.com/sanjayvacharya/sleeplessinslc/tree/master/product-gateway-docker) and then execute a mvn install from the root level of the project. This will result in the creation of Docker images and the running of the Docker based integration tests of the product gateway. Cheers!

Spring Cloud Netflix Eureka Example...It's so Bootiful

Introduction

Been looking at Netflix Eureka for Service Registry/Discovery and thought I'd share some thoughts via an example. My interest was more around Eureka 2.0 and I was eagerly awaiting their release in early 2015 per swags(?) but that did not materialize with the team stating its better to adopt Eureka 1.0. Not very thrilled I must say :-(.

Looking at Eureka 1.0, there appears two routes that one could go:

1. Use Stock Netflix Eureka
2. Use  Spring Cloud Netflix which is a Netflix Eureka and garnished with other Netflix OSS components, heated to a Spring temperature and presented as a Spring Boot Dish.

This Blog will utilize Spring Cloud Netflix and my favorite Notes Web Service to demonstrate a workable project that one can check out play with to understand Service Discovery with Eureka. The example will register an instance of  the Notes Service with Eureka Server (the registry) and a Notes Client will discover the service instance and issue a request to it simulating a full integration. Additional Technologies used:

• RestAssured - Rest Assured is used a fluent testing of the service API   
• Spring Fox - Used to expose Swagger UI for service documentation and testing
• Netlflix Ocelli - Client Side Load Balancing

For the Client side of the application, I choose to use Netflix Ocelli for Client side load balancing along with Spring Frameworks RestTemplate. Why Ocelli over Ribbon for Client Side Load Balancing? Well,  Netflix seems to be looking at Ocelli as the new Ribbon, at least that is what this email chain seems to indicate.

Eureka Components

The following are some components one deals with when working with Eureka

EurekaClientConfig

EurekaClientConfig provides the configuration required by eureka clients to register an instance with Eureka. Netflix Eureka provides an implementation DefaultEurekaClientConfig that is configured via a property file eureka.client.props. Spring Cloud Netflix extends EurekaClientConfig and provides a EurekaClientConfigBean which implements and EurekaClientConfig and works with Spring properties. Properties prefixed by eureka.client will map into the EurekaClientConfigBean via Project Lombok magic.

EurekaServerConfig

Follows a similar approach like the ClientConfig but this is used for Eureka Server properties

EurekaInstanceConfig

Provides the configuration required by an instance to register with Eureka server. If building a service, you might want to tweak some of  knobs here. You can control properties like:

• Name of Service
• Secure/Non-secure ports of operation
• Lease renewal and expiration
• Home page URL and Status page URL

If working with vanilla Netflix Eureka, you could use MyDataCenterInstanceConfig (for non-AWS data center or the CloudInstanceConfig (AWS). Spring Cloud Netflix users get a EurekaInstanceConfigBean spruced with Lombok magic to control their properties. The properties are prefixed by eureka.instance.

Discovery Client

Discovery Client uses a 'jersey 1' client to communicate with Eureka Server to:

• Register a service instance
• Renew the lease of a Service instance
• Cancel of a lease of service instance
• Query Eureka Server for registered instances

DiscoveryClient is configured by the DiscoveryManager with has information of the EurekaClientConfig and the EurekaInstanceConfig

The Notes Example

The Notes example defines a Server side component that will register with Netflix Eureka, a Client side library, the Notes Client which discovers a  Notes Service Instance from Eureka and an integration test that demonstrates this lifecycle.

Notes Service Code

The Heart of the Service code is the declaration of the Spring Boot Application as shown below:

@SpringBootApplication // Spring Boot 
@EnableEurekaClient    // Enable Eureka Client
@RestController 
@EnableSwagger2        // Swagger  
public class Application {
  @RequestMapping("/")
  public String home() {
    return "This is the Notes App";
  }

  @Bean
  public Docket swaggerSpringMvcPlugin() { // Minimalistic setting for Swagger Support
    return new Docket(DocumentationType.SWAGGER_2)
            .select()
            .paths(PathSelectors.any())
            .build().pathMapping("/");
  }

  public static void main(String args[]) {
    SpringApplication.run(Application.class, args);
  }
}

@RestController
@Api(basePath = "/notes", value = "Notes", description = "Note Creation and Retrieval", produces = "application/xml")
public class NotesController {
  @Inject
  private NotesService notesService;

  @RequestMapping(value = "/notes", method = RequestMethod.POST)
  public Long create(@RequestBody Note note) {
    return notesService.createNote(note);
  }

  @RequestMapping(value = "/notes/{id}", method = RequestMethod.GET, produces = MediaType.APPLICATION_XML_VALUE)
  public Note get(@PathVariable Long id) {
    return notesService.getNote(id);
  }
}

In the above Boot application, we enabled Eureka Client via the @EnableEurekaClient annotation and also set up swagger support by creating a Docket and adding the @EnableSwagger2 annotation. The JSON Swagger resource is available at http://localhost:9090/v2/api-docs and the Swagger HTML is available at http://localhost:9090/swagger-ui.html.  The configuration for the application is driven by a simple YAML file as shown below where the port, location of Eureka server instance and a custom instance Id are defined:

spring:
  application:
    name: Notes

server.port: 9090

eureka:
  client:
    serviceUrl:
          defaultZone: http://localhost:8761/eureka/v2/
  instance:
    metadataMap:
      instanceId: ${spring.application.name}:${spring.application.instance_id:${random.value}}

If one wished to integration test (exercise DB and other calls) in the Service application without having to worry about the Eureka registration and discovery, you could simply turn of the eureka client and use, say RestAssured to validate your service as shown below.

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = Application.class)
@WebAppConfiguration
@IntegrationTest({"server.port=0", // Pick an ephemeral port
                  "eureka.client.enabled=false" // Prevent registering/discovering with Eureka
})
public class NotesControllerTest {

  @Value("${local.server.port}")
 private  int port;

  @Before
  public void setUp() {
    RestAssured.port = port; 
  }

  @Test
  public void createAndGetNote() {
    given().contentType(MediaType.APPLICATION_XML)
           .body(new Note("Test"))
            .when()
            .post("http://localhost:" + port + "/notes")
            .then()
            .statusCode(200).body(equalTo("1"));

    when().get("/notes/{id}", 1).then()
            .assertThat()
            .statusCode(200)
             .body("note.content", equalTo("Test"));
  }
}

Notes Client Code

For the NoteClient, as mentioned, Ocelli is used as a Load Balancer with RestTemplate used to make the REST call.

public class NotesClientImpl implements NotesClient, Closeable {
  private final RestTemplate restTemplate;
  private final Subscription subscription;
  private final AtomicReference<List<InstanceInfo>> result;
  private final RoundRobinLoadBalancer<InstanceInfo> instanceInfoLoadBalancer;

  public NotesClientImpl(RestTemplate restTemplate, DiscoveryClient discoveryClient,
                         long refreshInterval, TimeUnit refreshIntervalTimeUnit) {
    this.restTemplate = restTemplate;
    this.result = new AtomicReference<List<InstanceInfo>>(new ArrayList<>());
    this.subscription = new EurekaInterestManager(discoveryClient).newInterest()
            .forApplication("Notes") // Notes Service
            .withRefreshInterval(refreshInterval, refreshIntervalTimeUnit) // Ocelli Refresh Interval
            .asObservable()
            .compose(InstanceCollector.<InstanceInfo>create())
            .subscribe(RxUtil.set(result));
    instanceInfoLoadBalancer = RoundRobinLoadBalancer.<InstanceInfo>create(); // Round Robin
  }

  private String getServiceInstanceUrl() {
    InstanceInfo instanceInfo = instanceInfoLoadBalancer.choose(result.get()); // Choose an instance
    if (instanceInfo != null) {
      return "http://" + instanceInfo.getIPAddr() + ":" + instanceInfo.getPort();
    }
    throw new RuntimeException("Service Not available");
  }


  @Override
  public Note getNote(Long noteId) {
    return restTemplate.getForEntity(getServiceInstanceUrl() + "/notes/{id}", Note.class, noteId).getBody(); 
  }
  ...
}

Ocelli uses an 'interest' manager to register a subscription for changes to Notes Service. A round robin load balancer is used in the example. The NotesClient is set up in the NotesClientConfig.

Notes Integration Test

NotesIntegrationTest  is where things get interesting. The integration test will -

• Start a Eureka Server in the pre-integration-test phase of maven using Cargo. Note that we are using the stock Netflix eureka WAR for this. One could also use the Spring Cloud version via an exec maven plugin.
• Have Notes Service start on a random port and register with the Eureka Server
• Have the Notes Client create and obtain a 'Note' by discovering the Notes Service from Eureka

Eureka operations are usually set in range of 20 to 30 seconds for things like registration, heartbeart, renewal and registry fetching. That works in a production environment but in an integration test waiting for so long is not an option. Thankfully, the Spring Boot test framework allows for easy overriding of these properties.

@RunWith(SpringJUnit4ClassRunner.class)
@SpringApplicationConfiguration(classes = {Application.class, NotesClientConfig.class}) // Note the Import of Server and Client
@WebIntegrationTest(value = {"eureka.instance.leaseRenewalIntervalInSeconds=10", // Lease Interval
        "eureka.client.instanceInfoReplicationIntervalSeconds=1",
        "eureka.client.initialInstanceInfoReplicationIntervalSeconds=1",
        "eureka.client.registryFetchIntervalSeconds=1",
        "eureka.client.serviceUrl.defaultZone=http://localhost:9000/eureka/v2/"}, // Overriding eureka location
        randomPort =  true)
public class NotesIntegrationTest {

  @Inject
  private NotesClient notesClient; // Notes Client injected

  @Value("${ocelliRefreshIntervalSeconds}")
  private Long ocelliRefreshIntervalSeconds; // How often Ocelli Client has been set up to refresh
  
  @Test
  public void createAndGetNote() throws InterruptedException {
    // Wait for Notes App to register
    waitForNotesRegistration(); // Waits for Notes Service to register before proceeding
    Thread.sleep((ocelliRefreshIntervalSeconds * 1000) + 1000L); // Ocelli might not be in sync
    Note note = new Note("Test");
    Long noteId = notesClient.createNote(note); // Create a note
    assertEquals(note, notesClient.getNote(noteId)); // Read the node
  }

  @Inject
  private DiscoveryClient discoveryClient;

  private void waitForNotesRegistration() throws InterruptedException {
   /* Discovery client is used to make sure service is registered before the Notes Client gets a chance to discover and exercise API */
  }
}

The important thing here to note is that I am using the 'war' version of the Netflix Eureka server to start up before the integration test runs. It is equally possible to launch the Spring Boot 'jar' version of the eureka server via the exec maven plugin prior to the start of the integration test.

Parting Thoughts

For the curious, this Notes Spring Cloud Netflix Eureka example can be found at sleeplessinslc github.

I would also like to share the below as my personal 'opinion' only

Netflix Eureka

The good - 

Awesome project, a Service Registry for the Web. Apps in any language can participate. REST + Jersey, preaching to the choir. More importantly, developed with a profound understanding of failure scenarios around a Microservice 'service registry'. Nitesh Kants presentation is a must see.

Good starter documentation and responsive folks on the mailing list. 

The bad - 

I understand Eureka was designed for AWS usage primarily by Netflix.  Maybe a more modular (plugin) approach would have been nicer ensuring a clean separation between AWS and non-AWS code, especially considering it is a fine candidate for private cloud.

Discovery client fetches entire registry rather than information about specific services. Eureka 2 architecture looks better in that regard where one only gets data about services they are interested in.

Jersey 1 - Lets update to Jersey 2.X folks! 

Eureka UI looks pretty horrid and that is coming from a 'server side' only programmer :-). 

There is an issue where shutting down a service instance leads to shutting down all other services on that host which is concerning if hosting multiple services on a node.

Ocelli vs. Ribbon. Can we have clear direction? Ocelli has not had a release since July 10th, Ribbon though has a more recent release. So is Ocelli really the new Ribbon? 

Spring Cloud Netflix

The good -

If using a Spring ecosystem, a no brainer to adopt. Meshes really well with Spring properties and enables Hystrix and other Netflix OSS.

Spring Boot is pretty sweet but you need to work in a 'controlled' environment in the 'Spring Boot Way' to enjoy full benefit. 

Spring Cloud Netflix re-bundled Eureka server is much more visually pleasing than the core Eureka UI. I believe they also have a workaround with reference to the Shutdown bug I mentioned.

The bad - 

You sold yourself to the Bootiful devil :-)

• Custom extension classes of Netflix interfaces (EurekaClientConfigBean..etc), rather than bridging
• Version of Eureka is older
• Documentation does not match up with implementation. Case in point the property "spring.cloud.client.hostname" defined in their documentation is not even part of current release. I spent some time chasing this
• Do we really need Lombok?

So, what I want to do next?

Leaning back on my reactive example,  lets say I have an aggregator Product Service that I want to test that utilizes a Base Product Service, a Price Service and an Inventory Service, all developed with Spring Cloud Eureka, how would I write a maven integration test for this? Off to Docker land my friends, onward and upward!