Source code

If you would like to try it by yourself, you may always take a look at my source code example. In order to do that you need to clone my repository sample-spring-microservices-kubernetes. Then just follow my instructions The good idea is to read the article Microservices with Spring Cloud Kubernetes before you move on.

Step 1. Enable Spring Cloud Kubernetes Load Balancer

You need to include a single Spring Boot Starter to enable Spring Cloud Kubernetes Load Balancer. It is spring-cloud-starter-kubernetes-loadbalancer. Consequently, you also need to have a REST client on the classpath. Spring RestTemplate is automatically included with the Spring Web module. We will also use the OpenFeign client, so you should include the right starter.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-loadbalancer</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

Step 2. Implement and use the REST client

Spring Cloud OpenFeign is a declarative REST client. Therefore, you need to create an interface with methods and Spring MVC annotations. It is important to set the right name inside @FeignClient annotation. This name needs to be the same as the name of the target Kubernetes Service. In the following code example, you see an implementation of the employee service client inside department-service.

@FeignClient(name = "employee")
public interface EmployeeClient {

   @GetMapping("/department/{departmentId}")
   List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
	
}

OpenFeign client is enabled after annotating the main class with @EnableFeignClients. After that you can inject it to the @RestController class. Finally, you may use its method findByDepartment to find employees assigned to the particular department.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;

    public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient) {
        this.repository = repository;
        this.employeeClient = employeeClient;
    }

    @GetMapping("/{id}/with-employees")
    public Department findByIdWithEmployees(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            department.setEmployees(employeeClient.findByDepartment(department.getId()));
            return department;
        }
        return null;
    }

    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Opposite to the OpenFeign, the RestTemplate is a low-level HTTP client. We need to enable Spring Cloud load balancing for it. To do that just annotate RestTemplate bean with @LoadBalanced.

@Bean
@LoadBalanced
RestTemplate restTemplate() {
    return new RestTemplate();
}

Here’s the similar implementation of @RestController class, but this time with the RestTemplate.

public class DepartmentWithRestTemplateController {


    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    RestTemplate restTemplate;

    public DepartmentWithRestTemplateController(DepartmentRepository repository, RestTemplate restTemplate) {
        this.repository = repository;
        this.restTemplate = restTemplate;
    }

    @GetMapping("/{id}/with-employees")
    public Department findByIdWithEmployees(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            department.setEmployees(findEmployeesByDepartment(department.getId()));
            return department;
        }
        return null;
    }


    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(findEmployeesByDepartment(d.getId())));
        return departments;
    }

    private List<Employee> findEmployeesByDepartment(String departmentId) {
        Employee[] employees = restTemplate
                .getForObject("http://employee//department/{departmentId}", Employee[].class, departmentId);
        return Arrays.asList(employees);
    }

}

Step 3. Deploy Spring Cloud applications on Kubernetes

The project is ready to be used with Skaffold. Therefore, you don’t have to worry about the deployment process. You just need to run a single command skaffold dev --port-forward to deploy our applications on Kubernetes. But before deploying them on the cluster, we perform a short overview of the communication process. We have three microservices. Each of them is running in two instances. They are using Mongo as a backend store. There is also the gateway application. It is built on top of Spring Cloud Gateway. It provides a single API endpoint to all downstream services. Of course, all our applications are using Spring Cloud Load Balancer for traffic management. To clarify, you can take a look at the picture below.

Here’s the Kubernetes deployment manifest for the employee-service. It sets two running pods in replicas parameter, and references ConfigMap and Secret to inject Mongo credentials.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: employee
  labels:
    app: employee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: employee
  template:
    metadata:
      labels:
        app: employee
    spec:
      containers:
      - name: employee
        image: piomin/employee
        ports:
        - containerPort: 8080
        env:
        - name: MONGO_DATABASE
          valueFrom:
            configMapKeyRef:
              name: mongodb
              key: database-name
        - name: MONGO_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-user
        - name: MONGO_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-password

The most important element in the communication between applications is Kubernetes Service object. The name of service must be the same as the hostname used by the RestTemplate or OpenFeign client. In our case that name is employee.

apiVersion: v1
kind: Service
metadata:
  name: employee
  labels:
    app: employee
    spring-boot: "true"
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: employee
  type: ClusterIP

Let’s verify the status after deploying all the applications. Here’s the list of running pods.

Here’s the list of Kubernetes Endpoints.

Step 4. Communication in the POD mode

By default, Spring Cloud Kubernetes Load Balancer uses the POD mode. In this mode, it gets the list of Kubernetes endpoints to detect the IP address of all the application pods. In that case, the only thing you need to do is to disable the Ribbon load balancer. Spring Cloud Kubernetes is still using it as a default client-side load balancer. To disable it you need to set the following property.

spring:
  cloud:
    loadbalancer:
      ribbon:
        enabled: false

After adding a test data we may send some requests to the endpoint GET /{id}/with-employees. It finds a department by the id. Then it communicates with the employee-service endpoint GET /department/{departmentId} to search all the employees assigned to the current department. The department-service is exposed on port 8081, since I enabled option port-forward on Skaffold.

$ curl http://localhost:8081/5f5896b3cb8caf7e6f6b9e1c/with-employees
{"id":"5f5896b3cb8caf7e6f6b9e1c","organizationId":"1","name":"test1","employees":[{"id":"5f5896e26092716e54f60a9a","name":"test1","age":30,"position":"d
eveloper"},{"id":"5f5896f29625e62c7d373906","name":"test2","age":40,"position":"tester"},{"id":"5f5897266092716e54f60a9b","name":"test3","age":45,"posit
ion":"developer"}]}

Let’s take a look on the logs from employee-service. I repeated the request visible above several times. The requests are load balanced between two available instances of employee-service. Skaffold prints the id of every pod in the logs, so you can verify that everything works fine.

Step 5. Communication across multiple namespaces

By default, Spring Cloud Kubernetes allows load balancing within the same namespace. You may enable discovery across multiple namespaces. To do that you need to use the property following property.

spring:
  cloud:
    kubernetes:
      discovery:
        all-namespaces: true

After setting that, we will deploy the employee-service application in the different namespace than department-service. To do that you need to set parameter -n on the skaffold command.

$ skaffold dev -n test-a

Here’s the current list of running pods.

You just need to send the same request as before curl http://localhost:8081/5f5896b3cb8caf7e6f6b9e1c/with-employees. Of course, the Kubernetes service name must be unique across all the namespaces.

Step 6. Load balancer on Spring Cloud Gateway

Spring Cloud Gateway uses the same load balancing mechanism as other Spring Cloud applications. To enable it on Kubernetes we need to include dependency spring-cloud-starter-kubernetes-loadbalancer. We should also enable Spring Cloud DiscoveryClient and disable Ribbon. Here’s the configuration of the gateway-service.

spring:
  application:
    name: gateway
  cloud:
    loadbalancer:
      ribbon:
        enabled: false
    gateway:
      discovery:
        locator:
          enabled: true
      routes:
        - id: employee
          uri: lb://employee
          predicates:
            - Path=/employee/**
          filters:
            - RewritePath=/employee/(?.*), /$\{path}
        - id: department
          uri: lb://department
          predicates:
            - Path=/department/**
          filters:
            - RewritePath=/department/(?.*), /$\{path}
        - id: organization
          uri: lb://organization
          predicates:
            - Path=/organization/**
          filters:
            - RewritePath=/organization/(?.*), /$\{path}

A gateway acts as an entry point to our system. It performs routing and load balancing to all the downstream services. Therefore, we can call any of our applications using already defined routes. Since gateway-service is available on port 8080, I can call any of the endpoints using the following requests.

We can expose API documentation of all the microservices on the gateway. To do that we may use the SpringFox project. First, we need to include SpringFox Starter to Maven dependencies.

<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-boot-starter</artifactId>
   <version>3.0.0</version>
</dependency>

Swagger documentation is usually generated for a single application. Because we want to expose documentation of all our microservices we need to override the Swagger resource provider. The custom implementation is visible below. It uses a discovery mechanism to the names of running services.

@Configuration
public class GatewayApi {

   @Autowired
   RouteDefinitionLocator locator;

   @Primary
   @Bean
   public SwaggerResourcesProvider swaggerResourcesProvider() {
      return () -> {
         List<SwaggerResource> resources = new ArrayList<>();
         Flux<RouteDefinition> definitions = locator.getRouteDefinitions();
         definitions
               .filter(routeDefinition -> !routeDefinition.getId().startsWith("ReactiveCompositeDiscoveryClient_"))
               .subscribe(routeDefinition -> resources.add(createResource(routeDefinition.getId(), "2.0")));
         try {
            Thread.sleep(1000);
         } catch (InterruptedException e) {
            e.printStackTrace();
         }
         return resources;
      };
   }

   private SwaggerResource createResource(String location, String version) {
      SwaggerResource swaggerResource = new SwaggerResource();
      swaggerResource.setName(location);
      swaggerResource.setLocation("/" + location + "/v2/api-docs");
      swaggerResource.setSwaggerVersion(version);
      return swaggerResource;
   }

}

Here’s the Swagger UI available at http://localhost:8080/swagger-ui/index.html on my local machine.

Step 7. Enabling circuit breaker

We can use a circuit breaker component with Spring Cloud Load Balancer. The default Spring Cloud implementation is based on Resilience4j. In order to enable it for your application, you need to include the following dependency.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>

The next step is to provide a configuration of a circuit breaker.

@Bean
Customizer<Resilience4JCircuitBreakerFactory> defaultCustomizer() {
   return factory -> factory.configureDefault(id ->
         new Resilience4JConfigBuilder(id)
               .timeLimiterConfig(TimeLimiterConfig.custom()
                     .timeoutDuration(Duration.ofMillis(1000))
                     .build())
               .circuitBreakerConfig(CircuitBreakerConfig.custom()
                     .slidingWindowSize(10)
                     .failureRateThreshold(66.6F)
                     .slowCallRateThreshold(66.6F)
                     .build())
               .build()
   );
}

We need to inject Resilience4JCircuitBreakerFactory to the Spring controller class. Then we are creating the circuit breaker instance using an injected factory. Finally, the client calling method is running inside the circuit breaker run method.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;
    Resilience4JCircuitBreakerFactory circuitBreakerFactory;

    public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient,
                                Resilience4JCircuitBreakerFactory circuitBreakerFactory) {
        this.repository = repository;
        this.employeeClient = employeeClient;
        this.circuitBreakerFactory = circuitBreakerFactory;
    }

    @GetMapping("/{id}/with-employees-and-delay")
    public Department findByIdWithEmployeesAndDelay(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            Resilience4JCircuitBreaker circuitBreaker = circuitBreakerFactory.create("delayed-circuit");
            List<Employee> employees = circuitBreaker.run(() ->
                    employeeClient.findByDepartmentWithDelay(department.getId()));
            department.setEmployees(employees);
            return department;
        }
        return null;
    }
}

Conclusion

Load balancing is one of the key patterns in a microservices architecture. Spring Cloud Load Balancer is replacing the Ribbon client. By default, load balancing in Kubernetes is based on Services. Therefore, you need to use additional tools for more advanced routing mechanisms. Spring Cloud Kubernetes comes with some interesting features. One of them is the ability to load balance across multiple namespaces. You can also use the additional Spring Cloud components like a circuit breaker. In comparison with tools like Istio, it is still not much. Is it a chance for improvement? We will see. Nevertheless, Spring Cloud Kubernetes is currently one of the most popular Spring Cloud projects. It may be a good choice if you are migrating your Spring Cloud microservices architecture to Kubernetes. Support for the load balancer may be an important step during such a migration.

The post Spring Cloud Kubernetes Load Balancer Guide appeared first on Piotr's TechBlog.

Spring Cloud video course

You may also find a video guide, where I’m showing all the features described here with more details. It is available on my YouTube channel and consists of four parts:

Part 1 – Introduction to Spring Boot (~35 minutes)
Part 2 – Distributed Configuration and Service Discovery (~36 minutes)
Part 3 – Inter-service communication (~34 minutes)
Part 4 – API Gateway (~22 minutes)

Architecture

The diagram visible below illustrates the architecture of our sample system. Here we have the characteristic elements for microservices like API gateway, discovery server, and configuration server. In the next sections of this article, I’ll show how to use Spring Cloud components that provide an implementation of those patterns. Currently, the main component for adding the API gateway to your system is Spring Cloud Gateway.
Spring Cloud provides integrations to several solutions that may be used as a discovery server: Netflix Eureka, HashiCorp Consul, Alibaba Nacos, or Apache ZooKeeper. The most common choice is between the first two of them. While Spring Cloud Netflix Eureka is dedicated just for discovery, Spring Cloud Consul may realize both discovery feature basing on Consul Services, and distributed configuration feature basing on Consul Key/Value engine.

In turn, Spring Cloud Config is responsible just for providing a mechanism for configuration management. However, it may also be integrated with third-party tools like Vault from HashiCorp.
We will figure out how to integrate our applications with discovery and configuration servers on the example of two simple Spring Boot applications callme-service and caller-service. The application caller-service is also calling endpoints exposed by the callme-service. We will enable such mechanisms on the caller-service like client-side load balancer with new Spring Cloud Load Balancer, and circuit breaker with new Spring Cloud Circuit Breaker built on top of Resilience4J.

Service discovery

Switching between discovery servers on the client-side is very easy with Spring Cloud due to the DiscoveryClient abstraction. This switch comes down to the replacement of a single dependency in Maven pom.xml. So if you are using Eureka you should add the following starter in your microservice.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

On the other hand, if you are using Consul you should add the following starter in your microservice.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-consul-discovery</artifactId>
</dependency>

The situation is a little bit more complicated if you are defining some non-default configuration settings for a discovery client. In that case, you need to use properties specific just for Eureka, or just for Consul. For example, if you are running more than one instance of a single application on the same host with dynamic HTTP server port feature enabled (option server.port=0), you have to set a unique id of every instance. Here’s the property used for Eureka’s client.

eureka:
  instance:
    instanceId: ${spring.cloud.client.hostname}:${spring.application.name}:${random.value}

For the Consul client the same configuration looks as shown below.

spring:
  cloud:
    consul:
      discovery:
        instanceId: ${spring.cloud.client.hostname}:${spring.application.name}:${random.value}

You can easily configure and run Eureka discovery in your microservices architecture using Spring Cloud Netflix Eureka Server module. You just need to create the Spring Boot application that includes that module.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>

We also need to enable Eureka for the application by annotating the main class with @EnableEurekaServer.

@SpringBootApplication
@EnableEurekaServer
class DiscoveryServerApplication

fun main(args: Array<String>) {
    runApplication<DiscoveryServerApplication>(*args)
}

The most convenient way to run Consul on the local machine is by using its Docker image. We can Consul on Docker container in development mode by executing the following command.

$ docker run -d --name=consul -e CONSUL_BIND_INTERFACE=eth0 -p 8500:8500 consul:1.7.2

Distributed configuration with Spring cloud

The next important element in our architecture is a configuration server. The most popular solution in Spring Cloud that provides mechanisms for distributed configuration is Spring Cloud Config. Spring Cloud Config provides server-side and client-side support for externalized configuration in a distributed system.
With the config server, you have a central place to manage external properties for applications across all environments. Some other solutions may be used in microservices-based architecture as a configuration server: Consul, ZooKeeper, or Alibaba Nacos. However, all these solutions are not strictly dedicated to distributed configuration, they can act as a discovery server as well.
Spring Cloud Config may integrate with different tools for storing data. The default implementation of the server storage backend uses git, but we can use some other tools like HashiCorp Vault for managing secrets and protecting sensitive data or just a simple file system. Such different backends may be together in the single config server application. We just need to activate the appropriate profiles in application properties with spring.profiles.active. We may override some default, for example, change the address of a Vault server or set an authentication token.

spring:
  application:
    name: config-server
  profiles:
    active: native,vault
  cloud:
    config:
      server:
        native:
          searchLocations: classpath:/config-repo
        vault:
          host: 192.168.99.100
          authentication: TOKEN
          token: spring-microservices-course

The same as for Consul we should use Docker to run an instance of Vault in development mode. We can set a static root token for authentication using the environment variable VAULT_DEV_ROOT_TOKEN_ID.

$ docker run -d --name vault --cap-add=IPC_LOCK -e 'VAULT_DEV_ROOT_TOKEN_ID=spring-microservices-course' -p 8200:8200 vault:1.4.0

When using Spring Cloud Config together with the discovery we may choose between two available approaches called Config First Bootstrap and Discovery First Bootstrap. In Discovery First Bootstrap a config server is registering itself in discovery service. Thanks to that each microservice can localize a config server basing on its registration id.
Since a configuration is injected in the bootstrap phase we need to use bootstrap.yml for setting properties on the client-side. To enable “discovering” for the config server on the client side we should set the property spring.cloud.config.discovery.enabled to true. We should also override registered service id of config server if it is different from auto-configured configserver (in our case it is config-server). Of course, we can also use Consul as a configuration properties source.

spring:
  application:
    name: callme-service
  cloud:
    config:
      discovery:
        enabled: true
        serviceId: config-server
    consul:
      host: 192.168.99.100
      config:
        format: YAML

Inter-service communication

Currently, there are three Spring components for inter-service communication over HTTP that have integration with service discovery: synchronous RestTemplate, reactive WebClient and declarative REST client OpenFeign. The RestTemplate component is available within the Spring Web module, WebClient within Spring WebFlux. To include Spring Cloud OpenFeign we need to include a dedicated starter.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

To use RestTemplate or WebClient for communication with discovery support, we need to register the beans and annotate them with @LoadBalanced. It is also worth setting the proper timeouts for such communication, especially if you are not using a circuit breaker.

@SpringBootApplication
@EnableFeignClients
class InterCallerServiceApplication {

    @Bean
    @LoadBalanced
    fun template(): RestTemplate = RestTemplateBuilder()
        .setReadTimeout(Duration.ofMillis(100))
        .setConnectTimeout(Duration.ofMillis(100))
        .build()

    @Bean
    @LoadBalanced
    fun clientBuilder(): WebClient.Builder {
        val tcpClient: TcpClient = TcpClient.create()
            .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 100)
            .doOnConnected { conn ->
                conn.addHandlerLast(ReadTimeoutHandler(100, TimeUnit.MILLISECONDS))
            }
        val connector = ReactorClientHttpConnector(HttpClient.from(tcpClient))
        return WebClient.builder().clientConnector(connector)
    }
    
}

Although Ribbon has been moved to maintenance mode almost 1.5 years ago it is still used as a default client-side load balancer in the newest stable version of Spring Cloud. Since Spring Cloud Load Balancer is included in common dependencies it is available in your application. Therefore, the only thing you need to do is to disable Ribbon in the configuration using spring.cloud.loadbalancer.ribbon.enabled property.
Currently, we don’t have many options for load balancer customization. One of them is the ability to configure client cache settings. By default, each client is caching the list of target services and refreshing them every 30 seconds. Such an interval may be too long in your situation.
We can easily change it in the configuration as shown below and set it to for example 1 second. If your load balancer is integrated with Eureka discovery you also need to decrease an interval of the fetching registry, which is by default 30 seconds. After those, both changes your client can refresh the list of currently running services almost immediately.

spring:
  cloud:
    loadbalancer:
      cache:
        ttl: 1s
      ribbon:
        enabled: false
eureka:
  client:
    registryFetchIntervalSeconds: 1

Circuit breaker

The circuit breaker is a popular design pattern used in a microservices architecture. It is designed to detect failures and encapsulates the logic of preventing a failure from constantly recurring. Spring Cloud provides an abstraction for using different circuit breaker implementations. Currently, we may use Netflix Hystrix, Sentinel, Spring Retry, and Resilience4J. To enable Spring Cloud Circuit Breaker based on Resilience4J we need to include the following dependency.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>

Here’s the code responsible for registering Customizer bean, that configures a circuit breaker behavior.

@Bean
fun defaultCustomizer(): Customizer<Resilience4JCircuitBreakerFactory> {
  return Customizer { factory: Resilience4JCircuitBreakerFactory ->
    factory.configureDefault { id: String? ->
      Resilience4JConfigBuilder(id)
        .timeLimiterConfig(TimeLimiterConfig.custom()
          .timeoutDuration(Duration.ofMillis(500))
          .build())
        .circuitBreakerConfig(CircuitBreakerConfig.custom()
          .slidingWindowSize(10)
          .failureRateThreshold(33.3F)
          .slowCallRateThreshold(33.3F)
        .build())
      .build()
    }
  }
}

The settings of the circuit breaker have been visualized in the picture below. The sliding window size sets the number of requests which are used for calculating the error rate. If we have more than 3 errors in the window of size 10 a circuit is open.

In the last step, we need to create a circuit breaker instance using Resilience4JCircuitBreakerFactory bean and enable it for the HTTP client as shown below.

@RestController
@RequestMapping("/caller")
class CallerController(private val template: RestTemplate, private val factory: Resilience4JCircuitBreakerFactory) {

  private var id: Int = 0

  @PostMapping("/random-send/{message}")
  fun randomSend(@PathVariable message: String): CallmeResponse? {
    val request = CallmeRequest(++id, message)
    val circuit = factory.create("random-circuit")
    return circuit.run { template.postForObject("http://inter-callme-service/callme/random-call",
      request, CallmeResponse::class.java) }
  }
    
}

Spring Cloud API gateway

The last missing element in our microservices architecture is an API Gateway. Spring Cloud Gateway is the project that helps us in implementing such a component. Currently, it is the second most popular Spring Cloud project just after Spring Cloud Netflix. It has around 2k stars on GitHub. It is built on top of the Spring WebFlux and Reactor project. It works reactively and requires Netty as a runtime framework.
The main goal of API gateway is to hide the complexity of the microservices system from an external client by providing an effective way of routing to APIs, but it can also solve some problems around security or resiliency. The main component used for configuring Spring Cloud Gateway is a route.
It is defined by an ID, a destination URI, a collection of predicates, and a collection of filters. A route is matched if the aggregate predicate is true. With filters, you can modify requests and responses before or after sending the downstream request.
With a predefined set of gateway filters, we may enable such mechanisms like path rewriting, rate limiting, discovery client, circuit breaking, fallback, or routing metrics. To enable all these features on the gateway we first need to include the following dependencies.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
   <groupId>org.jetbrains.kotlin</groupId>
   <artifactId>kotlin-reflect</artifactId>
</dependency>
<dependency>
   <groupId>org.jetbrains.kotlin</groupId>
   <artifactId>kotlin-stdlib</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-redis-reactive</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
</dependency>

To enable all the previously listed features we don’t have to implement much code. Almost everything is configurable in application properties.

spring:
  application:
    name: api-gateway
  cloud:
    gateway:
      discovery:
        locator:
          enabled: true 
          lowerCaseServiceId: true
      routes:
        - id: inter-callme-service
          uri: lb://inter-callme-service
          predicates:
            - Path=/api/callme/**
          filters:
            - RewritePath=/api(?/?.*), $\{path}
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 20
                redis-rate-limiter.burstCapacity: 40
            - name: CircuitBreaker
              args:
                name: sampleSlowCircuitBreaker
                fallbackUri: forward:/fallback/test
        - id: inter-caller-service
          uri: lb://inter-caller-service
          predicates:
            - Path=/api/caller/**
          filters:
            - StripPrefix=1
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 20
                redis-rate-limiter.burstCapacity: 40
    loadbalancer:
      ribbon:
        enabled: false
  redis:
    host: 192.168.99.100

management:
  endpoints.web.exposure.include: '*'
  endpoint:
    health:
      show-details: always

Some settings still need to be configured in the code. It is a configuration of the circuit breaker, that is based on Resilience4J project, where we need to register the bean Customizer. We also have to define a key for rate-limiting responsible setting a strategy of choosing requests for counting the limits.

@SpringBootApplication
class ApiGatewayApplication {

  @Bean
  fun keyResolver(): KeyResolver = KeyResolver { _ -> Mono.just("1") }

  @Bean
  fun defaultCustomizer(): Customizer<ReactiveResilience4JCircuitBreakerFactory> {
    return Customizer { factory: ReactiveResilience4JCircuitBreakerFactory ->
      factory.configureDefault { id: String? ->
        Resilience4JConfigBuilder(id)
          .timeLimiterConfig(TimeLimiterConfig.custom()
            .timeoutDuration(Duration.ofMillis(500))
            .build())
          .circuitBreakerConfig(CircuitBreakerConfig.custom()
            .slidingWindowSize(10)
            .failureRateThreshold(33.3F)
            .slowCallRateThreshold(33.3F)
            .build())
          .build()
      }
    }
  }

}

fun main(args: Array<String>) {
   runApplication<ApiGatewayApplication>(*args)
}

Conclusion

In this article, you may take a quick introduction to using the latest Spring Cloud components for building microservices architecture. For more details, you may refer to my video course published on YouTube.

The post A New Era Of Spring Cloud appeared first on Piotr's TechBlog.

1. Dependencies

To succesfully test some scenarios of using a circuit breaker pattern with Spring Cloud Gateway we need to include a reactive version of Spring Cloud Circuit Breaker since gateway is started on reactive Netty server. We will simulate downstream service using MockServer provided within the Testcontainers framework. It is provisioned inside the test by a mock client written in Java.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-cloud-starter-circuitbreaker-reactor-resilience4j</artifactId>
</dependency>
<dependency>
   <groupId>org.projectlombok</groupId>
   <artifactId>lombok</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.testcontainers</groupId>
   <artifactId>mockserver</artifactId>
   <version>1.12.3</version>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.mock-server</groupId>
   <artifactId>mockserver-client-java</artifactId>
   <version>3.10.8</version>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>com.carrotsearch</groupId>
   <artifactId>junit-benchmarks</artifactId>
   <version>0.7.2</version>
   <scope>test</scope>
</dependency>

2. Enabling Spring Cloud Gateway Circuit Breaker with Resilience4J

To enable circuit breaker built on top of Resilience4J we need to declare a Customizer bean that is passed a ReactiveResilience4JCircuitBreakerFactory. The very simple configuration contains default circuit breaker settings and and defines timeout duration using TimeLimiterConfig. For the first test I decided to set 200 milliseconds timeout.

@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
    return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
        .circuitBreakerConfig(CircuitBreakerConfig.ofDefaults())
        .timeLimiterConfig(TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(200)).build())
        .build());
}

3. Building Test Class

In the next step we are creating a test class. Before running the test it is starting and provisioning an instance of mock server. We are defining two endpoints. The second of them /2 adds a delay of 200 milliseconds, which exceeds the timeout defined in the circuit breaker configuration.
We are also setting configuration of Spring Cloud Gateway route which is addressed to the currently started instance of mock server. To enable the circuit breaker for our route we have to define a CircuitBreaker filter with a given name. The test is repeated 200 times. It calls the delayed and not delayed endpoint in 50/50 proportion. Here’s the Spring Cloud Gateway test class.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
@RunWith(SpringRunner.class)
public class GatewayCircuitBreakerTest {

    private static final Logger LOGGER = LoggerFactory.getLogger(GatewayRateLimiterTest.class);

    @Rule
    public TestRule benchmarkRun = new BenchmarkRule();

    @ClassRule
    public static MockServerContainer mockServer = new MockServerContainer();

    @Autowired
    TestRestTemplate template;
    int i = 0;

    @BeforeClass
    public static void init() {
        System.setProperty("spring.cloud.gateway.routes[0].id", "account-service");
        System.setProperty("spring.cloud.gateway.routes[0].uri", "http://192.168.99.100:" + mockServer.getServerPort());
        System.setProperty("spring.cloud.gateway.routes[0].predicates[0]", "Path=/account/**");
        System.setProperty("spring.cloud.gateway.routes[0].filters[0]", "RewritePath=/account/(?.*), /$\\{path}");
        System.setProperty("spring.cloud.gateway.routes[0].filters[1].name", "CircuitBreaker");
        System.setProperty("spring.cloud.gateway.routes[0].filters[1].args.name", "exampleSlowCircuitBreaker");
        MockServerClient client = new MockServerClient(mockServer.getContainerIpAddress(), mockServer.getServerPort());
        client.when(HttpRequest.request()
            .withPath("/1"))
            .respond(response()
                .withBody("{\"id\":1,\"number\":\"1234567890\"}")
                .withHeader("Content-Type", "application/json"));
        client.when(HttpRequest.request()
            .withPath("/2"))
            .respond(response()
                .withBody("{\"id\":2,\"number\":\"1234567891\"}")
                .withDelay(TimeUnit.MILLISECONDS, 200)
                .withHeader("Content-Type", "application/json"));
    }

    @Test
    @BenchmarkOptions(warmupRounds = 0, concurrency = 1, benchmarkRounds = 200)
    public void testAccountService() {
        int gen = 1 + (i++ % 2);
        ResponseEntity r = template.exchange("/account/{id}", HttpMethod.GET, null, Account.class, gen);
        LOGGER.info("{}. Received: status->{}, payload->{}, call->{}", i, r.getStatusCodeValue(), r.getBody(), gen);
    }

}

Here’s the result of the currently discussed test. With default settings it opens the circuit after processing 100 requests with 50% error rate. The logs visible below include a sequence number of requests, HTTP response status code, response body and URL of the called endpoint.

We may change the default settings a little. To do that we should define a custom CircuitBreakerConfig. One of the properties we can customize is slidingWindowSize. The property slidingWindowSize defines how many outcome calls has to be recorded when a circuit breaker is closed. Assuming we have the same test endpoints what will happen if we change this value to 10 as shown below?

@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
    return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
        .circuitBreakerConfig(CircuitBreakerConfig.custom()
            .slidingWindowSize(10)
            .build())
        .timeLimiterConfig(TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(200)).build()).build());
}

Here’s the result. The circuit is open just after processing 10 requests when at least 50% of them are timeouted.

Moreover, we may change failureRateThreshold. This property is responsible for configuring the failure rate threshold in percentage. If the failure rate is equal or greater than the threshold the circuit breaker is switched to open and starts short-circuiting calls. It is not difficult to predict what will happen if we change it for our current scenario to 66.6F. The circuit will never be opened.

@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
    return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
        .circuitBreakerConfig(CircuitBreakerConfig.custom()
            .slidingWindowSize(10)
            .failureRateThreshold(66.6F)
            .build())
        .timeLimiterConfig(TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(200)).build()).build());
}

4. Spring Cloud Gateway circuit breaker Customization

We were starting with really basic samples. Let’s do something more interesting! First, we will set a really small value of sliding window size. It is set to 5. Thanks to that we will be able to observe the full result of the current test scenario after processing only a few requests. The next step is to modify the rules defined on the mock server. Now, we will delay only 5 first requests sent to the /2 endpoint. After receiving 5 requests it starts to work fine without adding any delay. Thanks to that fact our circuit breaker would be able to back from OPEN state to CLOSE after some time. But first things first, here are the code defining mock endpoints for the current test.

MockServerClient client = new MockServerClient(mockServer.getContainerIpAddress(), mockServer.getServerPort());
client.when(HttpRequest.request()
   .withPath("/1"))
   .respond(response()
      .withBody("{\"id\":1,\"number\":\"1234567890\"}")
      .withHeader("Content-Type", "application/json"));
client.when(HttpRequest.request()
   .withPath("/2"), Times.exactly(5))
   .respond(response()
      .withBody("{\"id\":2,\"number\":\"1234567891\"}")
      .withDelay(TimeUnit.MILLISECONDS, 200)
      .withHeader("Content-Type", "application/json"));
client.when(HttpRequest.request()
   .withPath("/2"))
   .respond(response()
      .withBody("{\"id\":2,\"number\":\"1234567891\"}")
      .withHeader("Content-Type", "application/json"));

As I mentioned before the slidingWindowSize is now equal to 5. If there are 3 timeouts during the last 5 calls the circuit is switched to OPEN state. We can configure how long the circuit should stay in the OPEN state without trying to process any request. The parameter waitDurationInOpenState, which is responsible for that, has been set to 30 milliseconds. Therefore, after 30 milliseconds the circuit is switched to HALF_OPEN state, which means that the incoming requests are processed again. We can also configure a number of permitted calls in the HALF_OPEN state. The property permittedNumberOfCallsInHalfOpenState is set to 5 instead of default value 10. In these five attempts, we get only 2 timeouts, since we set 5 repeats for delayed service on the mock server and the first 3 timeouts have been in the beginning before opening a circuit. Here’s our current configuration of Spring Cloud Circuit Breaker.

@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
   return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
      .circuitBreakerConfig(CircuitBreakerConfig.custom()
         .slidingWindowSize(5)
         .permittedNumberOfCallsInHalfOpenState(5)
         .failureRateThreshold(50.0F)
         .waitDurationInOpenState(Duration.ofMillis(30))
         .build())
      .timeLimiterConfig(TimeLimiterConfig.custom().timeoutDuration(Duration.ofMillis(200)).build()).build());
}

The following diagram illustrates our scenario.

And here’s the result of our current test. The circuit has been opened after processing 6 requests. There were 3 incoming requests that had not been processed during 30 milliseconds of being in open state. After that time it has been switched to half open state and finally it moved back to close state.

What will happen if we increase the number of delayed requests in this scenario to 20?

client.when(HttpRequest.request()
   .withPath("/2"), Times.exactly(20))
   .respond(response()
      .withBody("{\"id\":2,\"number\":\"1234567891\"}")
      .withDelay(TimeUnit.MILLISECONDS, 200)
      .withHeader("Content-Type", "application/json"));

The circuit will be switched between OPEN and HALF_OPEN state until the downstream service is delaying the responses.

5. Adding Fallback

As you probably noticed, if the request to the downstream service has been finished with timeout the gateway returns HTTP status HTTP 504 - Gateway Timeout. Moreover, if a circuit is open the gateway is returning HTTP Status HTTP 503 - Service Unavailable. To prevent from returning error status code on the gateway we may enable fallback endpoint for our route. To do that we have to set property fallbackUri using forward: scheme. Here’s the current configuration of the test route. I included the endpoint /fallback/account as fallback URI.

System.setProperty("spring.cloud.gateway.routes[0].id", "account-service");
System.setProperty("spring.cloud.gateway.routes[0].uri", "http://192.168.99.100:" + mockServer.getServerPort());
System.setProperty("spring.cloud.gateway.routes[0].predicates[0]", "Path=/account/**");
System.setProperty("spring.cloud.gateway.routes[0].filters[0]", "RewritePath=/account/(?.*), /$\\{path}");
System.setProperty("spring.cloud.gateway.routes[0].filters[1].name", "CircuitBreaker");
System.setProperty("spring.cloud.gateway.routes[0].filters[1].args.name", "exampleSlowCircuitBreaker");
System.setProperty("spring.cloud.gateway.routes[0].filters[1].args.fallbackUri", "forward:/fallback/account");

The fallback endpoint is exposed on the gateway. I defined a simple controller class that implements a single fallback method.

@RestController
@RequestMapping("/fallback")
public class GatewayFallback {

    @GetMapping("/account")
    public Account getAccount() {
        Account a = new Account();
        a.setId(2);
        a.setNumber("123456");
        return a;
    }

}

Assuming we have exactly the same scenario as in the previous section the current test is returning only HTTP 200 instead of responses with HTTP 5xx as shown below.

6. Handling Slow Responses

In all previous examples we have set a short timeout on response, what results in HTTP 504 - Gateway Timeout or fallback. However, we don’t have to timeout the requests, but we can just set a threshold and failure rate for indicating slow responses. There are two parameters responsible for that: slowCallDurationThreshold and slowCallRateThreshold.

@Bean
public Customizer<ReactiveResilience4JCircuitBreakerFactory> defaultCustomizer() {
    return factory -> factory.configureDefault(id -> new Resilience4JConfigBuilder(id)
        .circuitBreakerConfig(CircuitBreakerConfig.custom()
           .slidingWindowSize(5)
           .permittedNumberOfCallsInHalfOpenState(5)
           .failureRateThreshold(50.0F)
           .waitDurationInOpenState(Duration.ofMillis(50))
           .slowCallDurationThreshold(Duration.ofMillis(200))
           .slowCallRateThreshold(50.0F)
           .build())
        .build());
}

Now, the delayed responses are not finished with timeout, however the circuit breaker is still recording these records. When the threshold is exceeded the circuit breaker is open as shown below.

The post Circuit Breaking In Spring Cloud Gateway With Resilience4J appeared first on Piotr's TechBlog.