Source code

If you would like to try it by yourself, you may always take a look at my source code example. In order to do that you need to clone my repository sample-spring-microservices-kubernetes. Then just follow my instructions The good idea is to read the article Microservices with Spring Cloud Kubernetes before you move on.

Step 1. Enable Spring Cloud Kubernetes Load Balancer

You need to include a single Spring Boot Starter to enable Spring Cloud Kubernetes Load Balancer. It is spring-cloud-starter-kubernetes-loadbalancer. Consequently, you also need to have a REST client on the classpath. Spring RestTemplate is automatically included with the Spring Web module. We will also use the OpenFeign client, so you should include the right starter.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-loadbalancer</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

Step 2. Implement and use the REST client

Spring Cloud OpenFeign is a declarative REST client. Therefore, you need to create an interface with methods and Spring MVC annotations. It is important to set the right name inside @FeignClient annotation. This name needs to be the same as the name of the target Kubernetes Service. In the following code example, you see an implementation of the employee service client inside department-service.

@FeignClient(name = "employee")
public interface EmployeeClient {

   @GetMapping("/department/{departmentId}")
   List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
	
}

OpenFeign client is enabled after annotating the main class with @EnableFeignClients. After that you can inject it to the @RestController class. Finally, you may use its method findByDepartment to find employees assigned to the particular department.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;

    public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient) {
        this.repository = repository;
        this.employeeClient = employeeClient;
    }

    @GetMapping("/{id}/with-employees")
    public Department findByIdWithEmployees(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            department.setEmployees(employeeClient.findByDepartment(department.getId()));
            return department;
        }
        return null;
    }

    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Opposite to the OpenFeign, the RestTemplate is a low-level HTTP client. We need to enable Spring Cloud load balancing for it. To do that just annotate RestTemplate bean with @LoadBalanced.

@Bean
@LoadBalanced
RestTemplate restTemplate() {
    return new RestTemplate();
}

Here’s the similar implementation of @RestController class, but this time with the RestTemplate.

public class DepartmentWithRestTemplateController {


    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    RestTemplate restTemplate;

    public DepartmentWithRestTemplateController(DepartmentRepository repository, RestTemplate restTemplate) {
        this.repository = repository;
        this.restTemplate = restTemplate;
    }

    @GetMapping("/{id}/with-employees")
    public Department findByIdWithEmployees(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            department.setEmployees(findEmployeesByDepartment(department.getId()));
            return department;
        }
        return null;
    }


    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(findEmployeesByDepartment(d.getId())));
        return departments;
    }

    private List<Employee> findEmployeesByDepartment(String departmentId) {
        Employee[] employees = restTemplate
                .getForObject("http://employee//department/{departmentId}", Employee[].class, departmentId);
        return Arrays.asList(employees);
    }

}

Step 3. Deploy Spring Cloud applications on Kubernetes

The project is ready to be used with Skaffold. Therefore, you don’t have to worry about the deployment process. You just need to run a single command skaffold dev --port-forward to deploy our applications on Kubernetes. But before deploying them on the cluster, we perform a short overview of the communication process. We have three microservices. Each of them is running in two instances. They are using Mongo as a backend store. There is also the gateway application. It is built on top of Spring Cloud Gateway. It provides a single API endpoint to all downstream services. Of course, all our applications are using Spring Cloud Load Balancer for traffic management. To clarify, you can take a look at the picture below.

Here’s the Kubernetes deployment manifest for the employee-service. It sets two running pods in replicas parameter, and references ConfigMap and Secret to inject Mongo credentials.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: employee
  labels:
    app: employee
spec:
  replicas: 2
  selector:
    matchLabels:
      app: employee
  template:
    metadata:
      labels:
        app: employee
    spec:
      containers:
      - name: employee
        image: piomin/employee
        ports:
        - containerPort: 8080
        env:
        - name: MONGO_DATABASE
          valueFrom:
            configMapKeyRef:
              name: mongodb
              key: database-name
        - name: MONGO_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-user
        - name: MONGO_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-password

The most important element in the communication between applications is Kubernetes Service object. The name of service must be the same as the hostname used by the RestTemplate or OpenFeign client. In our case that name is employee.

apiVersion: v1
kind: Service
metadata:
  name: employee
  labels:
    app: employee
    spring-boot: "true"
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: employee
  type: ClusterIP

Let’s verify the status after deploying all the applications. Here’s the list of running pods.

Here’s the list of Kubernetes Endpoints.

Step 4. Communication in the POD mode

By default, Spring Cloud Kubernetes Load Balancer uses the POD mode. In this mode, it gets the list of Kubernetes endpoints to detect the IP address of all the application pods. In that case, the only thing you need to do is to disable the Ribbon load balancer. Spring Cloud Kubernetes is still using it as a default client-side load balancer. To disable it you need to set the following property.

spring:
  cloud:
    loadbalancer:
      ribbon:
        enabled: false

After adding a test data we may send some requests to the endpoint GET /{id}/with-employees. It finds a department by the id. Then it communicates with the employee-service endpoint GET /department/{departmentId} to search all the employees assigned to the current department. The department-service is exposed on port 8081, since I enabled option port-forward on Skaffold.

$ curl http://localhost:8081/5f5896b3cb8caf7e6f6b9e1c/with-employees
{"id":"5f5896b3cb8caf7e6f6b9e1c","organizationId":"1","name":"test1","employees":[{"id":"5f5896e26092716e54f60a9a","name":"test1","age":30,"position":"d
eveloper"},{"id":"5f5896f29625e62c7d373906","name":"test2","age":40,"position":"tester"},{"id":"5f5897266092716e54f60a9b","name":"test3","age":45,"posit
ion":"developer"}]}

Let’s take a look on the logs from employee-service. I repeated the request visible above several times. The requests are load balanced between two available instances of employee-service. Skaffold prints the id of every pod in the logs, so you can verify that everything works fine.

Step 5. Communication across multiple namespaces

By default, Spring Cloud Kubernetes allows load balancing within the same namespace. You may enable discovery across multiple namespaces. To do that you need to use the property following property.

spring:
  cloud:
    kubernetes:
      discovery:
        all-namespaces: true

After setting that, we will deploy the employee-service application in the different namespace than department-service. To do that you need to set parameter -n on the skaffold command.

$ skaffold dev -n test-a

Here’s the current list of running pods.

You just need to send the same request as before curl http://localhost:8081/5f5896b3cb8caf7e6f6b9e1c/with-employees. Of course, the Kubernetes service name must be unique across all the namespaces.

Step 6. Load balancer on Spring Cloud Gateway

Spring Cloud Gateway uses the same load balancing mechanism as other Spring Cloud applications. To enable it on Kubernetes we need to include dependency spring-cloud-starter-kubernetes-loadbalancer. We should also enable Spring Cloud DiscoveryClient and disable Ribbon. Here’s the configuration of the gateway-service.

spring:
  application:
    name: gateway
  cloud:
    loadbalancer:
      ribbon:
        enabled: false
    gateway:
      discovery:
        locator:
          enabled: true
      routes:
        - id: employee
          uri: lb://employee
          predicates:
            - Path=/employee/**
          filters:
            - RewritePath=/employee/(?.*), /$\{path}
        - id: department
          uri: lb://department
          predicates:
            - Path=/department/**
          filters:
            - RewritePath=/department/(?.*), /$\{path}
        - id: organization
          uri: lb://organization
          predicates:
            - Path=/organization/**
          filters:
            - RewritePath=/organization/(?.*), /$\{path}

A gateway acts as an entry point to our system. It performs routing and load balancing to all the downstream services. Therefore, we can call any of our applications using already defined routes. Since gateway-service is available on port 8080, I can call any of the endpoints using the following requests.

We can expose API documentation of all the microservices on the gateway. To do that we may use the SpringFox project. First, we need to include SpringFox Starter to Maven dependencies.

<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-boot-starter</artifactId>
   <version>3.0.0</version>
</dependency>

Swagger documentation is usually generated for a single application. Because we want to expose documentation of all our microservices we need to override the Swagger resource provider. The custom implementation is visible below. It uses a discovery mechanism to the names of running services.

@Configuration
public class GatewayApi {

   @Autowired
   RouteDefinitionLocator locator;

   @Primary
   @Bean
   public SwaggerResourcesProvider swaggerResourcesProvider() {
      return () -> {
         List<SwaggerResource> resources = new ArrayList<>();
         Flux<RouteDefinition> definitions = locator.getRouteDefinitions();
         definitions
               .filter(routeDefinition -> !routeDefinition.getId().startsWith("ReactiveCompositeDiscoveryClient_"))
               .subscribe(routeDefinition -> resources.add(createResource(routeDefinition.getId(), "2.0")));
         try {
            Thread.sleep(1000);
         } catch (InterruptedException e) {
            e.printStackTrace();
         }
         return resources;
      };
   }

   private SwaggerResource createResource(String location, String version) {
      SwaggerResource swaggerResource = new SwaggerResource();
      swaggerResource.setName(location);
      swaggerResource.setLocation("/" + location + "/v2/api-docs");
      swaggerResource.setSwaggerVersion(version);
      return swaggerResource;
   }

}

Here’s the Swagger UI available at http://localhost:8080/swagger-ui/index.html on my local machine.

Step 7. Enabling circuit breaker

We can use a circuit breaker component with Spring Cloud Load Balancer. The default Spring Cloud implementation is based on Resilience4j. In order to enable it for your application, you need to include the following dependency.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-circuitbreaker-resilience4j</artifactId>
</dependency>

The next step is to provide a configuration of a circuit breaker.

@Bean
Customizer<Resilience4JCircuitBreakerFactory> defaultCustomizer() {
   return factory -> factory.configureDefault(id ->
         new Resilience4JConfigBuilder(id)
               .timeLimiterConfig(TimeLimiterConfig.custom()
                     .timeoutDuration(Duration.ofMillis(1000))
                     .build())
               .circuitBreakerConfig(CircuitBreakerConfig.custom()
                     .slidingWindowSize(10)
                     .failureRateThreshold(66.6F)
                     .slowCallRateThreshold(66.6F)
                     .build())
               .build()
   );
}

We need to inject Resilience4JCircuitBreakerFactory to the Spring controller class. Then we are creating the circuit breaker instance using an injected factory. Finally, the client calling method is running inside the circuit breaker run method.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;
    Resilience4JCircuitBreakerFactory circuitBreakerFactory;

    public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient,
                                Resilience4JCircuitBreakerFactory circuitBreakerFactory) {
        this.repository = repository;
        this.employeeClient = employeeClient;
        this.circuitBreakerFactory = circuitBreakerFactory;
    }

    @GetMapping("/{id}/with-employees-and-delay")
    public Department findByIdWithEmployeesAndDelay(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Optional<Department> optDepartment = repository.findById(id);
        if (optDepartment.isPresent()) {
            Department department = optDepartment.get();
            Resilience4JCircuitBreaker circuitBreaker = circuitBreakerFactory.create("delayed-circuit");
            List<Employee> employees = circuitBreaker.run(() ->
                    employeeClient.findByDepartmentWithDelay(department.getId()));
            department.setEmployees(employees);
            return department;
        }
        return null;
    }
}

Conclusion

Load balancing is one of the key patterns in a microservices architecture. Spring Cloud Load Balancer is replacing the Ribbon client. By default, load balancing in Kubernetes is based on Services. Therefore, you need to use additional tools for more advanced routing mechanisms. Spring Cloud Kubernetes comes with some interesting features. One of them is the ability to load balance across multiple namespaces. You can also use the additional Spring Cloud components like a circuit breaker. In comparison with tools like Istio, it is still not much. Is it a chance for improvement? We will see. Nevertheless, Spring Cloud Kubernetes is currently one of the most popular Spring Cloud projects. It may be a good choice if you are migrating your Spring Cloud microservices architecture to Kubernetes. Support for the load balancer may be an important step during such a migration.

The post Spring Cloud Kubernetes Load Balancer Guide appeared first on Piotr's TechBlog.

Source code

The source code is available on GitHub. If you want to clone the repository or just give me a star go here

Building applications with Spring Cloud Kubernetes

Spring Cloud Kubernetes provides implementations of well known Spring Cloud components based on Kubernetes API. It includes a discovery client, load balancer, and property sources support. We should add the following Maven dependency to enable it in our project.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
</dependency>

Our application connects to the Mongo database, exposes REST API, and communicates with other applications over HTTP. Therefore we need to include some additional dependencies.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

The overview of our system is visible in the picture below. We need to mock communication between applications and with Kubernetes API. We will also run an embedded in-memory Mongo database during tests. For more details about building microservices with Spring Cloud Kubernetes read the following article.

Testing API with Kubernetes MockServer

First, we need to include a Spring Boot Test starter, that contains basic dependencies used for JUnit tests implementation. Since our application is connected to Mongo and Kubernetes API, we should also mock them during the test. Here’s the full list of required dependencies.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>de.flapdoodle.embed</groupId>
    <artifactId>de.flapdoodle.embed.mongo</artifactId>
    <scope>test</scope>
</dependency>
<dependency>
    <groupId>io.fabric8</groupId>
    <artifactId>kubernetes-server-mock</artifactId>
    <version>4.10.3</version>
    <scope>test</scope>
</dependency>

Let’s discuss what exactly is happening during our test.
(1) First, we are enabling fabric8 Kubernetes Client JUnit5 extension in CRUD mode. It means that we can create a Kubernetes object on the mocked server.
(2) Then the KubernetesClient is injected to the test by the JUnit5 extension.
(3) TestRestTemplate is able to call endpoints exposed by the application that is started during the test.
(4) We need to set the basic properties for KubernetesClient like a default namespace name, master URL.
(5) We are creating ConfigMap that contains application.properties file. ConfigMap with name employee is automatically read by the application employee.
(6) In the test method we are using TestRestTemplate to call REST endpoints. We are mocking Kubernetes API and running Mongo database in the embedded mode.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@EnableKubernetesMockClient(crud = true) // (1)
@TestMethodOrder(MethodOrderer.Alphanumeric.class)
class EmployeeAPITest {

    static KubernetesClient client; // (2)

    @Autowired
    TestRestTemplate restTemplate; // (3)

    @BeforeAll
    static void init() {
        System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY,
            client.getConfiguration().getMasterUrl());
        System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY,
            "true");
        System.setProperty(
            Config.KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");
        System.setProperty(
            Config.KUBERNETES_AUTH_TRYSERVICEACCOUNT_SYSTEM_PROPERTY, "false");
        System.setProperty(Config.KUBERNETES_HTTP2_DISABLE, "true");
        System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY,
            "default"); // (4)
        client.configMaps().inNamespace("default").createNew()
            .withNewMetadata().withName("employee").endMetadata()
            .addToData("application.properties",
                "spring.data.mongodb.uri=mongodb://localhost:27017/test")
            .done(); // (5)
    }

    @Test // (6)
    void addEmployeeTest() {
        Employee employee = new Employee(1L, 1L, "Test", 30, "test");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        Assertions.assertNotNull(employee);
        Assertions.assertNotNull(employee.getId());
    }

    @Test
    void addAndThenFindEmployeeByIdTest() {
        Employee employee = new Employee(1L, 2L, "Test2", 20, "test2");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        Assertions.assertNotNull(employee);
        Assertions.assertNotNull(employee.getId());
        employee = restTemplate
            .getForObject("/{id}", Employee.class, employee.getId());
        Assertions.assertNotNull(employee);
        Assertions.assertNotNull(employee.getId());
    }

    @Test
    void findAllEmployeesTest() {
        Employee[] employees =
            restTemplate.getForObject("/", Employee[].class);
        Assertions.assertEquals(2, employees.length);
    }

    @Test
    void findEmployeesByDepartmentTest() {
        Employee[] employees =
            restTemplate.getForObject("/department/1", Employee[].class);
        Assertions.assertEquals(1, employees.length);
    }

    @Test
    void findEmployeesByOrganizationTest() {
        Employee[] employees =
            restTemplate.getForObject("/organization/1", Employee[].class);
        Assertions.assertEquals(2, employees.length);
    }

}

Integration Testing on Kubernetes with Hoverfly

To test HTTP communication between applications we usually need to use an additional tool for mocking API. Hoverfly is an ideal solution for such a use case. It is a lightweight, open-source API simulation tool not only for REST-based applications. It allows you to write tests in Java and Python. In addition, it also supports JUnit5. You need to include the following dependencies to enable it in your project.

<dependency>
	<groupId>io.specto</groupId>
	<artifactId>hoverfly-java-junit5</artifactId>
	<version>0.13.0</version>
	<scope>test</scope>
</dependency>
<dependency>
	<groupId>io.specto</groupId>
	<artifactId>hoverfly-java</artifactId>
	<version>0.13.0</version>
	<scope>test</scope>
</dependency>

You can enable Hoverfly in your tests with @ExtendWith annotation. It automatically starts Hoverfly proxy during a test. Our main goal is to mock the Kubernetes client. To do that we still need to set some properties inside @BeforeAll method. The default URL used by KubernetesClient is kubernetes.default.svc. In the first step, we are mocking configmap endpoint and returning predefined Kubernetes ConfigMap with application.properties. The name of ConfigMap is the same as the application name. We are testing communication from the department application to the employee application.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ExtendWith(HoverflyExtension.class)
public class DepartmentAPIAdvancedTest {

    @Autowired
    KubernetesClient client;

    @BeforeAll
    static void setup(Hoverfly hoverfly) {
        System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, "true");
        System.setProperty(Config.KUBERNETES_AUTH_TRYKUBECONFIG_SYSTEM_PROPERTY, "false");
        System.setProperty(Config.KUBERNETES_AUTH_TRYSERVICEACCOUNT_SYSTEM_PROPERTY,
            "false");
        System.setProperty(Config.KUBERNETES_HTTP2_DISABLE, "true");
        System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, "default");
        hoverfly.simulate(dsl(service("kubernetes.default.svc")
            .get("/api/v1/namespaces/default/configmaps/department")
            .willReturn(success().body(json(buildConfigMap())))));
    }

    private static ConfigMap buildConfigMap() {
        return new ConfigMapBuilder().withNewMetadata()
            .withName("department").withNamespace("default")
            .endMetadata()
            .addToData("application.properties",
                "spring.data.mongodb.uri=mongodb://localhost:27017/test")
            .build();
    }
	
    // TESTS ...
	
}

After application startup, we may use TestRestTemplate to call a test endpoint. The endpoint GET /organization/{organizationId}/with-employees retrieves data from the employee application. It finds the department by organization id and then finds all employees assigned to the department. We need to mock a target endpoint using Hoverfly. But before that, we are mocking Kubernetes APIs responsible for getting service and endpoint by name. The address and port returned by the mocked endpoints must be the same as the address of a target application endpoint.

@Autowired
TestRestTemplate restTemplate;

private final String EMPLOYEE_URL = "employee.default:8080";

@Test
void findByOrganizationWithEmployees(Hoverfly hoverfly) {
    Department department = new Department(1L, "Test");
    department = restTemplate.postForObject("/", department, Department.class);
    Assertions.assertNotNull(department);
    Assertions.assertNotNull(department.getId());

    hoverfly.simulate(
        dsl(service(prepareUrl())
            .get("/api/v1/namespaces/default/endpoints/employee")
            .willReturn(success().body(json(buildEndpoints())))),
        dsl(service(prepareUrl())
            .get("/api/v1/namespaces/default/services/employee")
            .willReturn(success().body(json(buildService())))),
        dsl(service(EMPLOYEE_URL)
            .get("/department/" + department.getId())
            .willReturn(success().body(json(buildEmployees())))));

    Department[] departments = restTemplate
        .getForObject("/organization/{organizationId}/with-employees", Department[].class, 1L);
    Assertions.assertEquals(1, departments.length);
    Assertions.assertEquals(1, departments[0].getEmployees().size());
}

private Service buildService() {
    return new ServiceBuilder().withNewMetadata().withName("employee")
            .withNamespace("default").withLabels(new HashMap<>())
            .withAnnotations(new HashMap<>()).endMetadata().withNewSpec().addNewPort()
            .withPort(8080).endPort().endSpec().build();
}

private Endpoints buildEndpoints() {
    return new EndpointsBuilder().withNewMetadata()
        .withName("employee").withNamespace("default")
        .endMetadata()
        .addNewSubset().addNewAddress()
        .withIp("employee.default").endAddress().addNewPort().withName("http")
        .withPort(8080).endPort().endSubset()
        .build();
}

private List<Employee> buildEmployees() {
    List<Employee> employees = new ArrayList<>();
    Employee employee = new Employee();
    employee.setId("abc123");
    employee.setAge(30);
    employee.setName("Test");
    employee.setPosition("test");
    employees.add(employee);
    return employees;
}

private String prepareUrl() {
    return client.getConfiguration().getMasterUrl()
        .replace("/", "")
        .replace("https:", "");
}

Conclusion

The approach described in this article allows you to create integration tests without running a Kubernetes instance. On the other hand, you could start a single-node Kubernetes instance like Microk8s and deploy your application there. You could as well use an existing cluster, and implement your tests with Arquillian Cube. It is able to communicate directly to the Kubernetes API.
Another key point is testing communication between applications. In my opinion, Hoverfly is the best tool for that. It is able to mock the whole traffic over HTTP in the single test. With Hoverfly, fabric8 and Spring Cloud you can improve your integration testing on Kubernetes.

The post Integration Testing on Kubernetes with JUnit5 appeared first on Piotr's TechBlog.

Source Code

By default, Spring Cloud Kubernetes also does not support ConfigMap or Secret versioning. That feature is available in my forked version of this library. For accessing it go to my GitHub repository: https://github.com/piomin/spring-cloud-kubernetes.

Concept

The concept around Kubernetes ConfigMap versioning is pretty easy. Especially with Spring Cloud Kubernetes. We are creating ConfigMap that needs to be labelled with app and version. The label app contains the name of Spring Boot application, that is configured by property spring.application.name. The label version indicates a number of application version, which is fetched by the library from property info.app.version. Both these properties need to be available inside application.yml orbootstrap.yml. Here’s sample version 1.0 of ConfigMap for application api-test.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-test-1
  labels:
    version: "1.0"
    app: api-test
data:
  application.yaml: |-
    property1: v1.0

We may maintain many versions of ConfigMap used by a given application just by changing name, labels and content. Here’s version 1.1 of ConfigMap for the same api-test application.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-test-2
  labels:
    version: "1.1"
    app: api-test
data:
  application.yaml: |-
    property1: v1.1

By default, Spring Cloud Kubernetes Config is able to inject ConfigMap to the application based on its metadata.name. I redefined this mechanism to base on the label app. Let’s take a look at some details.

Implementation

Now the question is – how to use that feature in our application. Of course, you need to include the required library to your Maven dependencies. Remember to use my forked version Spring Cloud Kubernetes Config instead of the official one.

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-kubernetes-config</artifactId>
</dependency>

Then the bootstrap.yaml file in your application needs to contains at least the following properties to enable ConfigMap versioning mechanism.

spring:
  application:
    name: api-test
  cloud:
    kubernetes:
      config:
        enableVersioning: true
info:
  app:
    version: 1.0

Let’s take a look on the key fragment of versioning mechanism implementation inside Spring Cloud Kubernetes Config. If it is not enabled with property spring.cloud.kubernetes.config.enableVersioning it works based on metadata.name – the same as before changes. We are also checking if property info.app.version is set inside the application settings. If not, the versioning mechanism is being disabled. Otherwise we are checking all ConfigMaps in the selected namespace, and filtering them by app and version labels.

String version = environment.getProperty("info.app.version");
LOG.info("Get Config: versioning->" + enableVersioning + ", name->" + name + ", version->" + version);
Map result = new HashMap<>();
ConfigMap map = null;
if (!enableVersioning || version == null) {
   map = StringUtils.isEmpty(namespace) ? client.configMaps().withName(name).get()
         : client.configMaps().inNamespace(namespace).withName(name).get();
} else {
   Optional optMap = StringUtils.isEmpty(namespace)
      ? client.configMaps().list().getItems().stream()
         .filter(configMap -> configMap.getMetadata().getLabels()
            .containsKey("app")
               && configMap.getMetadata().getLabels().get("app").equals(name)
               && configMap.getMetadata().getLabels().containsKey("version")
               && configMap.getMetadata().getLabels().get("version").equals(version))
            .findFirst()
      : client.configMaps().inNamespace(namespace).list().getItems().stream()
         .filter(configMap -> configMap.getMetadata().getLabels()
            .containsKey("app")
               && configMap.getMetadata().getLabels().get("app").equals(name)
               && configMap.getMetadata().getLabels().containsKey("version")
               && configMap.getMetadata().getLabels().get("version").equals(version))
            .findFirst();
   if (optMap.isPresent()) {
      map = optMap.get();
   }
}

Demo of Kubernetes ConfigMap versioning

Our sample Spring Boot application is very simple. It just exposes a single HTTP endpoint that displays the value of property injected from ConfigMap as shown below.

@RestController
public class ApiController {

   @Value("${property1}")
   private String property1;

   @GetMapping("/property")
   public String getProperty1() {
      return property1;
   }
   
}

Of course it is using the currently described versioning mechanism. First let’s run an instance of Kubernetes locally using Kind (Kubernetes IN Docker). Here’s the command that creates the local cluster.

$ kind create cluster --config=cluster.yaml

To make it available under virtual address 192.168.99.100 we need to provide the following configuration file cluster.yaml.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  apiServerAddress: 192.168.99.100
  apiServerPort: 6443

After running the following command, kind is ready to use.

Now, let’s create two test ConfigMaps for our application with kubectl apply command. Let’s take a look at the result.

Then we should deploy our sample app. The only difference between two subsequent deployments is in info.app.version property, that is 1.0 for first deployment, and 1.1 for the second deployment. Here’s the Deployment manifest. As you see below it does not inject any ConfigMap or Secret using Kubernetes mechanisms.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-test
spec:
  selector:
    matchLabels:
      app: api-test
  template:
    metadata:
      labels:
        app: api-test
    spec:
      containers:
      - name: api-test
        image: piomin/api-test
        ports:
        - containerPort: 8080

In the newest version of our sample application I just changed the value of property info.app.version from 1.0 to 1.1.

spring:
  application:
    name: api-test
  cloud:
    kubernetes:
      config:
        enableVersioning: true
info:
  app:
    version: 1.1

Let’s call our test endpoint. It returns the value of property1 taken from ConfigMap labelled with 1.1.

$ curl http://localhost:8080/property
v1.1

Here’s the history of deployments for api-test application. We may rollback the version of deployment to the previous one (1) by using command kubectl rollout undo.

Now we may call our test endpoint one more time. As you can see below it returns v1.0, which is set as a property1 value in ConfigMap labelled with version 1.0.

$ curl http://localhost:8080/property
v1.0

Conclusion

To use described versioning mechanism on Kubernetes you just need to create ConfigMap labelled with properly, include forked Spring Cloud Kubernetes Config from my repository and add property info.app.version to the Spring Boot application properties.

The post Kubernetes ConfigMap Versioning for Spring Boot Apps appeared first on Piotr's TechBlog.

<dependency>
  <groupId>com.github.piomin</groupId>
  <artifactId>spring-cloud-kubernetes-discovery-ext-client</artifactId>
  <version>1.0.1.RELEASE</version>
</dependency>

The registration feature is still disabled, since we won’t set property spring.cloud.kubernetes.discovery.register to true.

spring:
  cloud:
    kubernetes:
      discovery:
        register: true

If you are running an application you need to set a target namespace in Kubernetes, where it will be registered after startup. Here we are using a mechanism provided by Spring Cloud Kubernetes, which allows to set a default namespace for Fabric8 Kubernetes Client by setting environment variable KUBERNETES_NAMESPACE.

A registration mechanism is based on Kubernetes objects: Service and Endpoints. It creates a service with the name taken from property spring.application.name. In the annotations field of Service object it puts a path of health check endpoint. By default it is /actuator/health. The new Service object is created only if it does not exist. The following screen shows the details about Service created by library for application api-test.

The path of the health check endpoint may be overridden using property spring.cloud.kubernetes.discovery.healthUrl.

spring:
  cloud:
    kubernetes:
      discovery:
        healthUrl: /actuator/liveness 

The next step is to create an Endpoints object. Normally, you don’t have a lot to deal with Endpoints, since it just tracks the IP addresses of the pods the service send traffic to. The name of Endpoints is the same as the name of Service. The IP address of the application is stored in the Subset section. To distinguish Endpoints created by the library for external applications from Endpoints registered by the platform automatically each of them is labeled with external flag with value true.

We may display details about selected Endpoints by command kubectl describe to see the structure of this object.

The IP address of Spring Boot application is automatically detected by the library by calling Java method InetAddress.getLocalHost().getHostAddress(). You may set a static IP address by using property spring.cloud.kubernetes.discovery.ipAddress as shown below.

spring:
  cloud:
    kubernetes:
      discovery:
        ipAddress: 192.168.99.1

The library spring-cloud-kubernetes-discovery-ext-client is based on the Spring Cloud Kubernetes project. It uses the Kubernetes API client provided within this library. The version of Spring Cloud Release Train used by the library is Hoxton.RELEASE.

<dependencies>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-kubernetes</artifactId>
   </dependency>
</dependencies>
<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Hoxton.RELEASE</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

Assuming you have currently run a local instance of your Kubernetes cluster, you should at least provide an address of master API, and set property spring.cloud.kubernetes.client.trustCerts just for the development purposes. Here’s bootstrap.yml for my Spring Boot demo application.

spring:
  application:
    name: api-test
  cloud:
    kubernetes:
      discovery:
        register: true
      client:
        masterUrl: 192.168.99.100:8443
        trustCerts: true

If you shutdown your Spring Boot application gracefully spring-cloud-kubernetes-discovery-ext-client will unregister it from Kubernetes API. However, we always have to consider situations like forceful kill of application or network problems, that may cause unregistered instances in our Kubernetes API. Such situations should be handled on the platform side. Since Kubernetes Discovery does provide any built-in mechanisms for that (like heartbeat for applications running outside cluster), you may provide your implementation within Kubernetes Job or you can just use my library spring-cloud-kubernetes-discovery-ext-watcher responsible for detecting and removing inactive Endpoints.
The main idea behind that library is illustrated on the picture below. The module spring-cloud-kubernetes-discovery-ext-watcher is in fact Spring Boot application that needs to be run on Kubernetes. It periodically queries Kubernetes API in order to fetch the current list of external Endpoints registered by applications using spring-cloud-kubernetes-discovery-ext-client library. Then it is trying to call health endpoints registered for each application using IP address and ports taken from master API. If it won’t receive any response or receive response with HTTP status 5XX several times in row, it removes IP address from Subset section of Endpoints object.

By default, spring-cloud-kubernetes-discovery-ext-watcher application checks out endpoints registered in the same Kubernetes namespace as that application. This behaviour may be customized using configuration properties. We can set the default target namespace by setting property spring.cloud.kubernetes.watcher.targetNamespace or just enable watching for Endpoints labeled with external=true across all the namespace by setting property spring.cloud.kubernetes.watcher.allNamespaces to true. We can also override some default retry properties for calling application health endpoints like number of retries (by default 3) or connect timeout (by default 1000ms). The configuration settings need to be delivered to the Watcher application as application.yml or bootstrap.yml file.

spring:
  cloud:
    kubernetes:
      watcher:
        targetNamespace: test
        retries: 5
        retryTimeout: 5000

To deploy spring-cloud-kubernetes-discovery-ext-watcher application on your Kubernetes cluster you just need to apply the following Deployment definition using kubectl apply command.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: spring-cloud-discovery-watcher
spec:
  selector:
    matchLabels:
      app: spring-cloud-discovery-watcher
  template:
    metadata:
      labels:
        app: spring-cloud-discovery-watcher
    spec:
      containers:
      - name: watcher
        image: piomin/spring-cloud-discovery-watcher
        ports:
        - containerPort: 8080

Since that application uses Spring Cloud Kubernetes for accessing Master API you need to grant privileges to read objects like Service, Endpoints or ConfigMap. For development purposes you can just assign cluster-admin to the default ServiceAccount in target namespace.

$ kubectl create clusterrolebinding admin-external --clusterrole=cluster-admin --serviceaccount=external:default

In case you would like to override some default configuration settings you should define application.yml file and place it inside ConfigMap. Since spring-cloud-kubernetes-discovery-ext-watcher uses Spring Cloud Kubernetes Config we may take an advantage of its integration with ConfigMap. To do that just create ConfigMap with the same metadata.name as the application name.

apiVersion: v1
kind: ConfigMap
metadata:
  name: api-test
data:
  application.yaml: |-
    spring:
      cloud:
        kubernetes:
          watcher:
            allNamespaces: true
            retries: 5
            retryTimeout: 5000

Here’s our sample deployment in the external namespace.

Now, let’s take a look on the logs generated by the spring-cloud-kubernetes-discovery-ext-watcher application. Before running it I have started my sample application that uses spring-cloud-kubernetes-discovery-ext-client outside Kubernetes. I has been registered under address 192.168.99.1:8080. As you in the following logs in the beginning Watcher Application was able to communicate with sample application Actuator endpoint. Then I killed the sample application. Since Watcher was unable to call Actuator endpoint of previously checked application it finally remove that address from Subset section of api-test Endpoints

Here’s the Endpoints object after removal of address 192.168.99.1:8080.

Summary

The repository with Client library and Watcher application is available on GitHub: https://github.com/piomin/spring-cloud-kubernetes-discovery-ext.git. The library spring-cloud-kubernetes-discovery-ext-client is available in Maven Central Repository. The Docker image with Watcher application is available on Docker Hub: https://hub.docker.com/repository/docker/piomin/spring-cloud-discovery-watcher. You can also run it directly from a source code using Skaffold.

The post Using Spring Cloud Kubernetes External Library appeared first on Piotr's TechBlog.

Example

As usual the source code with sample applications is available on GitHub in the repository https://github.com/piomin/sample-spring-microservices-kubernetes.git. It was used as the example for some other articles on my blog, which may be helpful to better understand the idea behind Spring Cloud Kubernetes. One of them is Microservices with Spring Cloud Kubernetes.
Anyway, I’m using three sample Spring Boot applications from this repository for a demo purpose. Each application would be run in a different namespace inside Minikube. Spring Boot Admin should monitor only Spring Boot applications, with the assumptions that there are different types of applications and solutions running on Kubernetes. All the three applications are optimized to be built with Skaffold and Jib. So the only thing you have to do is to run command skaffold dev in the directory of each application.

The picture illustrating an architecture of applications for the current article is visible below. The application employee-service is deployed inside namespace a, department-service is deployed inside namespace b, while organization-service is deployed inside namespace c. Spring Boot Admin is also started using Spring Boot and is deployed in the namespace default.

Dependencies

Let’s begin from dependencies. Spring Boot Admin Server is available as a starter library. The current stable version is 2.2.2. We also need to include Spring Boot Web starter. Because our Spring Boot Admin application is integrating with Kubernetes discovery we should include Spring Cloud Kubernetes Discovery. If you also use Spring Cloud Kubernetes for direct integration with ConfigMap and Secret you may just include spring-cloud-starter-kubernetes-all starter instead of starter dedicated only for discovery. Here’s the list of dependencies inside our sample module admin-server with embedded Spring Boot Admin.

<parent>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-parent</artifactId>
   <version>2.2.4.RELEASE</version>
</parent>
<dependencies>
   <dependency>
      <groupId>de.codecentric</groupId>
      <artifactId>spring-boot-admin-starter-server</artifactId>
      <version>2.2.2</version>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-security</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
   </dependency>
</dependencies>
<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Hoxton.RELEASE</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

Spring Boot Admin Server and Kubernetes discovery

We need to add some annotations to the Spring Boot main class to enable Spring Boot Admin with Kubernetes Discovery. First of them is of course @EnableAdminServer (1). With Spring Cloud Kubernetes we still need to add annotation @EnableDiscoveryClient to enable DiscoveryClient based on Kubernetes API (2). The last annotation @EnableScheduling is not so obvious, but also required (3). Without it a discovery client won’t periodically call Kubernetes API to refresh the list of running services, and do it only once on startup. Since, we always want to have the current list of pods (for example after scaling up the number of application instances), we need to enable a scheduler responsible for watching service catalog for changes and updating the DiscoveryClient implementation accordingly.

@SpringBootApplication
@EnableAdminServer // (1)
@EnableDiscoveryClient // (2)
@EnableScheduling // (3)
public class AdminApplication {

   public static void main(String[] args) {
      SpringApplication.run(AdminApplication.class, args);
   }

}

Now, a crucial issue throughout the game – configuration. Those two properties visible below do the whole “magic”. First of them spring.cloud.kubernetes.discovery.all-namespace enables DiscoveryClient from all namespaces (1). After enabling it Spring Boot Admin would be able to monitor all the applications across the whole cluster! Great, but since we would like to force it to monitor just the Spring Boot application, we need to be able to filter them. Here comes Spring Cloud Kubernetes with another smart solution. The property spring.cloud.kubernetes.discovery.serviceLabels allows us to define the set of labels used for filtering list of services fetched from Kubernetes API. In simple words, we fetch only those Kubernetes Service, that has labels defined on the Spring Boot Admin server side with such values. Here’s application.yml for admin-service defined inside ConfigMap.

kind: ConfigMap
apiVersion: v1
metadata:
  name: admin
data:
  application.yml: |-
    spring:
     cloud:
      kubernetes:
        discovery:
          all-namespaces: true # (1)
          service-labels:
            spring-boot: true # (2)

Implementation on the client side

Thanks to using DiscoveryClient on Spring Boot Admin Server we don’t have to include any additional Spring Boot Admin Client library to our sample applications. Of course, they also don’t have to include Spring Cloud Kubernetes, since Kubernetes manages pods registration by itself. In fact, the only thing we have to do is to add label to the Kubernetes Service, which has been set as a filtering condition on the server side. Here’s the example of Service definition for department-service.

apiVersion: v1
kind: Service
metadata:
  name: department
  labels:
    app: department
    spring-boot: "true"
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: department

Spring Boot Admin is based on Spring Boot Actuator endpoints, so each application should at least include that project to Maven dependencies.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

By default, not all the endpoints are exposed outside application. Therefore we need to expose them by setting property management.endpoints.web.exposure.include to *. It is also worth showing details about application in health endpoint. That step is optional. Here’s our application.yml for department-service.

spring:
  application:
    name: department
management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint:
    health:
      show-details: ALWAYS

It’s also worth generating a build-info.properties file during build to provide more details for Actuator /info endpoint.

<plugin>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-maven-plugin</artifactId>
   <executions>
      <execution>
         <goals>
            <goal>build-info</goal>
         </goals>
      </execution>
   </executions>
</plugin>

Running Spring Boot Admin on Kubernetes

First, we should start Minikube. Because we are running applications and MongoDB I suggest changing the default memory limit 4 GB.

$ minikube start --vm-driver=virtualbox --memory='4000mb'

Let’s start from creating all the required namespaces on Minikube. Except default namespace, we also need namespaces a, b, and c, as it has been described in the section Example.

$ kubectl create namespace a
namespace/a created
$ kubectl create namespace b
namespace/b created
$ kubectl create namespace c
namespace/c created

Spring Boot Admin uses Spring Cloud Kubernetes, which requires extra privileges in order to access Kubernetes API. Let’s set cluster-admin as a default role for ServiceAccount just for a development purpose.

$ kubectl create clusterrolebinding admin-default --clusterrole=cluster-admin --serviceaccount=default:default

Assuming we have already succesfully deploy all the sample microservices and Spring Boot Admin Server application on Minikube you will get the following list of Services for all the namespaces. Option -L=spring-boot prints value of label spring-boot for all services.

We may also take a look into the list of pods. I have set two instances for employee deployment.

As you can see on the picture below Spring Boot Admin manages only Kubernetes Services labeled with spring-boot=true. For example mongodb, kubernetes or admin inside default namespace were omitted.

Spring Boot Admin provides some useful features for managing Spring Boot applications. It is worth considering using it on Kubernetes to monitor the whole set of your microservices distributed across many namespaces. The following picture illustrates the details page for a single application (pod) running on Minikube.

The post Spring Boot Admin on Kubernetes appeared first on Piotr's TechBlog.

Skaffold

Skaffold is a simple command-line tool that is able to handle the workflow for building, pushing and deploying your Java application on Kubernetes. It saves a lot of developer time by automating most of the work from source code to the deployment. It natively supports the most common image-building and application deployment strategies. Skaffold is an open-source project from Google. It is not the only one interesting tool from Google that may be used to help in local development on Kubernetes. Another one of them, Jib, is dedicated only for Java applications. It allows you to build optimized Docker and OCI images for your Java applications without a Docker daemon. It is available as Maven of Gradle plugin, or just as a Java library. With Jib you do not need to maintain a Dockerfile or even run a Docker daemon. It is also able to take advantage of image layering and registry caching to achieve fast, incremental builds. To use Jib during our application build we just need to include it to Maven pom.xml. We may easily customize the behaviour of Jib Maven Plugin by using properties inside configuration section. But for a standard Java application it is highly possible you could use default settings as shown below.

<plugin>
   <groupId>com.google.cloud.tools</groupId>
   <artifactId>jib-maven-plugin</artifactId>
   <version>1.8.0</version>
</plugin>

By default, Skaffold uses Dockerfile while building an image with our application. We may customize this behaviour to use Jib Maven Plugin instead of Dockerfile. We may do it by changing the Skaffold configuration file available in our project root directory – skaffold.yaml. We should also define there a name of the generated Docker image and its tagging policy.

apiVersion: skaffold/v2alpha1
kind: Config
build:
  artifacts:
    - image: piomin/department
       jib: {}
  tagPolicy:
    gitCommit: {}

If your Kubernetes deployment manifest is located inside k8s directory and its name is deployment.yaml you don’t have to provide any additional configuration. Here’s a structure of our sample project that fulfills Skaffold requirements.

Assuming you have successfully run a Minikube instance on your local machine, you just need to run command skaffold dev in your root project directory. This command starts the process of building a Docker image with your application and then deploys it on Minikube. After that it watches for any change in your source code and triggers a new build after every change in the filesystem. There are some parameters, which may be used for customization. Option --port-forward is responsible for running command kubectl port-forward for all the ports exposed outside the container. We may also disable auto-build triggered after file change, and enable only manual mode that triggers build on demand. It may be especially useful, for example if you are using autosave mode in your IDE like IntelliJ. The last option used in the example of command visible below, --no-prune, is responsible for disable removal of images, containers and deployment created by Skaffold.

$ skaffold dev --port-forward --trigger=manual --no-prune

Another useful Skaffold command during development is skaffold debug. It is very similar to skaffold dev, but configures our pipeline for debugging. For Java applications it is running JWDP agent exposed on port 5005 outside the container. Then you may easily connect with the agent, for example using your IDE.

$ skaffold debug --port-forward --no-prune

I think the most suitable way to show you Skaffold in action is on video. Here’s a 9 minutes long movie that shows you how to use Skaffold for local Java development, running and debugging a Spring Boot application on Kubernetes.

[wpvideo 3Op96XNi]

Cloud Code

Not every developer likes command-line tools. At this point Google comes GUI tools, which may be easily installed as plugins on your IDEs: IntelliJ or Visual Studio Code. This set of tools called Cloud Code help you write, run, and debug cloud-native applications quickly and easily. Cloud Code uses Skaffold in background, but hides it behind two buttons available in your Run Configurations (IntelliJ): Develop on Kubernetes and Run on Kubernetes.
Develop on Kubernetes is running Skaffold in the default notify mode that triggers build after every change of file inside your project.

Develop on Kubernetes is running Skaffold in the manual mode that starts the build on demand after you click that button.

Cloud Cloud offers some other useful features. It provides an auto-completion for syntax inside Kubernetes YAML manifests.

You may also display a graphical representation of your Kubernetes cluster as shown below.

Dekorate

We have already discussed some interesting tools for automating the deployment process beginning from a change in the source code to running an application on Kubernetes cluster (Minikube). Beginning from this section we will discuss interesting libraries and extensions to popular JVM frameworks, which helps you to speed-up your Java development on Kubernetes. First of them is Dekorate. Dekorate is a collection of compile time generators and decorators of Kubernetes manifests. It makes generating and decorating Kubernetes manifests as simple as adding a dependency to your project. It allows you to use well-known Java annotations style to define Kubernetes resources used by your application. It provides integration for Spring Boot and Quarkus frameworks.
To enable integration for your Spring Boot application you just need to include the following dependency to your Maven pom.xml.

<dependency>
  <groupId>io.dekorate</groupId>
  <artifactId>kubernetes-spring-starter</artifactId>
  <version>0.10.10</version>
</dependency>

Now, if you build your application using the Maven command visible below Dekorate is able to analyze your source code and generate Kubernetes manifest basing on that.

$ mvn clean install -Ddekorate.build=true -Ddekorate.deploy=true

Besides just an analysis of source code Dekarate allows to define Kubernetes resources using configuration files or annotations. The following code snippet shows you how to set 2 replicas of your application, expose it outside the cluster as route and refer to the existing ConfigMap on your OpenShift instance. You may also use @JvmOptions to set some JVM running parameters like maximum heap usage.

@SpringBootApplication
@OpenshiftApplication(replicas = 2, expose = true, envVars = {
        @Env(name="sample-app-config", configmap = "sample-app-config")
})
@JvmOptions(xms = 128, xmx = 256, heapDumpOnOutOfMemoryError = true)
@EnableSwagger2
public class SampleApp {

    public static void main(String[] args) {
        SpringApplication.run(SampleApp.class, args);
    }
   
}

Of course, I presented only a small set of options offered by Dekorate. You can also define Kubernetes labels, annotations, secrets, volumes and many more. For more details about using Dekorate with OpenShift you may in one of my previous articles Deploying Spring Boot Application on OpenShift with Dekorate.

Spring Cloud Kubernetes

If you are building your web applications on top of Spring Boot you should consider using Spring Cloud Kubernetes for integration with Kubernetes. Spring Cloud Kubernetes provides Spring Cloud common interface implementations that consume Kubernetes native services via master API. The main features of that project are:

• Kubernetes PropertySource implementation including auto-reload of configuration after ConfigMap or Secret change
• Kubernetes native discovery with DiscoveryClient implementation including multi-namespace discovery
• Client side load balancing with Spring Cloud Netflix Ribbon
• Pod health indicator

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
</dependency>   
<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Hoxton.RELEASE</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

After that discovery and configuration based on ConfigMap is enabled. If you also would like to use Secret<.code> as property source for the application you need to enable it in bootstrap.yml.

spring:
  application:
    name: department
  cloud:
    kubernetes:
      secrets:
        enableApi: true

The name of ConfigMap or Secret (property manifest.name) should be the same as the application name to use them without any configuration customization. Here’s sample ConfigMap for department application. It is managed by Spring Cloud Kubernetes without a necessity to inject it Deployment manifest.

apiVersion: v1
kind: ConfigMap
metadata:
  name: department
data:
  application.yml: |-
    spring:
     cloud:
      kubernetes:
        discovery:
          all-namespaces: true
    spring:
      data:
       mongodb:
        database: admin
        host: mongodb

Spring Cloud Kubernetes Discovery and Ribbon integration allows you to use any of Spring Rest Client to communicate with other services via name. Here’s the example of Spring Cloud OpenFeign usage.

@FeignClient(name = "employee")
public interface EmployeeClient {

   @GetMapping("/department/{departmentId}")
   List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
   
}

Another useful Spring Cloud Kubernetes feature is an ability to reload configuration after change in ConfigMap or Secret. That’s a pretty amazing thing for a developer, because it is possible to refresh some beans without restarting the whole pod with application. However, you should keep in mind that configuration beans annotated with @ConfigurationProperties or @RefreshScope are reloaded. By default, this feature is disabled. To enable you should use the following property.

spring:
  cloud:
    kubernetes:
      reload:
        enabled: true

For more details about Spring Cloud Kubernetes including source code examples you may refer to my previous article Microservices with Spring Cloud Kubernetes.

Micronaut

Similar to Spring Boot, Micronaut provides a library for integration with Kubernetes. In comparison to Spring Cloud Kubernetes it additionally allows to read ConfigMap and Secret from mounted volumes and allows to enable filtering services by their name during discovery. To enable Kubernetes discovery for Micronaut applications we first to include the following library to our Maven pom.xml.

<dependency>
    <groupId>io.micronaut.kubernetes</groupId>
    <artifactId>micronaut-kubernetes-discovery-client</artifactId>
</dependency>

This module also allows us to use Micronaut HTTP Client with discovery by service name.

@Client(id = "employee", path = "/employees")
public interface EmployeeClient {
 
    @Get("/department/{departmentId}")
    List<Employee> findByDepartment(Long departmentId);
 
}

You don’t have to include any additional library to enable integration with Kubernetes PropertySource, since it is provided in Micronaut Config Client core library. You just need to enable it in application bootstrap.yml. Unlike Spring Boot, Micronaut uses labels instead of metada.name to match ConfigMap or Secret with application. After enabling Kubernetes config client, also configuration auto-reload feature is enabled. Here’s our bootstrap.yml file.

micronaut:
  application:
    name: department
  config-client:
    enabled: true
kubernetes:
  client:
    config-maps:
      labels:
        - app: department
    secrets:
      enabled: true
      labels:
        - app: department

Now, our ConfigMap also needs to be labeled with app=department.

apiVersion: v1
kind: ConfigMap
metadata:
  name: department
  labels:
    app: department
data:
  application.yaml: |-
    mongodb:
      collection: department
      database: admin
    kubernetes:
      client:
        discovery:
          includes:
            - employee

For more details about integration between Micronaut and Kubernetes you may refer to my article Guide to Micronaut Kubernetes.

The post Local Java Development on Kubernetes appeared first on Piotr's TechBlog.

Architecture

For the current we may change a little architecture presented in my previous article about Spring Cloud and Kubernetes – Microservices With Spring Cloud Kubernetes. We move one of the sample microservices employee-service, described in the mentioned article, outside Kubernetes cluster. Now, the applications which are communicating with employee-service need to use the addresses outside the cluster. Also they should be able to handle a port number dynamically generated on the application during startup (server.port=0). Our applications are still distributed across different namespaces, so it is important to enable multi-namespaces discovery features – also described in my previous article. The application employee-service is connecting to MongoDB, which is still deployed on Kubernetes. In that case the integration is performed via Kubernetes Service. The following picture illustrates our current architecture.

Spring Cloud Kubernetes PropertySource

The situation with distributed configuration is clear. We don’t have to implement any additional code to be able to use it externally. Just before starting a client application we have to set the environment variable KUBERNETES_NAMESPACE. Since we set it to external we first need to create such a namespace.

Then we may apply some property sources to that namespace. The configuration is consisting of Kubernetes ConfigMap and Secret. We store there a Mongo location, credentials, and some other properties. Here’s our ConfigMap declaration.

apiVersion: v1
kind: ConfigMap
metadata:
  name: employee
data:
  application.yaml: |-
    logging.pattern.console: "%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
    spring:
      cloud:
        kubernetes:
          discovery:
            all-namespaces: true
            register: true
      data:
        mongodb:
          database: admin
          host: 192.168.99.100
          port: 32612

The port number is taken from mongodb Service, which is deployed as NodePort type.

And here’s our Secret.

apiVersion: v1
kind: Secret
metadata:
  name: employee
type: Opaque
data:
  spring.data.mongodb.username: UGlvdF8xMjM=
  spring.data.mongodb.password: cGlvdHI=

Then, we are creating resources inside external namespace.

In the bootstrap.yml file we need to set the address of Kubernetes API server and property responsible for trusting server’s cert. We should also enable using Secret as property source, which is disabled by default for Spring Cloud Kubernetes Config.

spring:
  application:
    name: employee
  cloud:
    kubernetes:
      secrets:
        enableApi: true
      client:
        masterUrl: 192.168.99.100:8443
        trustCerts: true

External registration with Spring cloud Kubernetes

The situation with service discovery is much more complicated. Since Spring Cloud Kubernetes delegates discovery to the platform, what is perfectly right for internal applications, the lack of auto-configured registration is a problem for an external application. That’s why I decided to implement a module for Spring Cloud Kubernetes auto-configured registration for an external application. Currently it is available inside our sample repository as a spring-cloud-kubernetes-discovery-ext module. It is implemented according to the Spring Cloud Discovery registration pattern. Let’s begin with dependencies. We just need to include spring-cloud-starter-kubernetes, which contains core and discovery modules.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes</artifactId>
</dependency>

Here’s our registration object. It implements Registration interface from Spring Cloud Commons, which defines some basic getters. We should provide hostname, port, serviceId etc.

public class KubernetesRegistration implements Registration {

    private KubernetesDiscoveryProperties properties;

    private String serviceId;
    private String instanceId;
    private String host;
    private int port;
    private Map<String, String> metadata = new HashMap<>();

    public KubernetesRegistration(KubernetesDiscoveryProperties properties) {
        this.properties = properties;
    }

    @Override
    public String getInstanceId() {
        return instanceId;
    }

    @Override
    public String getServiceId() {
        return serviceId;
    }

    @Override
    public String getHost() {
        return host;
    }

    @Override
    public int getPort() {
        return port;
    }

    @Override
    public boolean isSecure() {
        return false;
    }

    @Override
    public URI getUri() {
        return null;
    }

    @Override
    public Map<String, String> getMetadata() {
        return metadata;
    }

    @Override
    public String getScheme() {
        return "http";
    }

    public void setServiceId(String serviceId) {
        this.serviceId = serviceId;
    }

    public void setInstanceId(String instanceId) {
        this.instanceId = instanceId;
    }

    public void setHost(String host) {
        this.host = host;
    }

    public void setPort(int port) {
        this.port = port;
    }

    public void setMetadata(Map<String, String> metadata) {
        this.metadata = metadata;
    }

}

We have some additional configuration properties in comparison to Spring Cloud Kubernetes Discovery. They are available under the same prefix spring.cloud.kubernetes.discovery.

@ConfigurationProperties("spring.cloud.kubernetes.discovery")
public class KubernetesRegistrationProperties {

    private String ipAddress;
    private String hostname;
    private boolean preferIpAddress;
    private Integer port;
    private boolean register;
   
    // GETTERS AND SETTERS
   
}

There is also a class that should extend abstract AbstractAutoServiceRegistration. It is responsible for managing the registration process. First, it enables the registration mechanism only if an application is running outside Kubernetes. It uses PodUtils bean defined in Spring Cloud Kubernetes Core for that. It also implements a method for building a registration object. The port may be generated dynamically on startup. The rest of the process is performed inside the abstract subclass.

public class KubernetesAutoServiceRegistration extends AbstractAutoServiceRegistration<KubernetesRegistration> {

    private KubernetesDiscoveryProperties properties;
    private KubernetesRegistrationProperties registrationProperties;
    private KubernetesRegistration registration;
    private PodUtils podUtils;

    KubernetesAutoServiceRegistration(ServiceRegistry<KubernetesRegistration> serviceRegistry,
                                      AutoServiceRegistrationProperties autoServiceRegistrationProperties,
                                      KubernetesRegistration registration, KubernetesDiscoveryProperties properties,
                                      KubernetesRegistrationProperties registrationProperties, PodUtils podUtils) {
        super(serviceRegistry, autoServiceRegistrationProperties);
        this.properties = properties;
        this.registrationProperties = registrationProperties;
        this.registration = registration;
        this.podUtils = podUtils;
    }

    public void setRegistration(int port) throws UnknownHostException {
        String ip = registrationProperties.getIpAddress() != null ? registrationProperties.getIpAddress() : InetAddress.getLocalHost().getHostAddress();
        registration.setHost(ip);
        registration.setPort(port);
        registration.setServiceId(getAppName(properties, getContext().getEnvironment()) + "." + getNamespace(getContext().getEnvironment()));
        registration.getMetadata().put("namespace", getNamespace(getContext().getEnvironment()));
        registration.getMetadata().put("name", getAppName(properties, getContext().getEnvironment()));
        this.registration = registration;
    }

    @Override
    protected Object getConfiguration() {
        return properties;
    }

    @Override
    protected boolean isEnabled() {
        return !podUtils.isInsideKubernetes();
    }

    @Override
    protected KubernetesRegistration getRegistration() {
        return registration;
    }

    @Override
    protected KubernetesRegistration getManagementRegistration() {
        return registration;
    }

    public String getAppName(KubernetesDiscoveryProperties properties, Environment env) {
        final String appName = properties.getServiceName();
        if (StringUtils.hasText(appName)) {
            return appName;
        }
        return env.getProperty("spring.application.name", "application");
    }

    public String getNamespace(Environment env) {
        return env.getProperty("KUBERNETES_NAMESPACE", "external");
    }

}

The process should be initialized just after application startup. In order to catch a startup event we prepare a bean that implements SmartApplicationListener interface. The listener method calls bean KubernetesAutoServiceRegistration to prepare the registration object and start the process.

public class KubernetesAutoServiceRegistrationListener implements SmartApplicationListener {

    private static final Logger LOGGER = LoggerFactory.getLogger(KubernetesAutoServiceRegistrationListener.class);

    private final KubernetesAutoServiceRegistration autoServiceRegistration;

    KubernetesAutoServiceRegistrationListener(KubernetesAutoServiceRegistration autoServiceRegistration) {
        this.autoServiceRegistration = autoServiceRegistration;
    }

    @Override
    public boolean supportsEventType(Class<? extends ApplicationEvent> eventType) {
        return WebServerInitializedEvent.class.isAssignableFrom(eventType);
    }

    @Override
    public boolean supportsSourceType(Class<?> sourceType) {
        return true;
    }

    @Override
    public int getOrder() {
        return 0;
    }

    @Override
    public void onApplicationEvent(ApplicationEvent applicationEvent) {
        if (applicationEvent instanceof WebServerInitializedEvent) {
            WebServerInitializedEvent event = (WebServerInitializedEvent) applicationEvent;
            try {
                autoServiceRegistration.setRegistration(event.getWebServer().getPort());
                autoServiceRegistration.start();
            } catch (UnknownHostException e) {
                LOGGER.error("Error registering to kubernetes", e);
            }
        }
    }

}

Here’s the auto-configuration for all previously described beans.

@Configuration
@ConditionalOnProperty(name = "spring.cloud.kubernetes.discovery.register", havingValue = "true")
@AutoConfigureAfter({AutoServiceRegistrationConfiguration.class, KubernetesServiceRegistryAutoConfiguration.class})
public class KubernetesAutoServiceRegistrationAutoConfiguration {

    @Autowired
    AutoServiceRegistrationProperties autoServiceRegistrationProperties;

    @Bean
    @ConditionalOnMissingBean
    public KubernetesAutoServiceRegistration autoServiceRegistration(
            @Qualifier("serviceRegistry") KubernetesServiceRegistry registry,
            AutoServiceRegistrationProperties autoServiceRegistrationProperties,
            KubernetesDiscoveryProperties properties,
            KubernetesRegistrationProperties registrationProperties,
            KubernetesRegistration registration, PodUtils podUtils) {
        return new KubernetesAutoServiceRegistration(registry,
                autoServiceRegistrationProperties, registration, properties, registrationProperties, podUtils);
    }

    @Bean
    public KubernetesAutoServiceRegistrationListener listener(KubernetesAutoServiceRegistration registration) {
        return new KubernetesAutoServiceRegistrationListener(registration);
    }

    @Bean
    public KubernetesRegistration registration(KubernetesDiscoveryProperties properties) throws UnknownHostException {
        return new KubernetesRegistration(properties);
    }

    @Bean
    public KubernetesRegistrationProperties kubernetesRegistrationProperties() {
        return new KubernetesRegistrationProperties();
    }

}

Finally, we may proceed to the most important step – an integration with Kubernetes API. Spring Cloud Kubernetes uses Fabric Kubernetes Client for communication with master API. The KubernetesClient bean is already auto-configured, so we may inject it. The register and deregister methods are implemented in class KubernetesServiceRegistry that implements ServiceRegistry interface. Discovery in Kubernetes is configured via Endpoint API. Each Endpoint contains a list of EndpointSubset that stores a list of registered IPs inside EndpointAddress and a list of listening ports inside EndpointPort. Here’s the implementation of register and deregister methods.

public class KubernetesServiceRegistry implements ServiceRegistry<KubernetesRegistration> {

    private static final Logger LOG = LoggerFactory.getLogger(KubernetesServiceRegistry.class);

    private final KubernetesClient client;
    private KubernetesDiscoveryProperties properties;

    public KubernetesServiceRegistry(KubernetesClient client, KubernetesDiscoveryProperties properties) {
        this.client = client;
        this.properties = properties;
    }

    @Override
    public void register(KubernetesRegistration registration) {
        LOG.info("Registering service with kubernetes: " + registration.getServiceId());
        Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
                .inNamespace(registration.getMetadata().get("namespace"))
                .withName(registration.getMetadata().get("name"));
        Endpoints endpoints = resource.get();
        if (endpoints == null) {
            Endpoints e = client.endpoints().create(create(registration));
            LOG.info("New endpoint: {}",e);
        } else {
            try {
                Endpoints updatedEndpoints = resource.edit()
                        .editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
                        .addToAddresses(new EndpointAddressBuilder().withIp(registration.getHost()).build())
                        .endSubset()
                        .done();
                LOG.info("Endpoint updated: {}", updatedEndpoints);
            } catch (RuntimeException e) {
                Endpoints updatedEndpoints = resource.edit()
                        .addNewSubset()
                        .withPorts(new EndpointPortBuilder().withPort(registration.getPort()).build())
                        .withAddresses(new EndpointAddressBuilder().withIp(registration.getHost()).build())
                        .endSubset()
                        .done();
                LOG.info("Endpoint updated: {}", updatedEndpoints);
            }
        }

    }

    @Override
    public void deregister(KubernetesRegistration registration) {
        LOG.info("De-registering service with kubernetes: " + registration.getInstanceId());
        Resource<Endpoints, DoneableEndpoints> resource = client.endpoints()
                .inNamespace(registration.getMetadata().get("namespace"))
                .withName(registration.getMetadata().get("name"));

        EndpointAddress address = new EndpointAddressBuilder().withIp(registration.getHost()).build();
        Endpoints updatedEndpoints = resource.edit()
                .editMatchingSubset(builder -> builder.hasMatchingPort(v -> v.getPort().equals(registration.getPort())))
                .removeFromAddresses(address)
                .endSubset()
                .done();
        LOG.info("Endpoint updated: {}", updatedEndpoints);

        resource.get().getSubsets().stream()
                .filter(subset -> subset.getAddresses().size() == 0)
                .forEach(subset -> resource.edit()
                        .removeFromSubsets(subset)
                        .done());
    }

    private Endpoints create(KubernetesRegistration registration) {
        EndpointAddress address = new EndpointAddressBuilder().withIp(registration.getHost()).build();
        EndpointPort port = new EndpointPortBuilder().withPort(registration.getPort()).build();
        EndpointSubset subset = new EndpointSubsetBuilder().withAddresses(address).withPorts(port).build();
        ObjectMeta metadata = new ObjectMetaBuilder()
                .withName(registration.getMetadata().get("name"))
                .withNamespace(registration.getMetadata().get("namespace"))
                .build();
        Endpoints endpoints = new EndpointsBuilder().withSubsets(subset).withMetadata(metadata).build();
        return endpoints;
    }

}

The auto-configuration beans are registered in spring.factories file.

org.springframework.boot.autoconfigure.EnableAutoConfiguration=\
org.springframework.cloud.kubernetes.discovery.ext.KubernetesServiceRegistryAutoConfiguration,\
org.springframework.cloud.kubernetes.discovery.ext.KubernetesAutoServiceRegistrationAutoConfiguration

Enabling Registration

Now, we may include an already created library to any Spring Cloud application running outside Kubernetes, for example to the employee-service. We are using our example applications together with Spring Cloud Kubernetes.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
</dependency>
<dependency>
   <groupId>pl.piomin.services</groupId>
   <artifactId>spring-cloud-kubernetes-discovery-ext</artifactId>
   <version>1.0-SNAPSHOT</version>
</dependency>

The registration is still disabled, since we won’t set property spring.cloud.kubernetes.discovery.register to true.

spring:
  cloud:
    kubernetes:
      discovery:
        register: true

Sometimes it might be used to set static IP addresses in configuration, in case you would have multiple network interfaces.

spring:
  cloud:
    kubernetes:
      discovery:
        ipAddress: 192.168.99.1

By setting 192.168.99.1 as a static IP address I’m able to easily perform some tests with Minikube node, which is running on the VM available under 192.168.99.100.

Manual Testing

Let’s start employee-service locally. As you see on the screen below it has succesfully load configuration from Kubernetes and connected with MongoDB running on the cluster.

After startup the application has registered itself in Kubernetes.

We can view details of employee endpoint using kubectl describe endpoints command as shown below.

Finally we can perform some test calls, for example via gateway-service running on Minikube.

$ curl http://192.168.99.100:31854/employee/actuator/info

Since our Spring Cloud Kubernetes example does not allow discovery across all namespaces for a Ribbon client, we should override Ribbon configuration using DiscoveryClient as shown below. For more details you may refer to my article Microservices With Spring Cloud Kubernetes.

public class RibbonConfiguration {

    @Autowired
    private DiscoveryClient discoveryClient;

    private String serviceId = "client";
    protected static final String VALUE_NOT_SET = "__not__set__";
    protected static final String DEFAULT_NAMESPACE = "ribbon";

    public RibbonConfiguration () {
    }

    public RibbonConfiguration (String serviceId) {
        this.serviceId = serviceId;
    }

    @Bean
    @ConditionalOnMissingBean
    public ServerList<?> ribbonServerList(IClientConfig config) {

        Server[] servers = discoveryClient.getInstances(config.getClientName()).stream()
                .map(i -> new Server(i.getHost(), i.getPort()))
                .toArray(Server[]::new);

        return new StaticServerList(servers);
    }

}

Summary

There are some limitations related to discovery with Kubernetes. For example, there is no built-in heartbeat mechanism, so we should take care of removing application endpoints on shutdown. Also, I’m not considering security aspects related to allowing discovery across all namespaces and allowing access to API for external applications. I’m assuming you have guaranteed the required level of security when building your Kubernetes cluster, especially if you decide to allow external access to the API. In fact, API is still just API and we may use it. This article shows an example of a use case, which may be useful for you. If you compare it with my previous article with Spring Cloud Kubernetes example you see that with small configuration changes you can move an application outside a cluster without adding any new components for discovery or a distributed configuration.

The post Spring Cloud Kubernetes For Hybrid Microservices Architecture appeared first on Piotr's TechBlog.

Why Spring Cloud Kubernetes?

At the time we were migrating our microservices to OpenShift the project Spring Cloud Kubernetes was in the incubation stage. Since we haven’t got any other interesting choices for migration from Spring Cloud to OpenShift consisted in removing components for discovery (Eureka client) and config (Spring Cloud Config client) from Spring Boot application. Of course, we were still able to use other Spring Cloud components like OpenFeign, Ribbon (via Kubernetes services) or Sleuth. So, the question is do we really need Spring Cloud Kubernetes? And what features would be interesting for us.
First, let’s take a look at the motivation of building a new framework available on Spring Cloud Kubernetes documentation site.

Spring Cloud Kubernetes provide Spring Cloud common interface implementations that consume Kubernetes native services. The main objective of the projects provided in this repository is to facilitate the integration of Spring Cloud and Spring Boot applications running inside Kubernetes.

In simple terms, Spring Cloud Kubernetes provides integration with Kubernetes Master API to allow using discovery, config and load balancing in Spring Cloud way.
In this article I’m going to present the following useful features of Spring Cloud Kubernetes:

• Extending discovery across all namespaces with DiscoveryClient support
• Using ConfigMap and Secrets as Spring Boot property sources with Spring Cloud Kubernetes Config
• Implementing health check using Spring Cloud Kubernetes pod health indicator

Enable Spring Cloud Kubernetes

Assuming we will use more of the features provided by Spring Cloud Kubernetes we should include the following dependency to our Maven pom.xml. It contains modules for discovery, configuration and Ribbon load balancing.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
</dependency>

Source code

The source code of the sample applications is available under branch hybrid in sample-spring-microservices-kubernetes repository: https://github.com/piomin/sample-spring-microservices-kubernetes/tree/hybrid. In the master branch you may find the example for my previous article about Spring Boot microservices deployed on Kubernetes: Quick Guide to Microservices with Kubernetes, Spring Boot 2.0 and Docker.

Discovery across all namespaces

Spring Cloud Kubernetes allows to integrate Kubernetes discovery with Spring Boot application by providing implementation of DiscoveryClient. We can also take advantage of built-in integration with Ribbon client for communication directly to pods without using Kubernetes services. Ribbon client can be leveraged by higher-level HTTP client – OpenFeign. To implement such a model we have to enable a discovery client, Feign clients and Mongo repositories, since we use the Mongo database as a backend store.

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
@EnableMongoRepositories
public class DepartmentApplication {
   
   public static void main(String[] args) {
      SpringApplication.run(DepartmentApplication.class, args);
   }

}

Let’s consider the scenario where we have three microservices, each of them deployed in a different namespace. Divide into namespaces is just a logical grouping, for example we have three different teams responsible for every single microservice and we would like to give privileges to a namespace only to a team responsible for a given application. In communication between applications located in different namespaces we have to include a namespace name as a prefix on the calling URL. We also need to set a port number which may differ between applications. Spring Cloud Kubernetes discovery comes with help in such situations. Since Spring Cloud Kubernetes is integrated with master API it is able to get IPs of all pods created for the same application. Here’s the diagram that illustrates our scenario.

To enable discovery across all namespaces we just need use the following property.

spring:
  cloud:
    kubernetes:
      discovery:
        all-namespaces: true

Now, we can implement the Feign client interface responsible for consuming the target endpoint. Here’s a sample client from department-service dedicated for communication with employee-service.

@FeignClient(name = "employee")
public interface EmployeeClient {

   @GetMapping("/department/{departmentId}")
   List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
   
}

Spring Cloud Kubernetes requires access to Kubernetes API in order to be able to retrieve a list of addresses of pods running for a single service. The simplest way to do that when using Minikube is to create default ClusterRoleBinding with cluster-admin privilege. After running the following command you can be sure that every pod will have sufficient privileges to communicate with Kubernetes API.

$ kubectl create clusterrolebinding admin --clusterrole=cluster-admin --serviceaccount=default:default

Configuration with Kubernetes PropertySource

Spring Cloud Kubernetes PropertySource implementation allows us to use ConfigMap and Secret directly in the application without injecting them into Deployment. The default behaviour is based on metadata.name inside ConfigMap or Secret, which has to be the same as an application name (as defined by its spring.application.name property). You can also use more advanced behaviour where you may define a custom name of namespace and object for configuration injection. You can even use multiple ConfigMap or Secret instances. However, we use the default behaviour, so assuming we have the following bootstrap.yml:

spring:
  application:
    name: employee

We are going to define the following ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: employee
data:
  logging.pattern.console: "%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
  spring.cloud.kubernetes.discovery.all-namespaces: "true"
  spring.data.mongodb.database: "admin"
  spring.data.mongodb.host: "mongodb.default"

Alternatively you can use an embedded YAML file in ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: employee
data:
  application.yaml: |-
    logging.pattern.console: "%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
    spring.cloud.kubernetes.discovery.all-namespaces: true
    spring:
      data:
        mongodb:
          database: admin
          host: mongodb.default

In config map we define Mongo location, logs pattern and property responsible for allowing multi-namespace discovery. Mongo credentials should be defined inside Secret object. The rules are the same as for config maps.

apiVersion: v1
kind: Secret
metadata:
  name: employee
type: Opaque
data:
  spring.data.mongodb.username: UGlvdF8xMjM=
  spring.data.mongodb.password: cGlvdHI=

It is worth to note that by default, consuming secrets through the API is not enabled for security reasons. However, we have already set default cluster-admin role, so we don’t have to worry about it. The only thing we have to do is to enable consuming secrets through API for Spring Cloud Kubernetes, which is disabled by default. To do that we have to use set the following property in bootstrap.yml.

spring:
  cloud:
    kubernetes:
      secrets:
        enableApi: true

Deploying Spring Cloud apps on Minikube

First, let’s create required namespaces using kubectl create namespace command. Here are the commands that create namespaces a, b, c and d.

Then, let’s build the code by executing Maven mvn clean install command.

We also need to set cluster-admin for newly created namespaces in order to allow pods running inside these namespaces to read master API.

Now, let’s take a look on our Kubernetes deployment manifest. It is very simple, since it does not inject any properties from ConfigMap and Secret. It is already performed by Spring Cloud Kubernetes Config. Here’s a deployment YAML file for employee-service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: employee
  labels:
    app: employee
spec:
  replicas: 1
  selector:
    matchLabels:
      app: employee
  template:
    metadata:
      labels:
        app: employee
    spec:
      containers:
      - name: employee
        image: piomin/employee:1.1
        ports:
        - containerPort: 8080

Finally, we may deploy our applications on Kubernetes. Each microservice has ConfigMap, Secret, Deployment and Service objects. The YAML manifest is available in the Git repository inside /kubernetes directory. We are applying them sequentially using kubectl apply command as shown below.

For the test purposes you may expose the sample application outside a node by defining NodePort type.

apiVersion: v1
kind: Service
metadata:
  name: department
  labels:
    app: department
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: department
  type: NodePort

Exposing info about a pod

If you defined your Service as NodePort you can easily access it outside Minikube. To retrieve a target port just execute kubectl get svc as shown below. Now, you would be able to call it using address http://192.168.99.100:31119.

With Spring Cloud Kubernetes each Spring Boot application exposes information about pod ip, pod name and namespace name. To enter it you need to call /info endpoint as shown below.

Here’s the list of pods distributed between all namespaces after deploying all sample microservices and gateway.

And also a list of deployments.

Running gateway

The last element in our architecture is the gateway. We use Spring Cloud Netflix Zuul, which is integrated with Kubernetes discovery via Ribbon client. It is exposing Swagger documentation for all our sample microservices distributed across multiple namespaces. Here’s a list of required dependencies.

<dependencies>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-netflix-zuul</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-kubernetes-all</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-starter-sleuth</artifactId>
   </dependency>
   <dependency>
      <groupId>io.springfox</groupId>
      <artifactId>springfox-swagger-ui</artifactId>
      <version>2.9.2</version>
   </dependency>
   <dependency>
      <groupId>io.springfox</groupId>
      <artifactId>springfox-swagger2</artifactId>
      <version>2.9.2</version>
   </dependency>
</dependencies>

The configuration of routes is pretty simple. We just need to use the Spring Cloud Kubernetes discovery feature.

apiVersion: v1
kind: ConfigMap
metadata:
  name: gateway
data:
  logging.pattern.console: "%d{HH:mm:ss} ${LOG_LEVEL_PATTERN:-%5p} %m%n"
  spring.cloud.kubernetes.discovery.all-namespaces: "true"
  zuul.routes.department.path: "/department/**"
  zuul.routes.employee.path: "/employee/**"
  zuul.routes.organization.path: "/organization/**"

While Zuul proxy is automatically integrated with DiscoveryClient we may easily configure dynamic resolution Swagger endpoints exposed by microservices.

@Configuration
public class GatewayApi {

   @Autowired
   ZuulProperties properties;

   @Primary
   @Bean
   public SwaggerResourcesProvider swaggerResourcesProvider() {
      return () -> {
         List<SwaggerResource> resources = new ArrayList<>();
         properties.getRoutes().values().stream()
               .forEach(route -> resources.add(createResource(route.getId(), "2.0")));
         return resources;
      };
   }

   private SwaggerResource createResource(String location, String version) {
      SwaggerResource swaggerResource = new SwaggerResource();
      swaggerResource.setName(location);
      swaggerResource.setLocation("/" + location + "/v2/api-docs");
      swaggerResource.setSwaggerVersion(version);
      return swaggerResource;
   }

}

Normally, we would have to configure Kubernetes Ingress in order to access gateway. With Minikube we just have to create a service with type NodePort. Finally, we may start testing our applications using the Swagger UI exposed on the gateway. But here, we get an unexpected surprise… The discovery across all namespaces does not work for the Ribbon client. It only works for DiscoveryClient. I think that Ribbon auto-configuration should respect the property spring.cloud.kubernetes.discovery.all-namespaces, but in that case we don’t have any other choice than prepare a workaround. Our workaround is to override Ribbon client auto-configuration provided within Spring Cloud Kubernetes. We are using DiscoveryClient directly for it as shown below.

public class RibbonConfiguration {

    @Autowired
    private DiscoveryClient discoveryClient;

    private String serviceId = "client";
    protected static final String VALUE_NOT_SET = "__not__set__";
    protected static final String DEFAULT_NAMESPACE = "ribbon";

    public RibbonConfiguration () {
    }

    public RibbonConfiguration (String serviceId) {
        this.serviceId = serviceId;
    }

    @Bean
    @ConditionalOnMissingBean
    public ServerList<?> ribbonServerList(IClientConfig config) {

        Server[] servers = discoveryClient.getInstances(config.getClientName()).stream()
                .map(i -> new Server(i.getHost(), i.getPort()))
                .toArray(Server[]::new);

        return new StaticServerList(servers);
    }

}

The Ribbon configuration class needs to be set on the main class.

@SpringBootApplication
@EnableDiscoveryClient
@EnableZuulProxy
@EnableSwagger2
@AutoConfigureAfter(RibbonAutoConfiguration.class)
@RibbonClients(defaultConfiguration = RibbonConfiguration.class)
public class GatewayApplication {

   public static void main(String[] args) {
      SpringApplication.run(GatewayApplication.class, args);
   }

}

Now, we can finally take advantage of multi namespace discovery and load balancing and easily test it using the Swagger UI exposed on the gateway.

Summary

Spring Cloud Kubernetes is currently one of the most popular Spring Cloud projects. In this context, it may be a little surprising that it is not up-to-date with the latest Spring Cloud features. For example, it still uses Ribbon instead of the new Spring Cloud Load Balancer. Anyway, it provides some useful mechanisms that simplifies Spring Boot application deployment on Kubernetes. In this article I presented the most useful features like discovery across all namespaces or configuration property sources with Kubernetes ConfigMap and Secret.

The post Microservices With Spring Cloud Kubernetes appeared first on Piotr's TechBlog.

The topics covered in this article are:

• Using Spring Boot 2 in cloud-native development
• Providing service discovery for all microservices using Spring Cloud Kubernetes project
• Injecting configuration settings into application pods using Kubernetes Config Maps and Secrets
• Building application images using Docker and deploying them on Kubernetes using YAML configuration files
• Using Spring Cloud Kubernetes together with Zuul proxy to expose a single Swagger API documentation for all microservices

Spring Cloud and Kubernetes may be threatened as competitive solutions when you build microservices environments. Such components like Eureka, Spring Cloud Config, or Zuul provided by Spring Cloud may be replaced by built-in Kubernetes objects like services, config maps, secrets, or ingresses. But even if you decide to use Kubernetes components instead of Spring Cloud you can take advantage of some interesting features provided throughout the whole Spring Cloud project.

The one really interesting project that helps us in development is Spring Cloud Kubernetes (https://github.com/spring-cloud/spring-cloud-kubernetes). Although it is still in the incubation stage it is definitely worth dedicating some time to it. It integrates Spring Cloud with Kubernetes. I’ll show you how to use the implementation of discovery client, inter-service communication with Ribbon client, and Zipkin discovery using Spring Cloud Kubernetes.

Before we proceed to the source code, let’s take a look at the following diagram. It illustrates the architecture of our sample system. It is quite similar to the architecture presented in the already mentioned article about microservices on Spring Cloud. There are three independent applications (employee-service, department-service, organization-service), which communicate between each other through REST API. These Spring Boot microservices use some build-in mechanisms provided by Kubernetes: config maps and secrets for distributed configuration, etcd for service discovery, and ingresses for API gateway.

Let’s proceed to the implementation. Currently, the newest stable version of Spring Cloud is Finchley.RELEASE. This version of spring-cloud-dependencies should be declared as a BOM for dependency management.

<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Finchley.RELEASE</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

Spring Cloud Kubernetes is not released under Spring Cloud Release Trains. So, we need to explicitly define its version. Because we use Spring Boot 2.0 we have to include the newest SNAPSHOT version of spring-cloud-kubernetes artifacts, which is 0.3.0.BUILD-SNAPSHOT.

The source code of sample Spring Boot microservices on Kubernetes presented in this article is available on GitHub in repository https://github.com/piomin/sample-spring-microservices-kubernetes.git.

Pre-requirements

In order to be able to deploy and test our sample microservices we need to prepare a development environment. We can realize that in the following steps:

• You need at least a single node cluster instance of Kubernetes (Minikube) or Openshift (Minishift) running on your local machine. You should start it and expose the embedded Docker client provided by both of them. The detailed instruction for Minishift may be found there: Quick guide to deploying Java apps on OpenShift. You can also use that description to run Minikube – just replace word ‘minishift’ with ‘minikube’. In fact, it does not matter if you choose Kubernetes or Openshift – the next part of this tutorial would be applicable for both of them
• Spring Cloud Kubernetes requires access to Kubernetes API in order to be able to retrieve a list of addresses of pods running for a single service. If you use Kubernetes you should just execute the following command:

$ kubectl create clusterrolebinding admin --clusterrole=cluster-admin --serviceaccount=default:default

If you deploy your microservices on Minishift you should first enable admin-user addon, then login as a cluster-admin and grant required permissions.

$ minishift addons enable admin-user
$ oc login -u system:admin
$ oc policy add-role-to-user cluster-reader system:serviceaccount:myproject:default

• All our sample microservices use MongoDB as a backend store. So, you should first run an instance of this database on your node. With Minishift it is quite simple, as you can use predefined templates just by selecting service Mongo on the Catalog list. With Kubernetes, the task is more difficult. You have to prepare deployment configuration files by yourself and apply it to the cluster. All the configuration files are available under kubernetes directory inside sample Git repository. To apply the following YAML definition to the cluster you should execute command kubectl apply -f kubernetes\mongo-deployment.yaml. After it, Mongo database would be available under the name mongodb inside Kubernetes cluster.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb
  labels:
    app: mongodb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:latest
        ports:
        - containerPort: 27017
        env:
        - name: MONGO_INITDB_DATABASE
          valueFrom:
            configMapKeyRef:
              name: mongodb
              key: database-name
        - name: MONGO_INITDB_ROOT_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-user
        - name: MONGO_INITDB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-password
---
apiVersion: v1
kind: Service
metadata:
  name: mongodb
  labels:
    app: mongodb
spec:
  ports:
  - port: 27017
    protocol: TCP
  selector:
    app: mongodb

1. Inject configuration with Config Maps and Secrets

When using Spring Cloud the most obvious choice for realizing distributed configuration in your system is Spring Cloud Config. With Kubernetes, you can use the Config Map. It holds key-value pairs of configuration data that can be consumed in pods or used to store configuration data. It is used for storing and sharing non-sensitive, unencrypted configuration information. To use sensitive information in your clusters, you must use Secrets. Any usage of both these Kubernetes objects can be perfectly demonstrated based on the example of MongoDB connection settings. Inside the Spring Boot application, we can easily inject it using environment variables. Here’s a fragment of application.yml file with URI configuration.

spring:
  data:
    mongodb:
      uri: mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@mongodb/${MONGO_DATABASE}

While username or password are sensitive fields, a database name is not. So we can place it inside a config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongodb
data:
  database-name: microservices

Of course, username and password are defined as secrets.

apiVersion: v1
kind: Secret
metadata:
  name: mongodb
type: Opaque
data:
  database-password: MTIzNDU2
  database-user: cGlvdHI=

To apply the configuration to the Kubernetes cluster we run the following commands.

$ kubectl apply -f kubernetes/mongodb-configmap.yaml
$ kubectl apply -f kubernetes/mongodb-secret.yaml

After it we should inject the configuration properties into application’s pods. When defining container configuration inside Deployment YAML file we have to include references to environment variables and secrets as shown below

apiVersion: apps/v1
kind: Deployment
metadata:
  name: employee
  labels:
    app: employee
spec:
  replicas: 1
  selector:
    matchLabels:
      app: employee
  template:
    metadata:
      labels:
        app: employee
    spec:
      containers:
      - name: employee
        image: piomin/employee:1.0
        ports:
        - containerPort: 8080
        env:
        - name: MONGO_DATABASE
          valueFrom:
            configMapKeyRef:
              name: mongodb
              key: database-name
        - name: MONGO_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-user
        - name: MONGO_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongodb
              key: database-password

2. Building Spring Boot microservices discovery with Kubernetes

We usually run microservices on Kubernetes using Docker containers. One or more containers are grouped by pods, which are the smallest deployable units created and managed in Kubernetes. A good practice is to run only one container inside a single pod. If you would like to scale up your microservice you would just have to increase a number of running pods. All running pods that belong to a single microservice are logically grouped by Kubernetes Service. This service may be visible outside the cluster, and is able to load balance incoming requests between all running pods. The following service definition groups all pods labelled with field app equaled to employee.

apiVersion: v1
kind: Service
metadata:
  name: employee
  labels:
    app: employee
spec:
  ports:
  - port: 8080
    protocol: TCP
  selector:
    app: employee

Service can be used for accessing applications outside a Kubernetes cluster or for inter-service communication inside a cluster. However, the communication between microservices can be implemented more comfortably with Spring Cloud Kubernetes. First, we need to include the following dependency to project pom.xml.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes</artifactId>
   <version>0.3.0.BUILD-SNAPSHOT</version>
</dependency>

Then we should enable a discovery client for an application – the same as we have always done for discovery Spring Cloud Netflix Eureka. This allows you to query Kubernetes endpoints (services) by name. This discovery feature is also used by the Spring Cloud Kubernetes Ribbon or Zipkin projects to fetch respectively the list of the pods defined for a microservice to be load balanced or the Zipkin servers available to send the traces or spans.

@SpringBootApplication
@EnableDiscoveryClient
@EnableMongoRepositories
@EnableSwagger2
public class EmployeeApplication {

   public static void main(String[] args) {
      SpringApplication.run(EmployeeApplication.class, args);
   }
   
   // ...
}

The last important thing in this section is to guarantee that Spring Boot microservices name would be exactly the same as the Kubernetes service name for the application. For application employee-service it is employee.

spring:
  application:
    name: employee

3. Building microservice using Docker and deploying on Kubernetes

There is nothing unusual in our sample microservices. We have included some standard Spring dependencies for building REST-based microservices, integrating with MongoDB and generating API documentation using Swagger2.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-swagger2</artifactId>
   <version>2.9.2</version>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

In order to integrate with MongoDB we should create interface that extends standard Spring Data CrudRepository.

public interface EmployeeRepository extends CrudRepository<Employee, String> {
   
   List<Employee> findByDepartmentId(Long departmentId);
   List<Employee> findByOrganizationId(Long organizationId);
   
}

Entity class should be annotated with Mongo @Document and a primary key field with @Id.

@Document(collection = "employee")
public class Employee {

   @Id
   private String id;
   private Long organizationId;
   private Long departmentId;
   private String name;
   private int age;
   private String position;
   
   // ...
   
}

The repository bean has been injected to the controller class. Here’s the full implementation of our REST API inside employee-service.

@RestController
public class EmployeeController {

   private static final Logger LOGGER = LoggerFactory.getLogger(EmployeeController.class);
   
   @Autowired
   EmployeeRepository repository;
   
   @PostMapping("/")
   public Employee add(@RequestBody Employee employee) {
      LOGGER.info("Employee add: {}", employee);
      return repository.save(employee);
   }
   
   @GetMapping("/{id}")
   public Employee findById(@PathVariable("id") String id) {
      LOGGER.info("Employee find: id={}", id);
      return repository.findById(id).get();
   }
   
   @GetMapping("/")
   public Iterable<Employee> findAll() {
      LOGGER.info("Employee find");
      return repository.findAll();
   }
   
   @GetMapping("/department/{departmentId}")
   public List<Employee> findByDepartment(@PathVariable("departmentId") Long departmentId) {
      LOGGER.info("Employee find: departmentId={}", departmentId);
      return repository.findByDepartmentId(departmentId);
   }
   
   @GetMapping("/organization/{organizationId}")
   public List<Employee> findByOrganization(@PathVariable("organizationId") Long organizationId) {
      LOGGER.info("Employee find: organizationId={}", organizationId);
      return repository.findByOrganizationId(organizationId);
   }
   
}

In order to run our microservices on Kubernetes we should first build the whole Maven project with mvn clean install command. Each microservice has Dockerfile placed in the root directory. Here’s Dockerfile definition for employee-service.

FROM openjdk:8-jre-alpine
ENV APP_FILE employee-service-1.0-SNAPSHOT.jar
ENV APP_HOME /usr/apps
EXPOSE 8080
COPY target/$APP_FILE $APP_HOME/
WORKDIR $APP_HOME
ENTRYPOINT ["sh", "-c"]
CMD ["exec java -jar $APP_FILE"]

Let’s build Docker images for all three sample microservices.

$ cd employee-service
$ docker build -t piomin/employee:1.0 .
$ cd department-service
$ docker build -t piomin/department:1.0 .
$ cd organization-service
$ docker build -t piomin/organization:1.0 .

The last step is to deploy Docker containers with applications on Kubernetes. To do that just execute commands kubectl apply on YAML configuration files. The sample deployment file for employee-service has been demonstrated in step 1. All required deployment fields are available inside the project repository in kubernetes directory.

$ kubectl apply -f kubernetes\employee-deployment.yaml
$ kubectl apply -f kubernetes\department-deployment.yaml
$ kubectl apply -f kubernetes\organization-deployment.yaml

4. Communication between microservices with Spring Cloud Kubernetes Ribbon

All the microservice are deployed on Kubernetes. Now, it’s worth discussing some aspects related to inter-service communication. Application employee-service in contrast to other microservices did not invoke any other microservices. Let’s take a look at other microservices that call API exposed by employee-service and communicate between each other (organization-service calls department-service API).
First we need to include some additional dependencies to the project. We use Spring Cloud Ribbon and OpenFeign. Alternatively you can also use Spring @LoadBalanced RestTemplate.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId>
   <version>0.3.0.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

Here’s the main class of department-service. It enables Feign client using @EnableFeignClients annotation. It works the same as with discovery based on Spring Cloud Netflix Eureka. OpenFeign uses Ribbon for client-side load balancing. Spring Cloud Kubernetes Ribbon provides some beans that forces Ribbon to communicate with Kubernetes API through Fabric8 KubernetesClient.

@SpringBootApplication
@EnableDiscoveryClient
@EnableFeignClients
@EnableMongoRepositories
@EnableSwagger2
public class DepartmentApplication {
   
   public static void main(String[] args) {
      SpringApplication.run(DepartmentApplication.class, args);
   }
   
   // ...
   
}

Here’s implementation of Feign client for calling method exposed by employee-service.

@FeignClient(name = "employee")
public interface EmployeeClient {

   @GetMapping("/department/{departmentId}")
   List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);
   
}

Finally, we have to inject Feign client’s beans to the REST controller. Now, we may call the method defined inside EmployeeClient, which is equivalent to calling REST endpoints.

@RestController
public class DepartmentController {

   private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);
   
   @Autowired
   DepartmentRepository repository;
   @Autowired
   EmployeeClient employeeClient;
   
   // ...
   
   @GetMapping("/organization/{organizationId}/with-employees")
   public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
      LOGGER.info("Department find: organizationId={}", organizationId);
      List<Department> departments = repository.findByOrganizationId(organizationId);
      departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
      return departments;
   }
   
}

5. Building API gateway using Kubernetes Ingress

An Ingress is a collection of rules that allow incoming requests to reach the downstream services. In our microservices architecture ingress is playing a role of an API gateway. To create it we should first prepare a YAML description file. The descriptor file should contain the hostname under which the gateway will be available and mapping rules to the downstream services.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: gateway-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  backend:
    serviceName: default-http-backend
    servicePort: 80
  rules:
  - host: microservices.info
    http:
      paths:
      - path: /employee
        backend:
          serviceName: employee
          servicePort: 8080
      - path: /department
        backend:
          serviceName: department
          servicePort: 8080
      - path: /organization
        backend:
          serviceName: organization
          servicePort: 8080

You have to execute the following command to apply the configuration visible above to the Kubernetes cluster.

$ kubectl apply -f kubernetes\ingress.yaml

For testing this solution locally we have to insert the mapping between IP address and hostname set in ingress definition inside hosts file as shown below. After it we can access services through ingress using a defined hostname just like that: http://microservices.info/employee.

192.168.99.100 microservices.info

You can check the details of created ingress just by executing command kubectl describe ing gateway-ingress.

6. Enabling API specification on gateway using Swagger2

Ok, what if we would like to expose single swagger documentation for all microservices deployed on Kubernetes? Well, here the things are getting complicated… We can run a container with Swagger UI, and map all paths exposed by the ingress manually, but it is rather not a good solution…
In that case we can use Spring Cloud Kubernetes Ribbon one more time – this time together with Spring Cloud Netflix Zuul. Zuul will act as gateway only for serving Swagger API.
Here’s the list of dependencies used in my gateway-service project.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-zuul</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes</artifactId>
   <version>0.3.0.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-ribbon</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-kubernetes-ribbon</artifactId>
   <version>0.3.0.BUILD-SNAPSHOT</version>
</dependency>
<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-swagger-ui</artifactId>
   <version>2.9.2</version>
</dependency>
<dependency>
   <groupId>io.springfox</groupId>
   <artifactId>springfox-swagger2</artifactId>
   <version>2.9.2</version>
</dependency>

A Kubernetes discovery client will detect all services exposed on the cluster. We would like to display documentation only for our three microservices. That’s why I defined the following routes for Zuul.

zuul:
  routes:
    department:
      path: /department/**
    employee:
      path: /employee/**
    organization:
      path: /organization/**

Now we can use ZuulProperties bean to get route’s addresses from Kubernetes discovery, and configure them as Swagger resources as shown below.

@Configuration
public class GatewayApi {

   @Autowired
   ZuulProperties properties;

   @Primary
   @Bean
   public SwaggerResourcesProvider swaggerResourcesProvider() {
      return () -> {
         List<SwaggerResource> resources = new ArrayList<>();
         properties.getRoutes().values().stream()
               .forEach(route -> resources.add(createResource(route.getId(), "2.0")));
         return resources;
      };
   }

   private SwaggerResource createResource(String location, String version) {
      SwaggerResource swaggerResource = new SwaggerResource();
      swaggerResource.setName(location);
      swaggerResource.setLocation("/" + location + "/v2/api-docs");
      swaggerResource.setSwaggerVersion(version);
      return swaggerResource;
   }

}

Application gateway-service should be deployed on a cluster the same as other applications. You can list the running services by executing command kubectl get svc. Swagger documentation is available under address http://192.168.99.100:31237/swagger-ui.html.

Conclusion

I’m actually rooting for the Spring Cloud Kubernetes project, which is still at the incubation stage. Kubernetes’ popularity as a platform is rapidly growing during the last months, but it still has some weaknesses. One of them is inter-service communication. Kubernetes doesn’t give us many mechanisms out-of-the-box, which allows us to configure more advanced rules. This is a reason for creating frameworks for service mesh on Kubernetes like Istio or Linkerd. While these projects are still relatively new solutions, Spring Cloud is a stable, opinionated framework. Why not to provide service discovery, inter-service communication or load balancing? Thanks to Spring Cloud Kubernetes it is possible.

The post Quick Guide to Microservices with Kubernetes, Spring Boot 2 and Docker appeared first on Piotr's TechBlog.