You can read more about Istio, Spring Boot, and other Java and Kubernetes-related technologies in my latest book, Hands-On Java with Kubernetes.

You can also find many other articles on my blog about Istio. For example, this article is about Quarkus and tracing with Istio.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Two sample applications for this exercise are available in the spring-boot-istio directory.

Prerequisites

We will start with the most straightforward Istio installation on a local Kubernetes cluster. This could be Minikube, which you can run with the following command. You can set slightly lower resource limits than I did.

minikube start --memory='8gb' --cpus='6'

minikube start --memory='8gb' --cpus='6'

To complete the exercise below, you need to install istioctl in addition to kubectl. Here you will find the available distributions for the latest versions of kubectl and istioctl. I install them on my laptop using Homebrew.

$ brew install kubectl
$ brew install istioctl

$ brew install kubectl
$ brew install istioctl

To install Istio with default parameters, run the following command:

istioctl install

istioctl install

After a moment, Istio should be running in the istio-system namespace.

It is also worth installing Kiali to verify the Istio resources we have created. Kiali is an observability and management tool for Istio that provides a web-based dashboard for service mesh monitoring. It visualizes service-to-service traffic, validates Istio configuration, and integrates with tools like Prometheus, Grafana, and Jaeger.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.28/samples/addons/kiali.yaml

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.28/samples/addons/kiali.yaml

Once Kiali is successfully installed on Kubernetes, we can expose its web dashboard locally with the following istioctl command:

istioctl dashboard kiali

istioctl dashboard kiali

To test access to the Istio mesh from outside the cluster, you need to expose the ingress gateway. To do this, run the minikube tunnel command.

Use Spring Boot Istio Library

To test our library’s functionality, we will create a simple Spring Boot application that exposes a single REST endpoint.

@RestController
@RequestMapping("/callme")
public class CallmeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(CallmeController.class);

    @Autowired
    Optional<BuildProperties> buildProperties;
    @Value("${VERSION}")
    private String version;

    @GetMapping("/ping")
    public String ping() {
        LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() ?
            buildProperties.get().getName() : "callme-service", version);
        return "I'm callme-service " + version;
    }
    
}

@RestController
@RequestMapping("/callme")
public class CallmeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(CallmeController.class);

    @Autowired
    Optional<BuildProperties> buildProperties;
    @Value("${VERSION}")
    private String version;

    @GetMapping("/ping")
    public String ping() {
        LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() ?
            buildProperties.get().getName() : "callme-service", version);
        return "I'm callme-service " + version;
    }
    
}

Then, in addition to the standard Spring Web starter, add the istio-spring-boot-starter dependency.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>com.github.piomin</groupId>
  <artifactId>istio-spring-boot-starter</artifactId>
  <version>1.2.1</version>
</dependency>

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>com.github.piomin</groupId>
  <artifactId>istio-spring-boot-starter</artifactId>
  <version>1.2.1</version>
</dependency>

Finally, we must add the @EnableIstio annotation to our application’s main class. We can also enable Istio Gateway to expose the REST endpoint outside the cluster. An Istio Gateway is a component that controls how external traffic enters or leaves a service mesh.

@SpringBootApplication
@EnableIstio(enableGateway = true)
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true)
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Let’s deploy our application on the Kubernetes cluster. To do this, we must first create a role with the necessary permissions to manage Istio resources in the cluster. The role must be assigned to the ServiceAccount used by the application.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: callme-service-with-starter
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules", "gateways"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: callme-service-with-starter
subjects:
  - kind: ServiceAccount
    name: callme-service-with-starter
    namespace: spring
roleRef:
  kind: Role
  name: callme-service-with-starter
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: callme-service-with-starter

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: callme-service-with-starter
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules", "gateways"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: callme-service-with-starter
subjects:
  - kind: ServiceAccount
    name: callme-service-with-starter
    namespace: spring
roleRef:
  kind: Role
  name: callme-service-with-starter
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: callme-service-with-starter

Here are the Deployment and Service resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
  template:
    metadata:
      labels:
        app: callme-service-with-starter
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
  template:
    metadata:
      labels:
        app: callme-service-with-starter
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

The application repository is configured to run it with Skaffold. Of course, you can apply YAML manifests to the cluster with kubectl apply. To do this, simply navigate to the callme-service-with-starter/k8s directory and apply the deployment.yaml file. As part of the exercise, we will run our applications in the spring namespace.

skaffold dev -n spring

skaffold dev -n spring

The sample Spring Boot application creates two Istio objects at startup: a VirtualService and a Gateway. We can verify them in the Kiali dashboard.

The default host name generated for the gateway includes the deployment name and the .ext suffix. We can change the suffix name using the domain field in the @EnableIstio annotation. Assuming you run the minikube tunnel command, you can call the service using the Host header with the hostname in the following way:

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext"
I'm callme-service v1

Additional Capabilities with Spring Boot Istio

The library’s behavior can be customized by modifying the @EnableIstio annotation. For example, you can enable the fault injection mechanism using the fault field. Both abort and delay are possible. You can make this change without redeploying the app. The library updates the existing VirtualService.

@SpringBootApplication
@EnableIstio(enableGateway = true, fault = @Fault(percentage = 50))
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, fault = @Fault(percentage = 50))
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Now, you can call the same endpoint through the gateway. There is a 50% chance that you will receive this response.

Now let’s analyze a slightly more complex scenario. Let’s assume that we are running two different versions of the same application on Kubernetes. We use Istio to manage its versioning. Traffic is forwarded to the particular version based on the X-Version header in the incoming request. If the header value is v1 the request is sent to the application Pod with the label version=v1, and similarly, for the version v2. Here’s the annotation for the v1 application main class.

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v1",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v1") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v1",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v1") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

The Deployment manifest for the callme-service-with-starter-v1 should define two labels: app and version.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v1
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v1
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v1
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v1
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter

Unlike the skaffold dev command, the skaffold run command simply launches the application on the cluster and terminates. Let’s first release version v1, and then move on to version v2.

skaffold run -n spring

skaffold run -n spring

Then, we can deploy the v2 version of our sample Spring Boot application. In this exercise, it is just an “artificial” version, since we deploy the same source code, but with different labels and environment variables injected in the Deployment manifest.

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v2",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v2") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v2",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v2") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Here’s the callme-service-with-starter-v2 Deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v2
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v2
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v2"
      serviceAccountName: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v2
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v2
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v2"
      serviceAccountName: callme-service-with-starter

Service, on the other hand, remains unchanged. It still refers to Pods labeled with app=callme-service-with-starter. However, this time it includes both application instances marked as v1 or v2.

apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

Version v2 should be run in the same way as before, using the skaffold run command. There are three Istio objects generated during apps startup: Gateway, VirtualService and DestinationRule.

A generated DestinationRule contains two subsets for both v1 and v2 versions.

The automatically generated VirtualService looks as follows.

kind: VirtualService
apiVersion: networking.istio.io/v1
metadata:
  name: callme-service-with-starter-route
spec:
  hosts:
  - callme-service-with-starter
  - callme-service-with-starter.ext
  gateways:
  - callme-service-with-starter
  http:
  - match:
    - headers:
        X-Version:
          prefix: v1
    route:
    - destination:
        host: callme-service-with-starter
        subset: v1
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx
  - match:
    - headers:
        X-Version:
          prefix: v2
    route:
    - destination:
        host: callme-service-with-starter
        subset: v2
      weight: 100
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx

kind: VirtualService
apiVersion: networking.istio.io/v1
metadata:
  name: callme-service-with-starter-route
spec:
  hosts:
  - callme-service-with-starter
  - callme-service-with-starter.ext
  gateways:
  - callme-service-with-starter
  http:
  - match:
    - headers:
        X-Version:
          prefix: v1
    route:
    - destination:
        host: callme-service-with-starter
        subset: v1
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx
  - match:
    - headers:
        X-Version:
          prefix: v2
    route:
    - destination:
        host: callme-service-with-starter
        subset: v2
      weight: 100
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx

To test the versioning mechanism generated using the Spring Boot Istio library, set the X-Version header for each call.

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v1"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v2"
I'm callme-service v2

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v1"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v2"
I'm callme-service v2

Conclusion

I am still working on this library, and new features will be added in the near future. I hope it will be helpful for those of you who want to get started with Istio without getting into the details of its configuration.

The post Istio Spring Boot Library Released appeared first on Piotr's TechBlog.

Here is a brief overview of all my published books.

Motivation

I won’t hide that this post is mainly directed at my blog subscribers and people who enjoy reading it and value my writing style. As you know, all posts and content on my blog, along with sample application repositories on GitHub, are always accessible to you for free. Over the past eight years, I have worked to publish high-quality content on my blog, and I plan to keep doing so. It is a part of my life, a significant time commitment, but also a lot of fun and a hobby.

I want to explain why I decided to write this book, why now, and why in this way. But first, a bit of background. I wrote my last book first, then my first book, over seven years ago. It focused on topics I was mainly involved with at the time, specifically Spring Boot and Spring Cloud. Since then, a lot of time has passed, and much has changed – not only in the technology itself but also a little in my personal life. Today, I am more involved in Kubernetes and container topics than, for example, Spring Cloud. For years, I have been helping various organizations transition from traditional application architectures to cloud-native models based on Kubernetes. Of course, Java remains my main area of expertise. Besides Spring Boot, I also really like the Quarkus framework. You can read a lot about both in my book on Kubernetes.

Based on my experience over the past few years, involving development teams is a key factor in the success of the Kubernetes platform within an organization. Ultimately, it is the applications developed by these teams that are deployed there. For developers to be willing to use Kubernetes, it must be easy for them to do so. That is why I persuade organizations to remove barriers to using Kubernetes and to design it in a way that makes it easier for development teams. On my blog and in this book, I aim to demonstrate how to quickly and simply launch applications on Kubernetes using frameworks such as Spring Boot and Quarkus.

It’s an unusual time to publish a book. AI agents are producing more and more technical content online. More often than not, instead of grabbing a book, people turn to an AI chatbot for a quick answer, though not always the best one. Still, a book that thoroughly introduces a topic and offers a step-by-step guide remains highly valuable.

Content of the Book

This book demonstrates that Java is an excellent choice for building applications that run on Kubernetes. In the first chapter, I’ll show you how to quickly build your application, create its image, and run it on Kubernetes without writing a single line of YAML or Dockerfile. This chapter also covers the minimum Kubernetes architecture you must understand to manage applications effectively in this environment. The second chapter, on the other hand, demonstrates how to effectively organize your local development environment to work with a Kubernetes cluster. You’ll see several options for running a distribution of your cluster locally and learn about the essential set of tools you should have. The third chapter outlines best practices for building applications on the Kubernetes platform. Most of the presented requirements are supported by simple examples and explanations of the benefits of meeting them. The fourth chapter presents the most valuable tools for the inner development loop with Kubernetes. After reading the first four chapters, you will understand the main Kubernetes components related to application management, enabling you to navigate the platform efficiently. You’ll also learn to leverage Spring Boot and Quarkus features to adapt your application to Kubernetes requirements.

In the following chapters, I will focus on the benefits of migrating applications to Kubernetes. The first area to cover is security. Chapter five discusses mechanisms and tools for securing applications running in a cluster. Chapter six describes Spring and Quarkus projects that enable native integration with the Kubernetes API from within applications. In chapter seven, you’ll learn about the service mesh tool and the benefits of using it to manage HTTP traffic between microservices. Chapter eight addresses the performance and scalability of Java applications in a Kubernetes environment. Chapter Eight demonstrates how to design a CI/CD process that runs entirely within the cluster, leveraging Kubernetes-native tools for pipeline building and the GitOps approach. This book also covers AI. In the final, ninth chapter, you’ll learn how to run a simple Java application that integrates with an AI model deployed on Kubernetes.

Publication

I decided to publish my book on Leanpub. Leanpub is a platform for writing, publishing, and selling books, especially popular among technical content authors. I previously published a book with Packt, but honestly, I was alone during the writing process. Leanpub is similar but offers several key advantages over publishers like Packt. First, it allows you to update content collaboratively with readers and keep it current. Even though my book is finished, I don’t rule out adding more chapters, such as on AI on Kubernetes. I also look forward to your feedback and plan to improve the content and examples in the repository continuously. Overall, this has been another exciting experience related to publishing technical content.

And when you buy such a book, you can be sure that most of the royalties go to me as the author, unlike with other publishers, where most of the royalties go to them as promoters. So, I’m looking forward to improving my book with you!

Conclusion

My book aims to bring together all the most interesting elements surrounding Java application development on Kubernetes. It is intended not only for developers but also for architects and DevOps teams who want to move to the Kubernetes platform.

The post A Book: Hands-On Java with Kubernetes appeared first on Piotr's TechBlog.

If you are looking for similar articles in the area of Kubernetes app development you can read my post about Telepresence and Skaffold. Gefyra is the alternative to Telepresence. However, there are some significant differences between those two tools. Gefyra comes with Docker as a required dependency., while with Telepresence, Docker is optional. On the other hand, Telepresence uses a sidecar pattern to inject the proxy container to intercept the traffic. Gefyra just replaces the image with the “carrier” image. You can find more details in the docs. Enough with the theory, let’s get to practice.

Prerequisites

In order to start the exercise, we need to have a running Kubernetes cluster. It can be a local instance or a remote cluster managed by the cloud provider. In this exercise, I’m using Kubernetes on the Docker Desktop.

$ kubectx -c
docker-desktop

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. The code used in this article is available in the dev branch. Then you should just follow my instructions

Install Gefyra

In the first step, we need to install the gefyra CLI. There are the installation instructions for the different environments in the docs. Once you install the CLI you can verify it with the following command:

$ gefyra version
[INFO] Gefyra client version: 1.1.2

After that, we can install Gefyra on our Kubernetes cluster. Here’s the command for installing on Docker Desktop Kubernetes:

$ gefyra up --host=kubernetes.docker.internal

It will install Gefyra using the operator. Let’s verify a list of running pods in the gefyra namespace:

$  kubectl get po -n gefyra
NAME                               READY   STATUS    RESTARTS   AGE
gefyra-operator-7ff447866b-7gzkd   1/1     Running   0          1h
gefyra-stowaway-bb96bccfd-xg7ds    1/1     Running   0          1h

If you see the running pods, it means that the tool has been successfully installed. Now, we can use Gefyra in our app development on Kubernetes.

Use Case on Kubernetes for Gefyra

We will use exactly the same set of apps and the use case as in the article about Telepresence and Skaffold. Firstly, let’s analyze that case. There are three microservices: first-service, caller-service and callme-service. All of them expose a single REST endpoint GET /ping, which returns basic information about each microservice. In order to create applications, I’m using the Spring Boot framework. Our architecture is visible in the picture below. The first-service is calling the endpoint exposed by the caller-service. Then the caller-service is calling the endpoint exposed by the callme-service. Of course, we are going to deploy all the microservices on the Kubernetes cluster.

Now, let’s assume we are implementing a new version of the caller-service. We want to easily test with two other apps running on the cluster. Therefore, our goal is to forward the traffic that is sent to the caller-service on the Kubernetes cluster to our local instance running on our Docker. On the other hand, the local instance of the caller-service should call the endpoint exposed by the instance of the callme-service running on the Kubernetes cluster.

Build and Deploy Apps with Skaffold and Jib

Before we start development of the new version of caller-service we will deploy all three sample apps. To simplify the process we will use Skaffold and Jib Maven Plugin. Thanks to that you deploy all the using a single command. Here’s the configuration of the Skaffold in the repository root directory:

apiVersion: skaffold/v4beta5
kind: Config
metadata:
  name: simple-istio-services
build:
  artifacts:
    - image: piomin/first-service
      jib:
        project: first-service
        args:
          - -Pjib
          - -DskipTests
    - image: piomin/caller-service
      jib:
        project: caller-service
        args:
          - -Pjib
          - -DskipTests
    - image: piomin/callme-service
      jib:
        project: callme-service
        args:
          - -Pjib
          - -DskipTests
  tagPolicy:
    gitCommit: {}
manifests:
  rawYaml:
    - '*/k8s/deployment.yaml'
deploy:
  kubectl: {}

For more details about the deployment process, you may refer once again to my previous article. We will deploy apps in the demo-1 namespace. Here’s the skaffold command used for that:

$ skaffold run --tail -n demo-1

Once you run the command you will deploy all apps and see their logs in the console. These are very simple Spring Boot apps, which just expose a single REST endpoint and print a log message after receiving the request. Here’s the @RestController of callme-service:

@RestController
@RequestMapping("/callme")
public class CallmeController {

   private static final Logger LOGGER = 
       LoggerFactory.getLogger(CallmeController.class);

   @Autowired
   BuildProperties buildProperties;
   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping")
   public String ping() {
      LOGGER.info("Ping: name={}, version={}", 
         buildProperties.getName(), version);
      return "I'm callme-service " + version;
   }
}

And here’s the controller of caller-service. We will modify it during our development. It calls the endpoint exposed by the callme-service using its internal Kubernetes address http://callme-service:8080.

@RestController
@RequestMapping("/caller")
public class CallerController {

   private static final Logger LOGGER = 
      LoggerFactory.getLogger(CallerController.class);

   @Autowired
   BuildProperties buildProperties;
   @Autowired
   RestTemplate restTemplate;
   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping")
   public String ping() {
      LOGGER.info("Ping: name={}, version={}", 
         buildProperties.getName(), version);
      String response = restTemplate
         .getForObject("http://callme-service:8080/callme/ping", String.class);
      LOGGER.info("Calling: response={}", response);
      return "I'm caller-service " + version + ". Calling... " + response;
   }
}

Here’s a list of deployed apps in the demo-1 namespace:

$ kubectl get deploy -n demo-1              
NAME             READY   UP-TO-DATE   AVAILABLE   AGE
caller-service   1/1     1            1           68m
callme-service   1/1     1            1           68m
first-service    1/1     1            1           68m

Development on Kubernetes with Gefyra

Connect to services running on Kubernetes

Now, I will change the code in the CallerController class. Here’s the latest development version:

Let’s build the app on the local Docker daemon. We will leverage the Jib Maven plugin once again. We need to go to the caller-service directory and build the image using the jib goal.

$ cd caller-service
$ mvn clean package -DskipTests -Pjib jib:dockerBuild

Here’s the result. The image is available on the local Docker daemon as caller-service:1.1.0.

After that, we may run the container with the app locally using the gefyra command. We use several parameters in the command visible below. Firstly, we need to set the Docker image name using the -i parameter. We simulate running the app in the demo-1 Kubernetes namespace with the -n option. Then, we set a new value for the environment variable used by the into v2 and expose the app container port outside as 8090.

$ gefyra run --rm -i caller-service:1.1.0 \
    -n demo-1 \
    -N caller-service \
    --env VERSION=v2 \
    --expose 8090:8080

Gefyra starts our dev container on the on Docker:

docker ps -l
CONTAINER ID   IMAGE                  COMMAND                  CREATED              STATUS              PORTS                    NAMES
7fec52bed474   caller-service:1.1.0   "java -cp @/app/jib-…"   About a minute ago   Up About a minute   0.0.0.0:8090->8080/tcp   caller-service

Now, let’s try to call the endpoint exposed under the local port 8090:

$ curl http://localhost:8090/caller/ping
I'm a local caller-service v2. Calling on k8s... I'm callme-service v1

Here are the logs from our local container. As you see, it successfully connected to the callme-service app running on the Kubernetes cluster:

Let’s switch to the window with the skaffold run --tail command. It displays the logs for our three apps running on Kubernetes. As expected, there are no logs for the caller-service pod since traffic was forwarded to the local container.

Intercept the traffic sent to Kubernetes

Now, let’s do another try. This time, we will call the first-service running on Kubernetes. In order to do that, we will enable port-forward for the default port.

$ kubectl port-forward svc/first-service -n demo-1 8091:8080

We can the first-service running on Kubernetes using the local port 8091. As you see, all the calls are propagated inside the Kubernetes since the caller-service version is v1.

$ curl http://localhost:8091/first/ping 
I'm first-service v1. Calling... I'm caller-service v1. Calling... I'm callme-service v1

Just to ensure let’s switch to logs printed by skaffold:

In order to intercept the traffic to a container running on Kubernetes and send it to the development container, we need to run the gefyra bridge command. In that command, we have to set the name of the container running in Gefyra using the -N parameter (it was previously set in the gefyra run command). The command will intercept the traffic sent to the caller-service pod (--target parameter) from the demo-1 namespace (-n parameter).

$ gefyra bridge -N caller-service \
    -n demo-1 \
    --port 8080:8080 \
    --target deploy/caller-service/caller-service

You should have a similar output if the bridge has been successfully established:

Let’s call the first-service via the forwarded port once again. Pay attention to the number of the caller-service version.

$ curl http://localhost:8091/first/ping
I'm first-service v1. Calling... I'm a local caller-service v2. Calling on k8s... I'm callme-service v1

Let’s do a double in the logs. Here are the logs from Kubernetes. As you see, there no caller-service logs, but just for the first-service and callme-service.

Of course, the request is forwarded to caller-service running on the local Docker, and then caller-service invokes endpoint exposed by the callme-service running on the cluster.

Once we finish the development we may remove all the bridges:

$ gefyre unbridge -A

The post Local Application Development on Kubernetes with Gefyra appeared first on Piotr's TechBlog.

If you are looking for a quick intro to the Protocol Buffers with Java you can read my article available here. It was written more than 5 years ago. However, I updated it in recent days. I also updated the repository with code examples to the latest Spring Boot 3 and Java 17.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. It contains four apps. Two of them account-service and customer-service are related to my previous article, which introduces Protocol Buffers with Java. For the current article, please refer to another two apps account-service-grpc and customer-service-grpc. They are pretty similar to the corresponding apps but use our third-party Spring Boot and gRPC communication instead of REST. Also, they need to use Spring Boot 2, because our third-party starter still doesn’t support Spring Boot 3. Anyway, once you clone the repository just follow my instructions

Generate Model Classes and Services for gRPC

In the first step, we will generate model classes and gRPC services using the .proto manifests. We need to include some additional Protobuf schemas to use the google.protobuf.* package (1). Our gRPC service will provide methods for searching accounts using various criteria and a single method for adding a new account (2). Those methods will use primitives from the google.protobuf.* package and model classes defined inside the .proto file as messages. There are two messages defined. The Account message represents a single model class. It contains three fields: id, number, and customer_id (3). The Accounts message contains a list of Account objects (4).

syntax = "proto3";

package model;

option java_package = "pl.piomin.services.grpc.account.model";
option java_outer_classname = "AccountProto";

// (1)
import "empty.proto";
import "wrappers.proto";

// (2)
service AccountsService {
  rpc FindByNumber(google.protobuf.StringValue) returns (Account) {}
  rpc FindByCustomer(google.protobuf.Int32Value) returns (Accounts) {}
  rpc FindAll(google.protobuf.Empty) returns (Accounts) {}
  rpc AddAccount(Account) returns (Account) {}
}

// (3)
message Account {
  int32 id = 1;
  string number = 2;
  int32 customer_id = 3;
}

// (4)
message Accounts {
  repeated Account account = 1;
}

As you probably remember, there are two sample Spring Boot apps. Let’s take a look at the .proto schema for the second app customer-service-grpc. It is a little bit more complicated than the previous definition. Our gRPC service will also provide several methods for searching objects and a single method for adding a new customer (1). The customer-service-grpc is communicating with the account-service-grpc app, so we need to generate Account and Accounts messages (2). Of course, you may create an additional interface module with generated Protobuf classes and share it across both our sample apps. Finally, we have to define our model classes. The Customer class contains three primitive fields id, pesel, name, the enum type, and a list of accounts assigned to the particular customer (3). There is also the Customers message containing a list of Customer objects (4).

syntax = "proto3";

package model;

option java_package = "pl.piomin.services.grpc.customer.model";
option java_outer_classname = "CustomerProto";

import "empty.proto";
import "wrappers.proto";

// (1)
service CustomersService {
  rpc FindByPesel(google.protobuf.StringValue) returns (Customer) {}
  rpc FindById(google.protobuf.Int32Value) returns (Customer) {}
  rpc FindAll(google.protobuf.Empty) returns (Customers) {}
  rpc AddCustomer(Customer) returns (Customer) {}
}

// (2)
message Account {
  int32 id = 1;
  string number = 2;
  int32 customer_id = 3;
}

message Accounts {
  repeated Account account = 1;
}

// (3)
message Customer {
  int32 id = 1;
  string pesel = 2;
  string name = 3;
  CustomerType type = 4;
  repeated Account accounts = 5;
  enum CustomerType {
    INDIVIDUAL = 0;
    COMPANY = 1;
  }
}

// (4)
message Customers {
  repeated Customer customers = 1;
}

In order to generate Java classes from .proto schemas, we will use the Maven plugin. You can between some available plugins for that. My choice fell on the protoc-jar-maven-plugin plugin. In the configuration, we need to override the default location of our .proto schema to src/main/proto. We also need to include additional Protobuf schemas we used in the .proto manifest using the includeDirectories tag. Those manifests are located inside the src/main/proto-imports directory. The output target directory is src/main/generated. By default, the plugin does not generate gRPC services. In order to enable it we need to include the outputTarget with the grpc-java type. For generating classes we will use the protoc-gen-grpc-java library.

<plugin>
  <groupId>com.github.os72</groupId>
  <artifactId>protoc-jar-maven-plugin</artifactId>
  <version>3.11.4</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>run</goal>
      </goals>
      <configuration>
        <addProtoSources>all</addProtoSources>
        <includeMavenTypes>direct</includeMavenTypes>
        <outputDirectory>src/main/generated</outputDirectory>
        <inputDirectories>
          <include>src/main/proto</include>
        </inputDirectories>
        <includeDirectories>
          <include>src/main/proto-imports</include>
        </includeDirectories>
        <outputTargets>
           <outputTarget>
             <type>java</type>
             <outputDirectory>src/main/generated</outputDirectory>
           </outputTarget>
           <outputTarget>
             <type>grpc-java</type>
             <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.57.2</pluginArtifact>  
             <outputDirectory>src/main/generated</outputDirectory>
          </outputTarget>
        </outputTargets>
      </configuration>
    </execution>
  </executions>
</plugin>

We will also attach the generated Java code under the src/main/generated as a source directory with the build-helper-maven-plugin Maven plugin.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>add-source</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>add-source</goal>
      </goals>
      <configuration>
        <sources>
          <source>src/main/generated</source>
        </sources>
      </configuration>
    </execution>
  </executions>
</plugin>

Once you execute the mvn clean package command Maven will generate the required Java classes. Here’s the final structure of directories in the account-service-grpc app after generating the Java classes.

$ tree
.
├── pom.xml
└── src
    ├── main
    │   ├── generated
    │   │   └── pl
    │   │       └── piomin
    │   │           └── services
    │   │               └── grpc
    │   │                   └── account
    │   │                       └── model
    │   │                           ├── AccountProto.java
    │   │                           └── AccountsServiceGrpc.java
    │   ├── java
    │   │   └── pl
    │   │       └── piomin
    │   │           └── services
    │   │               └── grpc
    │   │                   └── account
    │   │                       ├── AccountApplication.java
    │   │                       ├── repository
    │   │                       │   └── AccountRepository.java
    │   │                       └── service
    │   │                           └── AccountsService.java
    │   ├── proto
    │   │   └── account.proto
    │   ├── proto-imports
    │   │   ├── empty.proto
    │   │   └── wrappers.proto
    │   └── resources
    └── test
        └── java
            └── pl
                └── piomin
                    └── services
                        └── grpc
                            └── account
                                └── AccountServicesTests.java

Using the gRPC Spring Boot Starter

Once we generate the required Protobuf model classes and gRPC stubs we can proceed to the implementation. In the first step, we need to include the following Spring Boot starter:

<dependency>
  <groupId>net.devh</groupId>
  <artifactId>grpc-server-spring-boot-starter</artifactId>
  <version>2.14.0.RELEASE</version>
</dependency>

Then we have to create the gRPC service implementation class. It needs to extend the AccountsServiceImplBase generated based on the .proto declaration. We also need to annotate the whole class with the @GrpcService (1). After that, we will override all the methods exposed over gRPC. Our service is using a simple in-memory repository (2). Each method provides a parameter object and the io.grpc.stub.StreamObserver class used for returning the responses in a reactive way (3) (4).

@GrpcService // (1)
public class AccountsService extends AccountsServiceGrpc.AccountsServiceImplBase {

    @Autowired
    AccountRepository repository; // (2)

    @Override
    public void findByNumber(StringValue request, StreamObserver<AccountProto.Account> responseObserver) { // (3)
        AccountProto.Account a = repository.findByNumber(request.getValue());
        responseObserver.onNext(a); # (4)
        responseObserver.onCompleted();
    }

    @Override
    public void findByCustomer(Int32Value request, StreamObserver<AccountProto.Accounts> responseObserver) {
        List<AccountProto.Account> accounts = repository.findByCustomer(request.getValue());
        AccountProto.Accounts a = AccountProto.Accounts.newBuilder().addAllAccount(accounts).build();
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }

    @Override
    public void findAll(Empty request, StreamObserver<AccountProto.Accounts> responseObserver) {
        List<AccountProto.Account> accounts = repository.findAll();
        AccountProto.Accounts a = AccountProto.Accounts.newBuilder().addAllAccount(accounts).build();
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }

    @Override
    public void addAccount(AccountProto.Account request, StreamObserver<AccountProto.Account> responseObserver) {
        AccountProto.Account a = repository.add(request.getCustomerId(), request.getNumber());
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }
}

Here’s the AccountRepository implementation:

public class AccountRepository {

    List<AccountProto.Account> accounts;
    AtomicInteger id;

    public AccountRepository(List<AccountProto.Account> accounts) {
        this.accounts = accounts;
        this.id = new AtomicInteger();
        this.id.set(accounts.size());
    }

    public List<AccountProto.Account> findAll() {
        return accounts;
    }

    public List<AccountProto.Account> findByCustomer(int customerId) {
        return accounts.stream().filter(it -> it.getCustomerId() == customerId).toList();
    }

    public AccountProto.Account findByNumber(String number) {
        return accounts.stream()
                .filter(it -> it.getNumber().equals(number))
                .findFirst()
                .orElseThrow();
    }

    public AccountProto.Account add(int customerId, String number) {
        AccountProto.Account a = AccountProto.Account.newBuilder()
                .setId(id.incrementAndGet())
                .setCustomerId(customerId)
                .setNumber(number)
                .build();
        return a;
    }

}

We are adding some test data on startup. Here’s our app main class:

@SpringBootApplication
public class AccountApplication {

    public static void main(String[] args) {
        SpringApplication.run(AccountApplication.class, args);
    }

    @Bean
    AccountRepository repository() {
        List<AccountProto.Account> accounts = new ArrayList<>();
        accounts.add(AccountProto.Account.newBuilder().setId(1).setCustomerId(1).setNumber("111111").build());
        accounts.add(AccountProto.Account.newBuilder().setId(2).setCustomerId(2).setNumber("222222").build());
        accounts.add(AccountProto.Account.newBuilder().setId(3).setCustomerId(3).setNumber("333333").build());
        accounts.add(AccountProto.Account.newBuilder().setId(4).setCustomerId(4).setNumber("444444").build());
        accounts.add(AccountProto.Account.newBuilder().setId(5).setCustomerId(1).setNumber("555555").build());
        accounts.add(AccountProto.Account.newBuilder().setId(6).setCustomerId(2).setNumber("666666").build());
        accounts.add(AccountProto.Account.newBuilder().setId(7).setCustomerId(2).setNumber("777777").build());
        return new AccountRepository(accounts);
    }
}

Before starting the app, we will also include Spring Boot Actuator to expose some metrics related to gRPC. We will expose under a different port than the gRPC service, so we also need to include the Spring Boot Web starter:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

In the application.yml file we should enable the metrics endpoint:

spring.application.name: account-service-grpc

management.endpoints.web.exposure.include: metrics
management.endpoint.metrics.enabled: true

By default, the gRPC services are available under the 9090 port. We can override that number using the grpc.server.port property. Set the port to 0 to use a free random port. Let’s start our sample app:

Calling gRPC Services

We can use the grpcurl CLI tool to call the gRPC services exposed by our sample app. By default, the gRPC server will be started on a port 9090 using PLAINTEXT mode. In order to print a list of available services we need to execute the following command:

$ grpcurl --plaintext localhost:9090 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
model.AccountsService

Then, let’s print the list of methods exposed by the model.AccountService:

$ grpcurl --plaintext localhost:9090 list model.AccountsService
model.AccountsService.AddAccount
model.AccountsService.FindAll
model.AccountsService.FindByCustomer
model.AccountsService.FindByNumber

We can also print the details about each method by using the describe keyword in the command:

$ grpcurl --plaintext localhost:9090 describe model.AccountsService.FindByNumber
model.AccountsService.FindByNumber is a method:
rpc FindByNumber ( .google.protobuf.StringValue ) returns ( .model.Account );

Now, let’s call the endpoint described with the command visible above. The name of our method is model.AccountsService.FindByNumber. We are also setting the input string parameter with the 111111 value.

$ grpcurl --plaintext -d '"111111"' localhost:9090 model.AccountsService.FindByNumber
{
  "id": 1,
  "number": "111111",
  "customer_id": 1
}

After that, we can take a look at the model.AccountsService.FindByNumber gRPC method. It takes an integer as the input parameter and returns a list of objects.

$ grpcurl --plaintext -d '1' localhost:9090 model.AccountsService.FindByCustomer
{
  "account": [
    {
      "id": 1,
      "number": "111111",
      "customer_id": 1
    },
    {
      "id": 5,
      "number": "555555",
      "customer_id": 1
    }
  ]
}

Finally, we can call the method for adding a new account. It takes the JSON object as the input parameter. Then it will return a newly created Account object with the incremented id field.

$ grpcurl --plaintext -d '{"customer_id": 6, "number": "888888"}' localhost:9090 model.AccountsService.AddAccount
{
  "id": 8,
  "number": "888888",
  "customer_id": 6
}

The gRPC Spring Boot starter adds three additional metrics to the Actuator.

We can display a number of requests per a gRPC method. Here’s the request and response for the FindByNumber method.

We can also display an average processing time per method as shown below.

Testing the gRPC Services

In the previous section, we ran the app and tested gRPC services manually using the grpcurl CLI tool. However, we can also implement unit or integrated tests based on the Spring Boot Test module. We will create the integration test for our app with the gRPC client. In order to do that, we need to include the following three dependencies in Maven pom.xml:

<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-testing</artifactId>
  <version>1.51.0</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-test</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>net.devh</groupId>
  <artifactId>grpc-client-spring-boot-starter</artifactId>
  <version>2.14.0.RELEASE</version>
  <scope>test</scope>
</dependency>

In the test implementation visible below, we need to enable an “in-process” server (1) and disable an external server (2). Then we have to configure the client to connect to the “in-process server” (3). We will use the gRPC client already generated during the Maven build. It is available as the AccountsServiceBlockingStub class. We just to inject and annotate it properly with the @GrpcClient (4). After that, we may use the client stub to call our gRPC services (5).

@SpringBootTest(properties = {
        "grpc.server.inProcessName=test", // (1)
        "grpc.server.port=-1", // (2)
        "grpc.client.inProcess.address=in-process:test" // (3)
})
@DirtiesContext
public class AccountServicesTests {

    @GrpcClient("inProcess") // (4)
    AccountsServiceGrpc.AccountsServiceBlockingStub service;

    @Test
    void shouldFindAll() {
        AccountProto.Accounts a = service.findAll(Empty.newBuilder().build()); // (5)
        assertNotNull(a);
        assertFalse(a.getAccountList().isEmpty());
    }

    @Test
    void shouldFindByCustomer() {
        AccountProto.Accounts a = service.findByCustomer(Int32Value.newBuilder().setValue(1).build());
        assertNotNull(a);
        assertFalse(a.getAccountList().isEmpty());
    }

    @Test
    void shouldFindByNumber() {
        AccountProto.Account a = service.findByNumber(StringValue.newBuilder().setValue("111111").build());
        assertNotNull(a);
        assertNotEquals(0, a.getId());
    }

    @Test
    void shouldAddAccount() {
        AccountProto.Account a = AccountProto.Account.newBuilder()
                .setNumber("123456")
                .setCustomerId(10)
                .build();

        a = service.addAccount(a);
        assertNotNull(a);
        assertNotEquals(0, a.getId());
    }

}

Here are the results of our tests:

Communication Between gRPC Microservices

In this section, we will switch to the customer-service-grpc app. The same as with the previous app, we need to generate the classes and gRPC service stubs with the Maven command mvn clean package. The service implementation is also similar to the account-service-grpc. However, this time, we use a client to call the external gRPC method. Here’s the implementation of the @GrpcService. As you see, we are injecting the AccountClient bean and then using it to call the gRPC method exposed by the account-service-grpc app (1). Then we use the client bean to find the account assigned to the particular customer (2).

@GrpcService
public class CustomersService extends CustomersServiceGrpc.CustomersServiceImplBase {

    @Autowired
    CustomerRepository repository;
    @Autowired
    AccountClient accountClient; // (1)

    @Override
    public void findById(Int32Value request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.findById(request.getValue());
        CustomerProto.Accounts a = accountClient.getAccountsByCustomerId(c.getId()); // (2)
        List<CustomerProto.Account> l = a.getAccountList();
        c = CustomerProto.Customer.newBuilder(c).addAllAccounts(l).build();
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void findByPesel(StringValue request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.findByPesel(request.getValue());
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void findAll(Empty request, StreamObserver<CustomerProto.Customers> responseObserver) {
        List<CustomerProto.Customer> customerList = repository.findAll();
        CustomerProto.Customers c = CustomerProto.Customers.newBuilder().addAllCustomers(customerList).build();
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void addCustomer(CustomerProto.Customer request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.add(request.getType(), request.getName(), request.getPesel());
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }
}

Now, let’s take a look at the implementation of the AccountClient class. We use the generated client stub to call the external gRPC method (1). Pay attention to the value inside inside the annotation. It is the name of our client.

@Service
public class AccountClient {

    private static final Logger LOG = LoggerFactory.getLogger(AccountClient.class);

    @GrpcClient("account-service-grpc") // (1)
    AccountsServiceGrpc.AccountsServiceBlockingStub stub;

    public CustomerProto.Accounts getAccountsByCustomerId(int customerId) {
        try {
            return stub.findByCustomer(Int32Value.newBuilder().setValue(customerId).build());
        } catch (final StatusRuntimeException e) {
            LOG.error("Error in communication", e);
            return null;
        }
    }
}

The last thing we need to do is to provide the address of the target service. Fortunately, the gRPC Spring Boot supports service discovery with Spring Cloud. We will use Eureka as a discovery server. Therefore both our sample apps need to include the Spring Cloud Eureka client.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

We also need to add the dependencyManagement section in the pom.xml containing the version of Spring Cloud we use.

<dependencyManagement>
  <dependencies>
    <dependency>
     <groupId>org.springframework.cloud</groupId>
       <artifactId>spring-cloud-dependencies</artifactId>
       <version>2021.0.8</version>
       <type>pom</type>
       <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

In order to avoid port conflicts with the account-service-grpc we will override the default gRPC and HTTP (Actuator) ports. We also need to provide several configuration settings for the @GrpcClient. First of all, we should have the same name as set inside the @GrpcClient annotation in the AccountClient class. The client communicates over the plaintext protocol and reads the address of the target service from the discovery server under the name set in the discovery:/// field.

server.port: 8081
grpc.server.port: 9091

grpc:
  client:
    account-service-grpc:
      address: 'discovery:///account-service-grpc'
      enableKeepAlive: true
      keepAliveWithoutCalls: true
      negotiationType: plaintext

Finally, we can run a discovery server and our two sample microservices. The Eureka server is available in our repository inside the discovery-server directory. Once you run you can go to the UI dashboard available under the http://localhost:8761 address.

Then run both our sample Spring Boot gRPC microservices. You can run all the apps using the following Maven command:

$ mvn spring-boot:run

Finally, let’s call the customer-service-grpc method that communicates with account-service-grpc. We use the grpcurl tool once again. As you see it returns a list of accounts inside the Customer object:

Final Thoughts

The gRPC Spring Boot Starter provides several useful features that simplify developer life. We can easily create services with @GrpcService, clients with @GrpcClient, or integrate gRPC with Spring Boot Actuator metrics and Spring Cloud discovery. However, there are also some drawbacks. The library is not very actively developed. There are about 2-3 releases per year, and there is still no support for Spring Boot 3. If you are looking for a more actively developed starter (but less popular) you can try the following one (I found out about it thanks to the discussion on Reddit after releasing the article).

The post Introduction to gRPC with Spring Boot appeared first on Piotr's TechBlog.

Skupper cluster interconnection works in Layer 7 (application layer). It means there is no need to create any VNPs or special firewall rules. Skupper is working according to the Virtual Application Network (VAN) approach. Thanks to that it can connect different Kubernetes clusters and guarantee communication between services without exposing them to the Internet. You can read more about the concept behind it in the Skupper docs.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. This time we will do almost everything using a command-line tool (skupper CLI). The repository contains just a sample app Spring Boot with Kubernetes Deployment manifests and Skaffold config. You will find here instructions on how to deploy the app with Skaffold, but you can as well use another tool. As always, follow my instructions for the details

Create Kubernetes clusters with Kind

In the first step, we will create three Kubernetes clusters with Kind. We need to give them different names: c1, c2 and c3. Accordingly, they are available under the context names: kind-c1, kind-c2 and kind-c3.

$ kind create cluster --name c1
$ kind create cluster --name c2
$ kind create cluster --name c3

By default, Skupper exposes itself as a Kubernetes LoadBalancer Service. Therefore, we need to enable the load balancer on Kind. In order to do that, we can install MetalLB. You can find the full installation instructions in the Kind docs here. Firstly, let’s switch to the c1 cluster:

$ kubectx kind-c1

Then, we have to apply the following YAML manifest:

$ kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.13.7/config/manifests/metallb-native.yaml

You should repeat the same procedure for the other two clusters: c2 and c3. However, it is not all. We also need to set up the address pool used by load balancers. To do that, let’s first check a range of IP addresses on the Docker network used by Kind. For me it is 172.19.0.0/16 172.19.0.1.

$ docker network inspect -f '{{.IPAM.Config}}' kind

According to the results, we need to choose the right IP address for all three Kind clusters. Then we have to create the IPAddressPool object, which contains the IPs range. Here’s the YAML manifest for the c1 cluster:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: example
  namespace: metallb-system
spec:
  addresses:
  - 172.19.255.200-172.19.255.250
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system

Here’s the pool configuration for e.g. the c2 cluster. It is important that the address range should not conflict with the ranges in two other Kind clusters.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: example
  namespace: metallb-system
spec:
  addresses:
  - 172.19.255.150-172.19.255.199
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system

Finally, the configuration for the c3 cluster:

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: example
  namespace: metallb-system
spec:
  addresses:
  - 172.19.255.100-172.19.255.149
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: empty
  namespace: metallb-system

After applying the YAML manifests with the kubectl apply -f command we can proceed to the next section.

Install Skupper on Kubernetes

We can install and manage Skupper on Kubernetes in two different ways: with CLI or through YAML manifests. Most of the examples in Skupper documentation use CLI for that, so I guess it is a preferable approach. Consequently, before we start with Kubernetes, we need to install CLI. You can find installation instructions in the Skupper docs here. Once you install it, just verify if it works with the following command:

$ skupper version

After that, we can proceed with Kubernetes clusters. We will create the same namespace interconnect inside all three clusters. To simplify our upcoming exercise we can also set a default namespace for each context (alternatively you can do it with the kubectl config set-context --current --namespace interconnect command).

$ kubectl create ns interconnect
$ kubens interconnect

Then, let’s switch to the kind-c1 cluster. We will stay in this context until the end of our exercise

$ kubectx kind-c1

Finally, we will install Skupper on our Kubernetes clusters. In order to do that, we have to execute the skupper init command. Fortunately, it allows us to set the target Kubernetes context with the -c parameter. Inside the kind-c1 cluster, we will also enable the Skupper UI dashboard (--enable-console parameter). With the Skupper console, we may e.g. visualize a traffic volume for all targets in the Skupper network.

$ skupper init --enable-console --enable-flow-collector
$ skupper init -c kind-c2
$ skupper init -c kind-c3

Let’s verify the status of the Skupper installation:

$ skupper status
$ skupper status -c kind-c2
$ skupper status -c kind-c3

Here’s the status for Skupper running in the kind-c1 cluster:

We can also display a list of running Skupper pods in the interconnect namespace:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
skupper-prometheus-867f57b89-dc4lq            1/1     Running   0          3m36s
skupper-router-55bbb99b87-k4qn5               2/2     Running   0          3m40s
skupper-service-controller-6bf57595dd-45hvw   2/2     Running   0          3m37s

Now, our goal is to connect both the c2 and c3 Kind clusters with the c1 cluster. In the Skupper nomenclature, we have to create a link between the namespace in the source and target cluster. Before we create a link we need to generate a secret token that signifies permission to create a link. The token also carries the link details. We are generating two tokens on the target cluster. Each token is stored as the YAML file. The first of them is for the kind-c2 cluster (skupper-c2-token.yaml), and the second for the kind-c3 cluster (skupper-c3-token.yaml).

$ skupper token create skupper-c2-token.yaml
$ skupper token create skupper-c3-token.yaml

We will consider several scenarios where we create a link using different parameters. Before that, let’s deploy our sample app on the kind-c2 and kind-c3 clusters.

Running the sample app on Kubernetes with Skaffold

After cloning the sample app repository go to the main directory. You can easily build and deploy the app to both kind-c2 and kind-c3 with the following commands:

$ skaffold dev --kube-context=kind-c2
$ skaffold dev --kube-context=kind-c3

After deploying the app skaffold automatically prints all the logs as shown below. It will be helpful for the next steps in our exercise.

Our app is deployed under the sample-spring-kotlin-microservice name.

Load balancing with Skupper – scenarios

Scenario 1: the same number of pods and link cost

Let’s start with the simplest scenario. We have a single pod of our app running on the kind-c2 and kind-c3 cluster. In Skupper we can also assign a cost to each link to influence the traffic flow. By default, link cost is set to 1 for a new link. In a service network, the routing algorithm attempts to use the path with the lowest total cost from the client to the target server. For now, we will leave a default value. Here’s a visualization of the first scenario:

Let’s create links to the c1 Kind cluster using the previously generated tokens.

$ skupper link create skupper-c2-token.yaml -c kind-c2
$ skupper link create skupper-c3-token.yaml -c kind-c3

If everything goes fine you should see a similar message:

We can also verify the status of links by executing the following commands:

$ skupper link status -c kind-c2
$ skupper link status -c kind-c3

It means that now c2 and c3 Kind clusters are “working” in the same Skupper network as the c1 cluster. The next step is to expose our app running in both the c2 and c3 clusters into the c1 cluster. Skupper works at Layer 7 and by default, it doesn’t connect apps unless we won’t enable that feature for the particular app. In order to expose our apps to the c1 cluster we need to run the following command on both c2 and c3 clusters.

$ skupper expose deployment/sample-spring-kotlin-microservice \
  --port 8080 -c kind-c2
$ skupper expose deployment/sample-spring-kotlin-microservice \
  --port 8080 -c kind-c3

Let’s take a look at what happened at the target (kind-c1) cluster. As you see Skupper created the sample-spring-kotlin-microservice Kubernetes Service that forwards traffic to the skupper-router pod. The Skupper Router is responsible for load-balancing requests across pods being a part of the Skupper network.

To simplify our exercise, we will enable port-forwarding for the Service visible above.

$ kubectl port-forward svc/sample-spring-kotlin-microservice 8080:8080

Thanks to that we don’t have to configure Kubernetes Ingress to call the service. Now, we can send some test requests over localhost, e.g. with siege.

$ siege -r 200 -c 5 http://localhost:8080/persons/1

We can easily verify that the traffic is coming to pods running on the kind-c2 and kind-c3 by looking at the logs. Alternatively, we can go to the Skupper console and see the traffic visualization:

Scenario 2: different number of pods and same link cost

In the next scenario, we won’t change anything in the Skupper network configuration. We will just run the second pod of the app in the kind-c3 cluster. So now, there is a single pod running in the kind-c2 cluster, and two pods running in the kind-c3 cluster. Here’s our architecture.

Once again, we can send some requests to the previously tested Kubernetes Service with the siege command:

$ siege -r 200 -c 5 http://localhost:8080/persons/2

Let’s take a look at traffic visualization in the Skupper dashboard. We can switch between all available pods. Here’s the diagram for the pod running in the kind-c2 cluster.

Here’s the same diagram for the pod running in the kind-c3 cluster. As you see it receives only ~50% (or even less depending on which pod we visualize) of traffic received by the pod in the kind-c2 cluster. That’s because Skupper there are two pods running in the kind-c3 cluster, while Skupper still balances requests across clusters equally.

Scenario 3: only one pod and different link costs

In the current scenario, there is a single pod of the app running on the c2 Kind cluster. At the same time, there are no pods on the c3 cluster (the Deployment exists but it has been scaled down to zero instances). Here’s the visualization of our scenario.

The important thing here is that the c3 cluster is preferred by Skupper since the link to it has a lower cost (2) than the link to the c2 cluster (4). So now, we need to remove the previous link, and then create a new one with the following commands:

$ skupper link create skupper-c2-token.yaml --cost 4 -c kind-c2
$ skupper link create skupper-c3-token.yaml --cost 2 -c kind-c3

Let’s take a look at the Skupper network status:

Let’s send some test requests to the exposed service. It works without any errors. Since there is only a single running pod, the whole traffic goes there:

Scenario 4 – more pods in one cluster and different link cost

Finally, the last scenario in our exercise. We will use the same Skupper configuration as in Scenario 3. However, this time we will run two pods in the kind-c3 cluster.

We can switch once again to the Skupper dashboard. Now, as you see, all the pods receive a very similar amount of traffic. Here’s the diagram for the pod running on the kind-c2 cluster.

Here’s a similar diagram for the pod running on the kind-c3 cluster. After setting the cost of the link assuming the number of pods running on the cluster I was able to split traffic equally between all the pods across both clusters. It works. However, it is not a perfect way for load-balancing. I would expect at least an option for enabling a round-robin between all the pods working in the same Skupper network. The solution presented in this scenario will work as expected unless we enable auto-scaling for the app.

Final Thoughts

Skupper introduces an interesting approach to the Kubernetes multicluster connectivity based fully on Layer 7. You can compare it to another solution based on the different layers like Submariner or Cilium cluster mesh. I described both of them in my previous articles. If you want to read more about Submariner visit the following post. If you are interested in Cilium read that article.

The post Kubernetes Multicluster Load Balancing with Skupper appeared first on Piotr's TechBlog.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.

Firstly, let’s discuss our repository. It contains five apps. There are three microservices (employee-service, department-service, organization-service) communicating with each other through the REST client and connecting to the Mongo database. There is also the API gateway (gateway-service) created with the Spring Cloud Gateway project. Finally, the admin-service directory contains the Spring Boot Admin app used for monitoring all other apps. You can easily deploy all the apps from the source code using a single Skaffold command. If you run the following command from the repository root directory it will build the images with Jib Maven Plugin and deploy all apps on your Kubernetes cluster:

$ skaffold run

On the other hand, you can go to the particular app directory and deploy only it using exactly the same command. All the required Kubernetes YAML manifests for each app are placed inside the k8s directories. There is also a global configuration with e.g. Mongo deployment in the project root k8s directory. Here’s the structure of our sample repo:

How It Works

In our sample architecture, we will use Spring Cloud Kubernetes Config for injecting configuration via ConfigMap and Secret and Spring Cloud Kubernetes Discovery for inter-service communication with the OpenFeign client. All our apps are running within the same namespace, but we could as well deploy them across several different namespaces and handle communication between them with OpenFeign. The only thing we should do in that case is to set the property spring.cloud.kubernetes.discovery.all-namespaces to true. For more details, you can refer to the following article.

In front of our services, there is an API gateway. It is a separate app, but we could as well install it on Kubernetes using the native CRD integration. For more details, you can refer to the following post on the Spring blog. In our case, this is a standard Spring Boot 3 app that just includes and uses the Spring Cloud Gateway module. It also uses Spring Cloud Kubernetes Discovery together with Spring Cloud OpenFeign to locate and call the downstream services. Here’s the diagram that illustrates our architecture.

Using Spring Cloud Kubernetes Config

I’ll describe implementation details by the example of department-service. It exposes some REST endpoints but also calls the endpoints exposed by the employee-service. Besides the standard modules, we need to include Spring Cloud Kubernetes in the Maven dependencies. Here, we have to decide if we use the Fabric8 client or the Kubernetes Java Client. Personally, I have an experience with Fabric8, so I’ll use the spring-cloud-starter-kubernetes-fabric8-all starter to include both config and discovery modules.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

As you see our app is connecting to the Mongo database. Let’s provide connection details and credentials required by the app. In the k8s directory, you will find the configmap.yaml file. It contains the address of Mongo and the database name. Those properties are injected into the pod as the application.properties file. And now the most important thing. The name of the ConfigMap has to be the same as the name of our app. The name of the Spring Boot is indicated by the spring.application.name property.

kind: ConfigMap
apiVersion: v1
metadata:
  name: department
data:
  application.properties: |-
    spring.data.mongodb.host: mongodb
    spring.data.mongodb.database: admin
    spring.data.mongodb.authentication-database: admin

In the current case, the name of the app is department. Here’s the application.yml file inside the app:

spring:
  application:
    name: department

The same naming rule applies to Secret. We are keeping sensitive data like the username and password to the Mongo database inside the following Secret. You can also find that content inside the secret.yaml file in the k8s directory.

kind: Secret
apiVersion: v1
metadata:
  name: department
data:
  spring.data.mongodb.password: UGlvdF8xMjM=
  spring.data.mongodb.username: cGlvdHI=
type: Opaque

Now, let’s proceed to the Deployment manifest. We will clarify the two first points here later. Spring Cloud Kubernetes requires special privileges on Kubernetes to interact with the master API (1). We don’t have to provide a tag for the image – Skaffold will handle it (2). In order to enable loading properties from ConfigMap we need to set the spring.config.import=kubernetes: property (a new way) or set the property spring.cloud.bootstrap.enabled to true (the old way). Instead of using properties directly, we will set the corresponding environment variables on the Deployment (3). By default, consuming secrets through the API is not enabled for security reasons. In order to enable it, we will set the SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI environment variable to true (4).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: department
  labels:
    app: department
spec:
  replicas: 1
  selector:
    matchLabels:
      app: department
  template:
    metadata:
      labels:
        app: department
    spec:
      serviceAccountName: spring-cloud-kubernetes # (1)
      containers:
      - name: department
        image: piomin/department # (2)
        ports:
        - containerPort: 8080
        env:
          - name: SPRING_CLOUD_BOOTSTRAP_ENABLED # (3)
            value: "true"
          - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI # (4)
            value: "true"

Using Spring Cloud Kubernetes Discovery

We have already included the Spring Cloud Kubernetes Discovery module in the previous section using the spring-cloud-starter-kubernetes-fabric8-all starter. In order to provide a declarative REST client we will also include the Spring Cloud OpenFeign module:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

Now, we can declare the @FeignClient interface. The important thing here is the name of a discovered service. It should be the same as the name of the Kubernetes Service defined for the employee-service app.

@FeignClient(name = "employee")
public interface EmployeeClient {

    @GetMapping("/department/{departmentId}")
    List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);

    @GetMapping("/department-with-delay/{departmentId}")
    List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId);
}

Here’s the Kubernetes Service manifest for the employee-service app. The name of the service is employee (1). The label spring-boot is set for Spring Boot Admin discovery purposes (2). You can find the following YAML in the employee-service/k8s directory.

apiVersion: v1
kind: Service
metadata:
  name: employee # (1)
  labels:
    app: employee
    spring-boot: "true" # (2)
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: employee
  type: ClusterIP

Just to clarify – here’s the implementation of the employee-service API methods called by the OpenFeign client in the department-service.

@RestController
public class EmployeeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(EmployeeController.class);
	
    @Autowired
    EmployeeRepository repository;

    // ... other endpoints implementation 

    @GetMapping("/department/{departmentId}")
    public List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId) {
        LOGGER.info("Employee find: departmentId={}", departmentId);
        return repository.findByDepartmentId(departmentId);
    }

    @GetMapping("/department-with-delay/{departmentId}")
    public List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId) throws InterruptedException {
        LOGGER.info("Employee find: departmentId={}", departmentId);
        Thread.sleep(2000);
        return repository.findByDepartmentId(departmentId);
    }
	
}

That’s all that we have to do. Now, we can just call the endpoint using the OpenFeign client from department-service. For example on the “delayed” endpoint, we can use Spring Cloud Circuit Breaker with Resilience4J.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;
    Resilience4JCircuitBreakerFactory circuitBreakerFactory;

    public DepartmentController(
        DepartmentRepository repository, 
        EmployeeClient employeeClient,
        Resilience4JCircuitBreakerFactory circuitBreakerFactory) {
            this.repository = repository;
            this.employeeClient = employeeClient;
            this.circuitBreakerFactory = circuitBreakerFactory;
    }

    @GetMapping("/{id}/with-employees-and-delay")
    public Department findByIdWithEmployeesAndDelay(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Department department = repository.findById(id).orElseThrow();
        CircuitBreaker circuitBreaker = circuitBreakerFactory.create("delayed-circuit");
        List<Employee> employees = circuitBreaker.run(() ->
                employeeClient.findByDepartmentWithDelay(department.getId()));
        department.setEmployees(employees);
        return department;
    }

    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") String organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Testing with Fabric8 Kubernetes

We have already finished the implementation of our service. All the Kubernetes YAML manifests are prepared and ready to deploy. Now, the question is – can we easily test that everything works fine before we proceed to the deployment on the real cluster? The answer is – yes. Moreover, we can choose between several tools. Let’s begin with the simplest option – Kubernetes mock server. In order to use it, we to include an additional Maven dependency:

<dependency>
  <groupId>io.fabric8</groupId>
  <artifactId>kubernetes-server-mock</artifactId>
  <version>6.7.1</version>
  <scope>test</scope>
</dependency>

Then, we can proceed to the test. In the first step, we need to provide several test annotations. Inside @SpringBootTest we should simulate the Kubernetes platform with spring.main.cloud-platform property set to KUBERNETES (1). Normally Spring Boot is able to autodetect if it is running on Kubernetes. In that case, we need “trick him”, because we are just simulating the API, not running the test on Kubernetes. We also need to enable the old way of ConfigMap injection with the spring.cloud.bootstrap.enabled=true property.

Once we annotate the test method with @EnableKubernetesMockClient (2) we can use an auto-configured static instance of the Fabric8 KubernetesClient (3). During the test Fabric8 library runs a web server that mocks all the API requests sent by the client. By the way, we are using Testcontainers for running Mongo (4). In the next step, we are creating the ConfigMap that injects Mongo connection settings into the Spring Boot app (5). Thanks to the Spring Cloud Kubernetes Config it is automatically loaded by the app and the app is able to connect the Mongo database on the dynamically generated port.

Spring Cloud Kubernetes comes with auto-configured Fabric8 KubernetesClient. We need to force it to connect to the mock API server. Therefore we should override kubernetes.master property used by the Fabric8 KubernetesClient into the master URL taken from the test “mocked” instance (6). Finally, we can just implement test methods in the standard way.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {
                "spring.main.cloud-platform=KUBERNETES",
                "spring.cloud.bootstrap.enabled=true"}) // (1)
@EnableKubernetesMockClient(crud = true) // (2)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesMockTest {

    private static final Logger LOG = LoggerFactory
        .getLogger(EmployeeKubernetesMockTest.class);

    static KubernetesClient client; // (3)

    @Container // (4)
    static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");

    @BeforeAll
    static void setup() {

        ConfigMap cm = client.configMaps()
                .create(buildConfigMap(mongodb.getMappedPort(27017)));
        LOG.info("!!! {}", cm); // (5)

        // (6)
        System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY, 
            client.getConfiguration().getMasterUrl());
        System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, "true");
        System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, "default");
    }

    private static ConfigMap buildConfigMap(int port) {
        return new ConfigMapBuilder().withNewMetadata()
                .withName("employee").withNamespace("default")
                .endMetadata()
                .addToData("application.properties",
                        """
                        spring.data.mongodb.host=localhost
                        spring.data.mongodb.port=%d
                        spring.data.mongodb.database=test
                        spring.data.mongodb.authentication-database=test
                        """.formatted(port))
                .build();
    }

    @Autowired
    TestRestTemplate restTemplate;

    @Test
    @Order(1)
    void addEmployeeTest() {
        Employee employee = new Employee("1", "1", "Test", 30, "test");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(2)
    void addAndThenFindEmployeeByIdTest() {
        Employee employee = new Employee("1", "2", "Test2", 20, "test2");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
        employee = restTemplate
                .getForObject("/{id}", Employee.class, employee.getId());
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(3)
    void findAllEmployeesTest() {
        Employee[] employees =
                restTemplate.getForObject("/", Employee[].class);
        assertEquals(2, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByDepartmentTest() {
        Employee[] employees =
                restTemplate.getForObject("/department/1", Employee[].class);
        assertEquals(1, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByOrganizationTest() {
        Employee[] employees =
                restTemplate.getForObject("/organization/1", Employee[].class);
        assertEquals(2, employees.length);
    }

}

Now, after running the tests we can take a look at the logs. As you see, our test is loading properties from the employee ConfigMap.

Finally, it is able to successfully connect Mongo on the dynamic port and run all the tests against that instance.

Testing with Testcontainers on k3s

As I mentioned before, there are several tools we can use for testing with Kubernetes. This time we will see how to do it with Testcomntainers. We have already used it in the previous section for running the Mongo database. But there is also the Testcontainers module for Rancher’s k3s Kubernetes distribution. Currently, it is in the incubating state, but it doesn’t bother us to try it. In order to use it in the project we need to include the following Maven dependency:

<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>k3s</artifactId>
  <scope>test</scope>
</dependency>

Here’s the implementation of the same tests as in the previous section, but this time with the k3s container. We don’t have to create any mocks. Instead, we will create the K3sContainer object (1). Before running the tests we need to create and initialize KubernetesClient. Testcontainers K3sContainer provides the getKubeConfigYaml() method for getting kubeconfig data. With the Fabric8 Config object we can initialize the client from that kubeconfig (2) (3). After that, we will create the ConfigMap with Mongo connection details (4). Finally, we have to override the master URL for Spring Cloud Kubernetes auto-configured Fabric8 client. In comparison to the previous section, we also need to set Kubernetes client certificates and keys (5).

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {
                "spring.main.cloud-platform=KUBERNETES",
                "spring.cloud.bootstrap.enabled=true"})
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesTest {

   private static final Logger LOG = LoggerFactory
      .getLogger(EmployeeKubernetesTest.class);

   @Container
   static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");
   @Container
   static K3sContainer k3s = new K3sContainer(DockerImageName
      .parse("rancher/k3s:v1.21.3-k3s1")); // (1)

   @BeforeAll
   static void setup() {
      Config config = Config
         .fromKubeconfig(k3s.getKubeConfigYaml()); // (2)
      DefaultKubernetesClient client = new 
         DefaultKubernetesClient(config); // (3)

      ConfigMap cm = client.configMaps().inNamespace("default")
         .create(buildConfigMap(mongodb.getMappedPort(27017)));
      LOG.info("!!! {}", cm); // (4)

      System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY, 
         client.getConfiguration().getMasterUrl());
      
      // (5) 
      System.setProperty(Config.KUBERNETES_CLIENT_CERTIFICATE_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getClientCertData());
      System.setProperty(Config.KUBERNETES_CA_CERTIFICATE_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getCaCertData());
       System.setProperty(Config.KUBERNETES_CLIENT_KEY_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getClientKeyData());
      System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, 
         "true");
      System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, 
         "default");
    }

    private static ConfigMap buildConfigMap(int port) {
        return new ConfigMapBuilder().withNewMetadata()
                .withName("employee").withNamespace("default")
                .endMetadata()
                .addToData("application.properties",
                        """
                        spring.data.mongodb.host=localhost
                        spring.data.mongodb.port=%d
                        spring.data.mongodb.database=test
                        spring.data.mongodb.authentication-database=test
                        """.formatted(port))
                .build();
    }

    @Autowired
    TestRestTemplate restTemplate;

    @Test
    @Order(1)
    void addEmployeeTest() {
        Employee employee = new Employee("1", "1", "Test", 30, "test");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(2)
    void addAndThenFindEmployeeByIdTest() {
        Employee employee = new Employee("1", "2", "Test2", 20, "test2");
        employee = restTemplate
           .postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
        employee = restTemplate
                .getForObject("/{id}", Employee.class, employee.getId());
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(3)
    void findAllEmployeesTest() {
        Employee[] employees =
                restTemplate.getForObject("/", Employee[].class);
        assertEquals(2, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByDepartmentTest() {
        Employee[] employees =
                restTemplate.getForObject("/department/1", Employee[].class);
        assertEquals(1, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByOrganizationTest() {
        Employee[] employees =
                restTemplate.getForObject("/organization/1", Employee[].class);
        assertEquals(2, employees.length);
    }

}

Run Spring Kubernetes Apps on Minikube

In this exercise, I’m using Minikube, but you can as well use any other distribution like Kind or k3s. Spring Cloud Kubernetes requires additional privileges on Kubernetes to be able to interact with the master API. So, before running the apps we will create the spring-cloud-kubernetes ServiceAccount with the required privileges. Our role needs to have access to the configmaps, pods, services, endpoints and secrets. If we do not enable discovery across all namespaces (the spring.cloud.kubernetes.discovery.all-namespaces property), it can be Role within the namespace. Otherwise, we should create a ClusterRole.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spring-cloud-kubernetes
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spring-cloud-kubernetes
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spring-cloud-kubernetes
  namespace: default
subjects:
  - kind: ServiceAccount
    name: spring-cloud-kubernetes
    namespace: default
roleRef:
  kind: ClusterRole
  name: spring-cloud-kubernetes

Of course, you don’t have to apply the manifests visible above by yourself. As I mentioned at the beginning of the article, there is a skaffold.yaml file in the repository root directory file that contains the whole configuration. It runs manifests with Mongo Deployment (1) and with privileges (2) together with all the services.

apiVersion: skaffold/v4beta5
kind: Config
metadata:
  name: sample-spring-microservices-kubernetes
build:
  artifacts:
    - image: piomin/admin
      jib:
        project: admin-service
    - image: piomin/department
      jib:
        project: department-service
        args:
          - -DskipTests
    - image: piomin/employee
      jib:
        project: employee-service
        args:
          - -DskipTests
    - image: piomin/gateway
      jib:
        project: gateway-service
    - image: piomin/organization
      jib:
        project: organization-service
        args:
          - -DskipTests
  tagPolicy:
    gitCommit: {}
manifests:
  rawYaml:
    - k8s/mongodb-*.yaml # (1)
    - k8s/privileges.yaml # (2)
    - admin-service/k8s/*.yaml
    - department-service/k8s/*.yaml
    - employee-service/k8s/*.yaml
    - gateway-service/k8s/*.yaml
    - organization-service/k8s/*.yaml

All we need to do it to deploy all the apps by executing the following skaffold command:

$ skaffold dev

Once we will do it we can display a list of running s pods:

kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
admin-5f8c8498f-vtstx           1/1     Running   0          2m38s
department-746774879b-llrdn     1/1     Running   0          2m38s
employee-5bbf6b765f-7hsv7       1/1     Running   0          2m37s
gateway-578cb64558-m9n7f        1/1     Running   0          2m37s
mongodb-7f68b8b674-dbfnb        1/1     Running   0          2m38s
organization-5688c58656-bv8n6   1/1     Running   0          2m37s

We can also display a list of services. Some of them, like admin or gateway, are exposed as NodePort. Thanks to that we can easily access them outside of our Kubernetes cluster.

kubectl get svc
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
admin          NodePort    10.101.220.141   <none>        8080:31368/TCP   3m53s
department     ClusterIP   10.108.144.90    <none>        8080/TCP         3m52s
employee       ClusterIP   10.99.75.2       <none>        8080/TCP         3m52s
gateway        NodePort    10.96.7.237      <none>        8080:31518/TCP   3m52s
kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP          38h
mongodb        ClusterIP   10.108.198.233   <none>        27017/TCP        3m53s
organization   ClusterIP   10.107.102.26    <none>        8080/TCP         3m52s

Let’s obtain the Minikube IP address on our local machine:

$ minikube ip

Now, we can use that IP address to access e.g. Spring Boot Admin Server on the target port. For me its 31368. Spring Boot Admin should successfully discover all three microservices and connect to the /actuator endpoints exposed by that apps.

We can go to the details of each Spring Boot app. As you depatment-service is running on my local Minikube.

Once you stop the skaffold dev command, all the apps and configured will be removed from your Kubernetes cluster.

Final Thoughts

If you are running only Spring Boot apps on your Kubernetes cluster, Spring Cloud Kubernetes is an interesting option. It allows us to easily integrate with Kubernetes discovery, config maps, and secrets. Thanks to that we can take advantage of other Spring Cloud components like load balancer, circuit breaker, etc. However, if you are running apps written in different languages and frameworks, and using language-agnostic tools like service mesh (Istio, Linkerd), Spring Cloud Kubernetes may not be the best choice.

The post Spring Cloud Kubernetes with Spring Boot 3 appeared first on Piotr's TechBlog.

If you want to compare Microcks with other testing tools, you may be interested in my article about contract testing with Quarkus and Pact available here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

Install Microcks on Kubernetes

We can install Microcks on Kubernetes with Helm chart or with the operator. Since I’m using OpenShift as a Kubernetes platform in this exercise, the simplest way is through the operator. Assuming we have already installed the OLM (Operator Lifecycle Manager) on Kubernetes we need to apply the following YAML manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: microcks
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: operatorgroup
  namespace: microcks
spec:
  targetNamespaces:
  - microcks
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-microcks
  namespace: microcks
spec:
  channel: stable
  name: microcks
  source: operatorhubio-catalog
  sourceNamespace: olm

With OpenShift we can install the Microcks operator using the UI dashboard. Once we do it, we need to create the object responsible for Microcks installation. In the OpenShift console, we need to click the “MicrocksInstall” link as shown below.

Microcks requires some additional staff like Keycloak or MongoDB to be installed on the cluster. Here’s the YAML manifest responsible for installation on Kubernetes.

apiVersion: microcks.github.io/v1alpha1
kind: MicrocksInstall
metadata:
  name: my-microcksinstall
  namespace: microcks
spec:
  name: my-microcksinstall
  version: 1.7.0
  microcks:
    replicas: 1
  postman:
    replicas: 1
  keycloak:
    install: true
    persistent: true
    volumeSize: 1Gi
  mongodb:
    install: true
    persistent: true
    volumeSize: 2Gi

By applying the YAML manifest with the MicrocksInstall object, we will start the installation process. We should have the following list of running pods if it finished successfully:

$ kubectl get pod -n microcks
NAME                                                     READY   STATUS    RESTARTS   AGE
microcks-ansible-operator-56ddbdcccf-6lrdl               1/1     Running   0          4h14m
my-microcksinstall-76b67f4f77-jdcj7                      1/1     Running   0          4h13m
my-microcksinstall-keycloak-79c5f68f45-5nm69             1/1     Running   0          4h13m
my-microcksinstall-keycloak-postgresql-97f69c476-km6p2   1/1     Running   0          4h13m
my-microcksinstall-mongodb-846f7c7976-tl5jh              1/1     Running   0          4h13m
my-microcksinstall-postman-runtime-574d4bf7dc-xfct7      1/1     Running   0          4h13m

In order to prepare the contract testing process on Kubernetes with Microcks we need to access it dashboard. It is exposed by the my-microcksinstall service under the 8080 port. I’m accessing it through the OpenShift Route object. If you use Kubernetes you can enable port forwarding (the kubectl port-forward command) for that service or expose it through the Ingress object.

Here’s the Microcks UI dashboard. In the first step, we need to import our API documentation and samples by clicking the Importers button. Before we do it, we need to prepare the document in one of the supported specifications. In our case, it is the OpenAPI specification.

Create the Provider Side App

We create a simple app with Quarkus that exposes some REST endpoints. In order to access it go to the employee-service directory in our sample Git repository. Here’s the controller class responsible for the endpoints implementation.

@Path("/employees")
@Produces(MediaType.APPLICATION_JSON)
public class EmployeeController {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(EmployeeController.class);

   @Inject
   EmployeeRepository repository;

   @POST
   public Employee add(@Valid Employee employee) {
      LOGGER.info("Employee add: {}", employee);
      return repository.add(employee);
   }

   @Path("/{id}")
   @GET
   public Employee findById(@PathParam("id") Long id) {
      LOGGER.info("Employee find: id={}", id);
      return repository.findById(id);
   }

   @GET
   public Set<Employee> findAll() {
      LOGGER.info("Employee find");
      return repository.findAll();
   }

   @Path("/department/{departmentId}")
   @GET
   public Set<Employee> findByDepartment(@PathParam("departmentId") Long departmentId) {
      LOGGER.info("Employee find: departmentId={}", departmentId);
      return repository.findByDepartment(departmentId);
   }

   @Path("/organization/{organizationId}")
   @GET
   public Set<Employee> findByOrganization(@PathParam("organizationId") Long organizationId) {
      LOGGER.info("Employee find: organizationId={}", organizationId);
      return repository.findByOrganization(organizationId);
   }

}

If we add the module responsible for generating and exposing OpenAPI documentation we can easily access it under the /q/openapi path. In order to achieve to include the following Maven dependency:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-smallrye-openapi</artifactId>
</dependency>

Here’s the OpenAPI descriptor automatically generated by Quarkus for our employee-service app.

openapi: 3.0.3
info:
  title: employee-service API
  version: "1.2"
paths:
  /employees:
    get:
      tags:
      - Employee Controller
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
    post:
      tags:
      - Employee Controller
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Employee'
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Employee'
  /employees/department/{departmentId}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: departmentId
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
  /employees/organization/{organizationId}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: organizationId
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
  /employees/{id}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: id
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Employee'
components:
  schemas:
    Employee:
      required:
      - organizationId
      - departmentId
      - name
      - position
      type: object
      properties:
        id:
          format: int64
          type: integer
        organizationId:
          format: int64
          type: integer
        departmentId:
          format: int64
          type: integer
        name:
          pattern: \S
          type: string
        age:
          format: int32
          maximum: 100
          minimum: 1
          type: integer
        position:
          pattern: \S
          type: string

We need to add the examples section to each responses and parameters section. Let’s begin with the GET /employees endpoint. It returns all existing employees. The name of the example is not important. You can set any name you want. As the returned value we set the JSON array with three Employee objects.

/employees:
  get:
    tags:
      - Employee Controller
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              uniqueItems: true
              type: array
              items:
                $ref: '#/components/schemas/Employee'
            examples:
              all_persons:
                value: |-
                  [
                     {"id": 1, "name": "Test User 1", "age": 20, "organizationId": 1, "departmentId": 1, "position": "developer"},
                     {"id": 2, "name": "Test User 2", "age": 30, "organizationId": 1, "departmentId": 2, "position": "architect"},
                     {"id": 3, "name": "Test User 3", "age": 40, "organizationId": 2, "departmentId": 3, "position": "developer"},
                  ]

For comparison, let’s take a look at the OpenAPI docs for the POST /employees endpoint. It returns a single JSON object as a response. We also had to add examples in the requestBody section.

/employees:
  post:
    tags:
      - Employee Controller
    requestBody:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Employee'
          examples:
            add_employee:
              summary: Hire a new employee
              description: Should return 200
              value: '{"name": "Test User 4", "age": 50, "organizationId": 2, "departmentId": 3, "position": "tester"}'
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Employee'
            examples:
              add_employee:
                value: |-
                  {"id": 4, "name": "Test User 4", "age": 50, "organizationId": 2, "departmentId": 3, "position": "tester"}

Finally, we can proceed to the endpoint GET /employees/department/{departmentId}, which returns all the employees assigned to the particular department. This endpoint is called by another app – department-service. We need to add the example value of the path variable referring to the id of a department. The same as before, we also have to include an example JSON in the responses section.

/employees/department/{departmentId}:
  get:
    tags:
      - Employee Controller
    parameters:
      - name: departmentId
        in: path
        required: true
        schema:
          format: int64
          type: integer
        examples:
          find_by_dep_1:
            summary: Main id of department
            value: 1
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              uniqueItems: true
              type: array
              items:
                $ref: '#/components/schemas/Employee'
            examples:
              find_by_dep_1:
                value: |-
                  [
                    { "id": 1, "name": "Test User 1", "age": 20, "organizationId": 1, "departmentId": 1, "position": "developer" }
                  ]

Create the Consumer Side App

As I mentioned before, we have another app – department-service. It consumer endpoint exposed by the employee-service. Here’s the implementation of the Quarkus declarative REST client responsible for calling the GET /employees/department/{departmentId} endpoint:

@ApplicationScoped
@Path("/employees")
@RegisterRestClient(configKey = "employee")
public interface EmployeeClient {

    @GET
    @Path("/department/{departmentId}")
    @Produces(MediaType.APPLICATION_JSON)
    List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId);

}

That client is then used by the department-service REST controller to return all the employees in the particular organization with the division into departments.

@Path("/departments")
@Produces(MediaType.APPLICATION_JSON)
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory
       .getLogger(DepartmentController.class);

    @Inject
    DepartmentRepository repository;
    @Inject
    @RestClient
    EmployeeClient employeeClient;

    // ... implementation of other endpoints

    @Path("/organization/{organizationId}/with-employees")
    @GET
    public Set<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        Set<Department> departments = repository.findByOrganization(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Finally, we need to configure the base URI for the client in the Quarkus application.properties file. We can also set that address in the corresponding environment variable QUARKUS_REST_CLIENT_EMPLOYEE_URL.

quarkus.rest-client.employee.url = http://employee:8080

Test API with Microcks

Once we implemented our apps, we can back to the Microcks dashboard. In the Importers section import your openapi.yml file.

Once you will import the OpenAPI definition you can go to the APIs | Services section. You should already have the record with your API description as shown below. The current version of our is 1.2. The employee-service exposes five REST endpoints.

We can display the details of the service by clicking on the record or on the Details button. It displays a list of available endpoints with a number of examples declared in the OpenAPI manifest.

Let’s display the details of the GET /employees/department/{departmentId} endpoint. This endpoint is called by the department-service. Microcks creates the mock on Kubernetes for the purposes of contract testing. The mock is available inside Kubernetes and as well as outside it through the OpenShift Route. Now, our goal is to verify the compliance of our mock with the real service.

In order to verify the Microcks mock with the real service, we need to deploy our employee-service app on Kubernetes. We can easily do it using the Quarkus Kubernetes extension. Thanks to that it is possible to build the app, build the image and deploy it on Kubernetes or OpenShift using a single Maven command. Here’s the dedicated Maven profile for that. For OpenShift, we need to include the single dependency quarkus-openshift, and set the property quarkus.kubernetes.deploy to true.

<profile>
  <id>openshift</id>
  <activation>
    <property>
      <name>openshift</name>
    </property>
  </activation>
  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-openshift</artifactId>
    </dependency>
  </dependencies>
  <properties>
    <quarkus.kubernetes.deploy>true</quarkus.kubernetes.deploy>
    <quarkus.profile>openshift</quarkus.profile>
    <quarkus.container-image.builder>openshift</quarkus.container-image.builder>
  </properties>
</profile>

Then, we just need to activate the openshift profile during the Maven build. Go to the employee-service directory and execute the following command:

$ mvn clean package -Popenshift

The internal address of our app is defined by the name of the Kubernetes service:

$ oc get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
employee-service   ClusterIP   172.30.141.85   <none>        80/TCP    27h

Finally, we can our samples against real endpoints. We need to set the address of our service in the Test Endpoint field. Then, we choose OPEN_API_SCHEMA as the Runner. Finally, we can execute the test by clicking the Launch test button.

Viewing and Analysing Test Results

Once we run the test we can display the results with the Microcks dashboard. By default, it calls all the endpoints defined in the OpenAPI manifest. We can override this behavior and choose a list of endpoints that should be tested.

We can verify the details of each test. For example, we can display a response from our employee-service running on the cluster for the GET /employees/department/{departmentId} endpoint.

We can automate the testing process with Microcks using its CLI. You can read more details about it in the docs. In order to execute the test in a similar way as we did before via the Microcks dashboard, we need to use the test command. It requires several input arguments. We need to pass the name and version of our API, the URI of the test endpoint, and the type of runner. By default, Microcks creates a client in Keycloak. The client id is microcks-serviceaccount. You need to log in to your Keycloak instance and copy the value of the client secret. As the microcksURL argument, we need to pass the external address of Microcks with the /api path. If you would run it inside the Kubernetes cluster, you could run the internal address http://my-microcksinstall.microcks.svc.cluster.local:8080/api.

$ microcks-cli test 'employee-service API:1.2' \
  http://employee-service.demo-microcks.svc.cluster.local \
  OPEN_API_SCHEMA \
  --microcksURL=https://my-microcksinstall-microcks.apps.pvxvtsz4.eastus.aroapp.io/api \
  --keycloakClientId microcks-serviceaccount \
  --keycloakClientSecret ab54d329-e435-41ae-a900-ec6b3fe15c54

Once we run a test, Microcks calculates the conformance score after finishing. It is a kind of grade that estimates how your API contract is actually covered by the samples you’ve attached to it. It is computed based on the number of samples you’ve got on each operation, and the complexity of dispatching rules of this operation. As you there is still a place for improvement in my case

Verify Contract on the Consumer Side

Let’s create a simple test that verifies the contract with employee-service in the department-service. It will call the department-service GET /departments/organization/{organizationId}/with-employees endpoint that interacts with the employee-service. We won’t mock the REST client.

@QuarkusTest
public class DepartmentExternalContractTests {

    @Test
    void findByOrganizationWithEmployees() {
        when().get("/departments/organization/{organizationId}/with-employees", 1L).then()
                .statusCode(200)
                .body("size()", notNullValue());
    }

}

Instead of mocking the client directly in the test, we use the mock endpoint exposed by Microcks. We will use the internal of this mock for the employee-service is https://my-microcksinstall.microcks.svc.cluster.local/rest/employee-service+API/1.2/employees. We can leverage the Quarkus environment variable to set the address for the client. As I mentioned before we can use the QUARKUS_REST_CLIENT_EMPLOYEE_URL environment variable.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: maven-contract-tests
spec:
  params:
    - default: >-
        image-registry.openshift-image-registry.svc:5000/openshift/java:latest
      description: Maven base image
      name: MAVEN_IMAGE
      type: string
    - default: .
      description: >-
        The context directory within the repository for sources on which we want
        to execute maven goals.
      name: CONTEXT_DIR
      type: string
    - name: EMPLOYEE_URL
      default: http://my-microcksinstall.microcks.svc.cluster.local:8080/rest/employee-service+API/1.2
  steps:
    - image: $(params.MAVEN_IMAGE)
      name: mvn-command
      env:
        - name: QUARKUS_REST_CLIENT_EMPLOYEE_URL
          value: $(params.EMPLOYEE_URL)
      resources: {}
      script: >
        #!/usr/bin/env bash

        /usr/bin/mvn clean package -Pmicrocks

      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
  workspaces:
    - name: source

As you see in the code above, we are activating the microcks profile during the Maven build. That’s because we want to run the test defined in DepartmentExternalContractTests class only if the build is performed on Kubernetes.

<profile>
  <id>microcks</id>
  <activation>
    <property>
      <name>microcks</name>
    </property>
  </activation>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${surefire-plugin.version}</version>
        <configuration>
          <includes>
            <include>**/DepartmentExternalContractTests.java</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>

Finally, we can define and start a pipeline that clones our Git repository and runs the test against the endpoint mocked by the Microcks. Here’s the definition of our pipeline.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: microcks-pipeline
spec:
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-quarkus-microservices.git'
        - name: revision
          value: master
        - name: submodules
          value: 'true'
        - name: depth
          value: '1'
        - name: sslVerify
          value: 'false'
        - name: crtFileName
          value: ca-bundle.crt
        - name: deleteExisting
          value: 'true'
        - name: verbose
          value: 'true'
        - name: gitInitImage
          value: >-
            registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8@sha256:a538c423e7a11aae6ae582a411fdb090936458075f99af4ce5add038bb6983e8
        - name: userHome
          value: /tekton/home
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: maven-contract-tests
      params:
        - name: MAVEN_IMAGE
          value: >-
            image-registry.openshift-image-registry.svc:5000/openshift/java:latest
        - name: CONTEXT_DIR
          value: department-service
        - name: EMPLOYEE_URL
          value: >-
            http://my-microcksinstall.microcks.svc.cluster.local:8080/rest/employee-service+API/1.2
      runAfter:
        - git-clone
      taskRef:
        kind: Task
        name: maven-contract-tests
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir

Final Thoughts

In this article, I described just one of several possible scenarios for contract testing on Kubernetes with Microcks. You can as well use it to test interactions, for example with Kafka or with GraphQL endpoints. Microcks can be a central point for testing contracts of apps running on Kubernetes. It is able to verify contracts by calling real endpoints exposed by the apps. It provides UI for visualizing and running tests, as well as CLI for automation.

The post Contract Testing on Kubernetes with Microcks appeared first on Piotr's TechBlog.

Before you start, it is worth familiarizing yourself with the Quarkus framework. There are several articles about Quarkus on my blog. If you want about to read about interesting and useful Quarkus features please refer to the post “Quarkus Tips, Tricks and Techniques” available here. For some more advanced features like testing strategies, you can read the “Advanced Testing with Quarkus” article available here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions. Let’s begin.

Architecture

In the exercise today we will add several contract tests to the existing architecture of sample Quarkus microservices. There are three sample apps that communicate with each other through HTTP. We use Quarkus declarative REST client to call remote HTTP endpoints. The department-service app calls the endpoint exposed by the employee-service app to get a list of employees assigned to the particular department. On the other hand, the organization-service app calls endpoints exposed by department-service and employee-service.

We will implement some contract tests to verify described interactions. Each contract is signed between two sides of communication: the consumer and the provider. Pact assumes that contract code is generated and published by the consumer side, and then verified by the provider side. It provides a tool for storing and sharing contracts between consumers and providers – Pact Broker. Pact Broker exposes a simple RESTful API for publishing and retrieving contracts, and an embedded web dashboard for navigating the API. We will run it as a Docker container. However, our goal is also to run it during the CI build and then use it to exchange contracts between the tests.

Here’s the diagram that illustrates the described architecture.

Running Pact Broker

Before we create any test, we will start Pact broker on the local machine. In order to do that, we need to run two containers on Docker. Pact broker requires database, so in the first step we will start the postgres container:

$ docker run -d --name postgres \
  -p 5432:5432 \
  -e POSTGRES_USER=pact \ 
  -e POSTGRES_PASSWORD=pact123 \
  -e POSTGRES_DB=pact \
  postgres

After that, we can run the container with Pact broker. We will link it to the postgres container and set the autentication credentials:

$ docker run -d --name pact-broker \
  --link postgres:postgres \
  -e PACT_BROKER_DATABASE_USERNAME=pact \
  -e PACT_BROKER_DATABASE_PASSWORD=pact123 \ 
  -e PACT_BROKER_DATABASE_HOST=postgres \
  -e PACT_BROKER_DATABASE_NAME=pact \
  -p 9292:9292 \
  pactfoundation/pact-broker

If you prefer to run everything with the single command you can use docker-compose.yml in the repository root directory. It will run not only Postgres and Pact broker, but also our three sample microservices.

version: "3.7"
services:
  postgres:
    container_name: postgres
    image: postgres
    environment:
      POSTGRES_USER: pact
      POSTGRES_PASSWORD: pact123
      POSTGRES_DB: pact
    ports:
      - "5432"
  pact-broker:
    container_name: pact-broker
    image: pactfoundation/pact-broker
    ports:
      - "9292:9292"
    depends_on:
      - postgres
    links:
      - postgres
    environment:
      PACT_BROKER_DATABASE_USERNAME: pact
      PACT_BROKER_DATABASE_PASSWORD: pact123
      PACT_BROKER_DATABASE_HOST: postgres
      PACT_BROKER_DATABASE_NAME: pact
  employee:
    image: quarkus/employee-service:1.2
    ports:
      - "8080"
  department:
    image: quarkus/department-service:1.1
    ports:
      - "8080"
    links:
      - employee
  organization:
    image: quarkus/organization-service:1.1
    ports:
      - "8080"
    links:
      - employee
      - department

Since the docker-compose.yml includes images with our sample microservices, you first need to build the Docker images of the apps. We can easily do it with Quarkus. Once we included the quarkus-container-image-jib dependency, we may build the image using Jib Maven plugin by activating the quarkus.container-image.build property as shown below. Additionally, don’t forget about skipping the tests.

$ mvn clean package -DskipTests -Dquarkus.container-image.build=true

Then just run the following command:

$ docker compose up

Finally, you can access the Pact broker UI under the http://localhost:9292 address. Of course, there are no contracts saved there, so you just see the example pact.

Create Contract Test for Consumer

Once we started a Pact broker we can proceed to the implementation of the tests. We will start from the consumer side. Both departament-service and organization-service consuming endpoints exposed by the employee-service. In the first step, we will include the Quarkus Pact Consumer extension to the Maven dependencies.

<dependency>
  <groupId>io.quarkiverse.pact</groupId>
  <artifactId>quarkus-pact-consumer</artifactId>
  <version>1.0.0.Final</version>
  <scope>provided</scope>
</dependency>

Here’s the REST client interface responsible for calling the employee-service GET /employees/department/{id} endpoint from the departament-service.

@ApplicationScoped
@Path("/employees")
@RegisterRestClient(configKey = "employee")
public interface EmployeeClient {

    @GET
    @Path("/department/{departmentId}")
    @Produces(MediaType.APPLICATION_JSON)
    List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId);

}

We will test the EmployeeClient directly in the contract test. In order to implement a contract test on the consumer side we need to declare the PactConsumerTestExt JUnit5 extension. In the callFindByDepartment method, we need to prepare the expected response template as the RequestResponsePact object. The method should return an array of employees. Therefore we are using the PactDslJsonArray to construct the required object. The name of the provider is employee-service, while the name of the consumer is department-service. In order to use Pact MockServer I had to declare v3 of Pact instead of the latest v4. Then we are setting the mock server address as the RestClientBuilder base URI and test the contract.

@QuarkusTest
@ExtendWith(PactConsumerTestExt.class)
public class EmployeeClientContractTests {

    @Pact(provider = "employee-service", 
          consumer = "department-service")
    public RequestResponsePact callFindByDepartment(
        PactDslWithProvider builder) {
        DslPart body = PactDslJsonArray.arrayEachLike()
                .integerType("id")
                .stringType("name")
                .stringType("position")
                .numberType("age")
                .closeObject();
        return builder.given("findByDepartment")
                .uponReceiving("findByDepartment")
                    .path("/employees/department/1")
                    .method("GET")
                .willRespondWith()
                    .status(200)
                    .body(body).toPact();
    }

    @Test
    @PactTestFor(providerName = "employee-service", 
                 pactVersion = PactSpecVersion.V3)
    public void verifyFindDepartmentPact(MockServer mockServer) {
        EmployeeClient client = RestClientBuilder.newBuilder()
                .baseUri(URI.create(mockServer.getUrl()))
                .build(EmployeeClient.class);
        List<Employee> employees = client.findByDepartment(1L);
        assertNotNull(employees);
        assertTrue(employees.size() > 0);
        assertNotNull(employees.get(0).getId());
    }
}

The test for the integration between organization-service and department-service is pretty similar. Let’s take a look at the REST client interface.

@ApplicationScoped
@Path("/departments")
@RegisterRestClient(configKey = "department")
public interface DepartmentClient {

    @GET
    @Path("/organization/{organizationId}")
    @Produces(MediaType.APPLICATION_JSON)
    List<Department> findByOrganization(@PathParam("organizationId") Long organizationId);

    @GET
    @Path("/organization/{organizationId}/with-employees")
    @Produces(MediaType.APPLICATION_JSON)
    List<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId);

}

Here’s the implementation of our contract test. However, instead of a single endpoint, we are testing two interactions with the GET /departments/organization/{id} and GET /departments/organization/{id}/with-employees.

@QuarkusTest
@ExtendWith(PactConsumerTestExt.class)
public class DepartmentClientContractTests {

   @Pact(provider = "department-service", 
         consumer = "organization-service")
   public RequestResponsePact callFindDepartment(
      PactDslWithProvider builder) {
      DslPart body = PactDslJsonArray.arrayEachLike()
            .integerType("id")
            .stringType("name")
            .closeObject();
      DslPart body2 = PactDslJsonArray.arrayEachLike()
            .integerType("id")
            .stringType("name")
            .array("employees")
               .object()
                  .integerType("id")
                  .stringType("name")
                  .stringType("position")
                  .integerType("age")
               .closeObject()
            .closeArray();
      return builder
            .given("findByOrganization")
               .uponReceiving("findByOrganization")
                  .path("/departments/organization/1")
                  .method("GET")
               .willRespondWith()
                  .status(200)
                  .body(body)
            .given("findByOrganizationWithEmployees")
               .uponReceiving("findByOrganizationWithEmployees")
                  .path("/departments/organization/1/with-employees")
                  .method("GET")
               .willRespondWith()
                  .status(200)
                  .body(body2)
            .toPact();
   }

   @Test
   @PactTestFor(providerName = "department-service", 
                pactVersion = PactSpecVersion.V3)
   public void verifyFindByOrganizationPact(MockServer mockServer) {
      DepartmentClient client = RestClientBuilder.newBuilder()
             .baseUri(URI.create(mockServer.getUrl()))
             .build(DepartmentClient.class);
      List<Department> departments = client.findByOrganization(1L);
      assertNotNull(departments);
      assertTrue(departments.size() > 0);
      assertNotNull(departments.get(0).getId());

      departments = client.findByOrganizationWithEmployees(1L);
      assertNotNull(departments);
      assertTrue(departments.size() > 0);
      assertNotNull(departments.get(0).getId());
      assertFalse(departments.get(0).getEmployees().isEmpty());
   }

}

Publish Contracts to the Pact Broker

That’s not all. We are still on the consumer side. After running the tests we need to publish the contract to the Pact broker. It is not performed automatically by Pact. To achieve it, we first need to include the following Maven plugin:

<plugin>
  <groupId>au.com.dius.pact.provider</groupId>
  <artifactId>maven</artifactId>
  <version>4.6.0</version>
  <configuration>
    <pactBrokerUrl>http://localhost:9292</pactBrokerUrl>
  </configuration>
</plugin>

In order to publish the contract after the test we need to include the pact:publish goal to the build command as shown below.

$ mvn clean package pact:publish

Now, we can switch the Pact Broker UI. As you see, there are several pacts generated during our tests. We can recognize it by the name of the consumer and provider.

We can go to the details of each contract. Here’s the description of the integration between the department-service and employee-service.

Create Contract Test for Provider

Once we published pacts to the broker, we can proceed to the implementation of contract tests on the provider side. In the current case, it is employee-service. Firstly, let’s include the Quarkus Pact Provider extension in the Maven dependencies.

<dependency>
  <groupId>io.quarkiverse.pact</groupId>
  <artifactId>quarkus-pact-provider</artifactId>
  <version>1.0.0.Final</version>
  <scope>test</scope>
</dependency>

We need to annotate the test class with the @Provider annotation and pass the name of the provider used on the consumer side (1). In the @PactBroker annotation, we have to pass the address of the broker (2). The test will load the contract published by the consumer side and test it against the running instance of the Quarkus app (under the test instance port) (3). We also need to extend the test template with the PactVerificationInvocationContextProvider class (4). Thanks to that, Pact will trigger the verification of contracts for each interaction defined by the @State method (6) (7). We also let Pact publish the verification results of each contract to the Pact broker (5).

@QuarkusTest
@Provider("employee-service") // (1)
@PactBroker(url = "http://localhost:9292") // (2)
public class EmployeeContractTests {

   @ConfigProperty(name = "quarkus.http.test-port") 
   int quarkusPort;
    
   @TestTarget
   HttpTestTarget target = new HttpTestTarget("localhost", 
      this.quarkusPort); // (3)

   @TestTemplate
   @ExtendWith(PactVerificationInvocationContextProvider.class) // (4)
   void pactVerificationTestTemplate(PactVerificationContext context) {
      context.verifyInteraction();
      System.setProperty("pact.provider.version", "1.2"); 
      System.setProperty("pact.verifier.publishResults", "true"); // (5)
   }

   @BeforeEach
   void beforeEach(PactVerificationContext context) {
      context.setTarget(new HttpTestTarget("localhost",
         this.quarkusPort));
   }

   @State("findByDepartment") // (6)
   void findByDepartment() {

   }

   @State("findByOrganization") // (7)
   void findByOrganization() {

   }
}

The value of @State refers to the name of the integration set on the consumer side. For example, line (6) in the code source above verifies the contract defined in the department-sevice as shown below.

As I mentioned before, the contract verification results are published to the Pact broker. You can check the status of verification using Pact Broker UI:

Run Tests in the CI Pipeline

Our last goal is to prepare the CI process for running Pact broker and contract tests during the build of our Quarkus apps. We will use CircleCI for that. Before running the contract tests we need to run the Pact broker container using Docker Compose. In order to do that we first need to use the Linux machine as a default executor and the docker orb (1). After that, we need to install Docker Compose and then use it to run the already prepared configuration in our docker-compose.yml file (2) (3). Then we can use the maven orb to run tests and publish contracts to the instance of the broker running during the tests (4).

version: 2.1

jobs:
  analyze:
    executor: # (1)
      name: docker/machine
      image: ubuntu-2204:2022.04.2
    steps:
      - checkout
      - docker/install-docker-compose # (2)
      - maven/with_cache:
          steps:
            - run:
                name: Build Images
                command: mvn package -DskipTests -Dquarkus.container-image.build=true
      - run: # (3)
          name: Run Pact Broker
          command: docker-compose up -d
      - maven/with_cache: # (4)
          steps:
            - run:
                name: Run Tests
                command: mvn package pact:publish -Dquarkus.container-image.build=false
      - maven/with_cache:
          steps:
            - run:
                name: Sonar Analysis
                command: mvn package sonar:sonar -DskipTests -Dquarkus.container-image.build=false


orbs:
  maven: circleci/maven@1.4.1
  docker: circleci/docker@2.2.0

workflows:
  maven_test:
    jobs:
      - analyze:
          context: SonarCloud

Here’s the final result of our build.

Final Thoughts

Contract testing is a useful approach for verifying interactions between microservices. Thanks to the Quarkus Pact extensions you can easily implement contract tests for your Quarkus apps. In this article, I showed how to use a Pact broker to store and share contracts between the tests. However, you can as well use the @PactFolder options to keep the contract JSON manifests inside the Git repository.

The post Contract Testing with Quarkus and Pact appeared first on Piotr's TechBlog.

In general, we will cover the following topics in this article:

• Using Spring Boot 3 in cloud-native development
• Provide service discovery for all microservices with Spring Cloud Netflix Eureka. Anticipating your questions – yes, Eureka is still there. It’s the last of Netflix microservices components still available in Spring Cloud
• Spring Cloud OpenFeign in inter-service communication
• Distributed configuration with Spring Cloud Config
• API Gateway pattern with Spring Cloud Gateway including a global OpenAPI documentation with the Springdoc project
• Collecting traces with Micrometer OpenTelemetry and Zipkin

Fortunately, the migration from Spring Boot 2 to 3 is not a painful process. You can even check it out in my example repository, which was originally written in Spring Boot 2. The list of changes is not large. However, times have changed during the last five years… And we will begin our considerations from that point.

Running Environment

Here are the results of my quick 1-day voting poll run on Twitter. I assume that those results are meaningful since around 900 people voted. As you probably expect, currently, the first-choice platform for running your Spring Boot microservices is Kubernetes. I don’t have a survey conducted five years ago, but the results would probably be significantly different. Even if you had Kubernetes in your organization 5 years ago, you were probably starting a migration of your apps or at least it was in progress. Of course, there might be some exceptions, but I’m thinking about the vast majority.

You could migrate to Kubernetes during that time, but also Kubernetes ecosystem has changed a lot. There are many useful tools and platform services you may easily integrate with your apps. We can at least mention Kubernetes native solutions like service mesh (e.g. Istio) or serverless (e.g. Knative). The main question here is: if I’m running microservices on Kubernetes are Spring Cloud components still relevant? The answer is: in most cases no. Of course, you can still use Eureka for service discovery, Spring Cloud Config for a distributed configuration, or Spring Cloud Gateway for the API gateway pattern. However, you can easily replace them with Kubernetes built-in mechanisms and additional platform services.

To conclude, this article is not aimed at Kubernetes users. It shows how to easily run microservices architecture anywhere. If you are looking for staff mainly related to Kubernetes you can read my articles about the best practices for Java apps and microservices there.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.

Before we proceed to the source code, let’s take a look at the following diagram. It illustrates the architecture of our sample system. We have three independent Spring Boot 3 microservices, which register themself in service discovery, fetch properties from the configuration service, and communicate with each other. The whole system is hidden behind the API gateway. Our Spring Boot 3 microservices send traces to the Zipkin instance using the Micrometer OTEL project.

Currently, the newest version of Spring Cloud is 2022.0.1. This version of spring-cloud-dependencies should be declared as a BOM for dependency management.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>2022.0.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Step 1: Configuration Server with Spring Cloud Config

To enable Spring Cloud Config feature for an application, we should first include spring-cloud-config-server to your project dependencies.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-config-server</artifactId>
</dependency>

Then enable running the embedded configuration server during application boot use @EnableConfigServer annotation.

@SpringBootApplication
@EnableConfigServer
public class ConfigApplication {

   public static void main(String[] args) {
      new SpringApplicationBuilder(ConfigApplication.class).run(args);
   }

}

By default Spring Cloud Config Server stores the configuration data inside the Git repository. We will change that behavior by activating the native mode. In this mode, Spring Cloud Config Server reads property sources from the classpath. We place all the YAML property files inside src/main/resources/config. Here’s the config server application.yml file. It activates the native mode and overrides a default port to 8088.

server:
  port: 8088
spring:
  profiles:
    active: native

The YAML filename will be the same as the name of the service. For example, the YAML file of discovery-service is located here: src/main/resources/config/discovery-service.yml. Besides a default profile, we will also define the custom docker profile. Therefore the name of the config file will contain the docker suffix. On the default profile, we are connecting services through localhost with dynamically assigned ports. So, the typical configuration file for the default profile will look like that:

server:
  port: 0

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8061/eureka/

Here’s the typical configuration file for the default profile:

server:
  port: 8080

eureka:
  client:
    serviceUrl:
      defaultZone: http://discovery-service:8061/eureka/

In order to connect the config server on the client side we need to include the following module in Maven dependencies:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-config</artifactId>
</dependency>

Depending on the running environment (localhost or docker) we need to provide different addresses for the config server:

spring:
  config:
    import: "optional:configserver:http://config-service:8088"
    activate:
      on-profile: docker
---
spring:
  application:
    name: discovery-service
  config:
    import: "optional:configserver:http://localhost:8088"

Step 2: Discovery Server with Spring Cloud Netflix Eureka

Of course, you can replace Eureka with any other discovery server supported by Spring Cloud. It can be Consul, Alibaba Nacos, or Zookeeper. The best way to run the Eureka server is just to embed it into the Spring Boot app. In order to do that, we first need to include the following Maven dependency:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>

Then we need to set the @EnableEurekaServer annotation on the main class.

@SpringBootApplication
@EnableEurekaServer
public class DiscoveryApplication {

   public static void main(String[] args) {
      new SpringApplicationBuilder(DiscoveryApplication.class).run(args);
   }

}

There is nothing new with that. As I already mentioned, the configuration files, discovery-service.yml or discovery-service-docker.yml, should be placed inside config-service module. We have changed Eureka’s running port from the default value (8761) to 8061. For the standalone Eureka instance, we have to disable registration and omit to fetch the registry. We just want to activate a single-node, demo discovery server.

server:
  port: 8061

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/

Once you have successfully started the application you may visit Eureka Dashboard available under the address http://localhost:8061/.

Step 3: Build Apps with Spring Boot 3 and Spring Cloud

Let’s take a look at a list of required Maven modules for our microservices. Each app has to get a configuration from the config-service and needs to register itself in the discovery-service. It also exposes REST API, automatically generates API documentation, and export tracing info to the Zipkin instance. We use the springdoc-openapi v2 library dedicated to Spring Boot 3. It generates documentation in both JSON and YAML formats available under the v3/api-docs path (or /v3/api-docs.yaml for the YAML format). In order to export traces to the Zipkin server, we will include the opentelemetry-exporter-zipkin module.

<dependencies>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-api</artifactId>
    <version>2.0.2</version>
  </dependency>
</dependencies>

For the apps that call other services, we also need to include a declarative REST client. We will use Spring Cloud OpenFeign.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

OpenFeign client automatically integrates with the service discovery. We just to set the name under which it is registered in Eureka inside the @FeingClient annotation. In order to create a client, we need to define an interface containing all the endpoints it has to call.

@FeignClient(name = "employee-service")
public interface EmployeeClient {

   @GetMapping("/organization/{organizationId}")
   List<Employee> findByOrganization(@PathVariable("organizationId") Long organizationId);
	
}

During the demo, we will send all the traces to Zipkin. It requires setting the value of the probability parameter to 1.0. In order to override the default URL of Zipkin we need to use the management.zipkin.tracing.endpoint property.

management:
  tracing:
    sampling:
      probability: 1.0
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans

Here’s the implementation of the @RestController in department-service. It injects the repository bean to interact with the database, and the Feign client bean to communicate with employee-service. The rest of the code is pretty simple.

@RestController
public class DepartmentController {

  private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

  DepartmentRepository repository;
  EmployeeClient employeeClient;

  public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient) {
    this.repository = repository;
    this.employeeClient = employeeClient;
  }

  @PostMapping("/")
  public Department add(@RequestBody Department department) {
    LOGGER.info("Department add: {}", department);
    return repository.add(department);
  }
	
  @GetMapping("/{id}")
  public Department findById(@PathVariable("id") Long id) {
    LOGGER.info("Department find: id={}", id);
    return repository.findById(id);
  }
	
  @GetMapping("/")
  public List<Department> findAll() {
    LOGGER.info("Department find");
    return repository.findAll();
  }
	
  @GetMapping("/organization/{organizationId}")
  public List<Department> findByOrganization(@PathVariable("organizationId") Long organizationId) {
    LOGGER.info("Department find: organizationId={}", organizationId);
    return repository.findByOrganization(organizationId);
  }
	
  @GetMapping("/organization/{organizationId}/with-employees")
  public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
    LOGGER.info("Department find: organizationId={}", organizationId);
    List<Department> departments = repository.findByOrganization(organizationId);
    departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
    return departments;
  }
	
}

As you see there are almost no differences in the app implementation between Spring Boot 2 and 3. The only thing you would have to do is to change all the javax.persistence to the jakarta.persistance.

Step 4: API Gateway with Spring Cloud Gateway

A gateway-service is the last app in our microservices architecture with Spring Boot 3. Beginning from Spring Boot 2 Spring Cloud Gateway replaced Netflix Zuul. We can also install it on Kubernetes using, for example, the Helm chart provided by VMWare Tanzu.

We will create a separate application with the embedded gateway. In order to do that we need to include Spring Cloud Gateway Starter in the Maven dependencies. Since our gateway has to interact with discovery and config services, it also includes Eureka Client Starter and Spring Cloud Config Starter. We don’t want to use it just as a proxy to the downstream services, but also we expose there OpenAPI documentation generated by all the apps. Since Spring Cloud Gateway is built on top of Spring WebFlux, we need to include Springdoc starters dedicated to that project.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>
<dependency>
  <groupId>org.springdoc</groupId>
  <artifactId>springdoc-openapi-starter-webflux-api</artifactId>
  <version>2.0.2</version>
</dependency>
<dependency>
  <groupId>org.springdoc</groupId>
  <artifactId>springdoc-openapi-starter-webflux-ui</artifactId>
  <version>2.0.2</version>
</dependency>

In order to expose OpenAPI documentation from multiple v3/api-docs endpoints we need to use the GroupedOpenApi object. It should provide a way to switch between documentation generated by employee-service, department-service and organization-service. Those services run on dynamic addresses (or at least random ports). In that case, we will use the RouteDefinitionLocator bean to grab the current URL of each service. Then we just need to filter a list of routes to find only those related to our three microservices. Finally, we create the GroupedOpenApi containing a service name and path.

@SpringBootApplication
public class GatewayApplication {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(GatewayApplication.class);

   public static void main(String[] args) {
      SpringApplication.run(GatewayApplication.class, args);
   }

   @Autowired
   RouteDefinitionLocator locator;

   @Bean
   public List<GroupedOpenApi> apis() {
      List<GroupedOpenApi> groups = new ArrayList<>();
      List<RouteDefinition> definitions = locator
         .getRouteDefinitions().collectList().block();
      assert definitions != null;
      definitions.stream().filter(routeDefinition -> routeDefinition
         .getId()
         .matches(".*-service"))
         .forEach(routeDefinition -> {
            String name = routeDefinition.getId()
               .replaceAll("-service", "");
            groups.add(GroupedOpenApi.builder()
               .pathsToMatch("/" + name + "/**").group(name).build());
         });
      return groups;
   }

}

Here’s the configuration of gateway-service. We should enable integration with the discovery server by setting the property spring.cloud.gateway.discovery.locator.enabled to true. Then we may proceed to define the route rules. We use the Path Route Predicate Factory for matching the incoming requests, and the RewritePath GatewayFilter Factory for modifying the requested path to adapt it to the format exposed by downstream services. The uri parameter specifies the name of the target service registered in the discovery server. For example, organization-service is available on the gateway under the /organization/** path thanks to the predicate Path=/organization/**, and the rewrite path from /organization/** to the /**.

spring:
  output:
    ansi:
      enabled: always
  cloud:
    gateway:
      discovery:
        locator:
          enabled: true
      routes:
      - id: employee-service
        uri: lb://employee-service
        predicates:
        - Path=/employee/**
        filters:
        - RewritePath=/employee/(?<path>.*), /$\{path}
      - id: department-service
        uri: lb://department-service
        predicates:
        - Path=/department/**
        filters:
        - RewritePath=/department/(?<path>.*), /$\{path}
      - id: organization-service
        uri: lb://organization-service
        predicates:
        - Path=/organization/**
        filters:
        - RewritePath=/organization/(?<path>.*), /$\{path}
      - id: openapi
        uri: http://localhost:${server.port}
        predicates:
        - Path=/v3/api-docs/**
        filters:
        - RewritePath=/v3/api-docs/(?<path>.*), /$\{path}/v3/api-docs

springdoc:
  swagger-ui:
    urls:
      - name: employee
        url: /v3/api-docs/employee
      - name: department
        url: /v3/api-docs/department
      - name: organization
        url: /v3/api-docs/organization

As you see above, we are also creating a dedicated route for Springdoc OpenAPI. It rewrites the path for the /v3/api-docs context to serve it properly in the Swagger UI.

Step 5: Running Spring Boot 3 Microservices

Finally, we can run all our microservices. With the current configuration in the repository, you can start them directly on your laptop or with Docker containers.

Option 1: Starting directly on the laptop

In total, we have 6 apps to run: 3 microservices, a discovery server, a config server, and a gateway. We also need to run Zipkin to collect and store traces from communication between the services. In the first step, we should start the config-service. We can use Spring Boot Maven plugin for that. Just go to the config-service directory and the following command. It is exposed on the 8088 port.

$ mvn spring-boot:run

We should repeat the same step for all the other apps. The discovery-service is listening on the 8061 port, while the gateway-service on the 8060 port. Microservices will start on the dynamically generated port number thanks to the server.port=0 property in config. In the final step, we can run Zipkin using its Docker container with the following command:

$ docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

Option 2: Build images and run them with Docker Compose

In the first step, we will build the whole Maven project and Docker images for all the apps. I created a profile build-image that needs to be activated to build images. It mostly uses the build-image step provided by the Spring Boot Maven Plugin. However, for config-service and discovery-service I’m using Jib because it is built on top of the base image with curl installed. For both these services Docker compose needs to verify health checks before starting other containers.

$ mvn clean package -Pbuild-image

The docker-compose.yml is available in the repository root directory. The whole file is visible below. We need to run config-service before all other apps since it provides property sources. Secondly, we should start discovery-service. In both these cases, we are defining a health check that tests the HTTP endpoint using curl inside the container. Once we start and verify config-service and discovery-service we may run gateway-service and all the microservices. All the apps are running with the docker Spring profile activated thanks to the SPRING_PROFILES_ACTIVE environment variable. It corresponds to the spring.profiles.active param that may be defined in configuration properties.

version: "3.7"
services:
  zipkin:
    container_name: zipkin
    image: openzipkin/zipkin
    ports:
      - "9411:9411"
  config-service:
    image: piomin/config-service:1.1-SNAPSHOT
    ports:
      - "8088:8088"
    healthcheck:
      test: curl --fail http://localhost:8088/employee/docker || exit 1
      interval: 5s
      timeout: 2s
      retries: 3
  discovery-service:
    image: piomin/discovery-service:1.1-SNAPSHOT
    ports:
      - "8061:8061"
    depends_on:
      config-service:
        condition: service_healthy
    links:
      - config-service
    healthcheck:
      test: curl --fail http://localhost:8061/eureka/v2/apps || exit 1
      interval: 4s
      timeout: 2s
      retries: 3
    environment:
      SPRING_PROFILES_ACTIVE: docker
  employee-service:
    image: piomin/employee-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  department-service:
    image: piomin/department-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - employee-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  organization-service:
    image: piomin/organization-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - employee-service
      - department-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  gateway-service:
    image: piomin/gateway-service:1.1-SNAPSHOT
    ports:
      - "8060:8060"
    depends_on:
      discovery-service:
        condition: service_healthy
    environment:
      SPRING_PROFILES_ACTIVE: docker
    links:
      - config-service
      - discovery-service
      - employee-service
      - department-service
      - organization-service
      - zipkin

Finally, let’s run all the apps using Docker Compose:

$ docker-compose up

Try it out

Once you start all the apps you can perform some test calls to the services through the gateway-service. It listening on the 8060 port. There is some test data automatically generated during startup. You can call the following endpoint to test all the services and communication between them:

$ curl http://localhost:8060/employee/
$ curl http://localhost:8060/department/organization/1
$ curl http://localhost:8060/department/organization/1/with-employees
$ curl http://localhost:8060/organization/
$ curl http://localhost:8060/organization/1/with-departments

Here are the logs generated by the apps during the calls visible above:

Let’s display Swagger UI exposed on the gateway. You can easily switch between contexts for all three microservices as you see below:

We can go to the Zipkin dashboard to verify the collected traces:

Final Thoughts

Treat this article as a quick guide to the most common components related to microservices with Spring Boot 3. I focused on showing you some new features since my last article on this topic. You could read how to implement tracing with Micrometer OpenTelemetry, generate API docs with Springdoc, or build Docker images with Spring Boot Maven Plugin.

The post Microservices with Spring Boot 3 and Spring Cloud appeared first on Piotr's TechBlog.

Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme and secure-caller. After that, you should just follow my instructions. Let’s begin.

How it works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.

Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap or Secret. Then it performs a rolling upgrade on pods, which use the particular ConfigMap or Secret. Here is the visualization of the described architecture.

Prerequisites

Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:

1. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
2. Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager

Install Cert Manager and Stakater Reloader

In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart using the following command:

$ helm install my-release cert-manager jetstack/cert-manager

The same with the Stakater Reloader – first we need to add the stakater charts repository:

$ helm repo add stakater https://stakater.github.io/stakater-charts

Then, we can install the latest version of the chart:

$ helm install my-reloader stakater/reloader

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0          1m
my-reloader-reloader-7566fdc68c-qj9l4         1/1     Running   0          1m

That’s all. Now we can proceed to the implementation.

HTTPS with Spring Boot

Our first app secure-callme exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property. Here’s the configuration for our Spring Boot inside the application.yml file.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

We will set the values of CERT_PATH and PASSWORD at the level of Kubernetes Deployment. Now, let’s switch to the secure-caller implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient dependency.

<dependency>
  <groupId>org.apache.httpcomponents.client5</groupId>
  <artifactId>httpclient5</artifactId>
</dependency>

Now, we will use Apache HttpClient as a low-level client for the Spring RestTemplate. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate @Bean we use RestTemplateBuilder.

@SpringBootApplication
public class SecureCaller {

   public static void main(String[] args) {
      SpringApplication.run(SecureCaller.class, args);
   }

   @Autowired
   ClientSSLProperties clientSSLProperties;

   @Bean
   RestTemplate builder(RestTemplateBuilder builder) throws 
      GeneralSecurityException, IOException {

      final SSLContext sslContext = new SSLContextBuilder()
         .loadTrustMaterial(
            new File(clientSSLProperties.getTrustStore()),
            clientSSLProperties.getTrustStorePassword().toCharArray())
         .loadKeyMaterial(
            new File(clientSSLProperties.getKeyStore()),
            clientSSLProperties.getKeyStorePassword().toCharArray(),
            clientSSLProperties.getKeyStorePassword().toCharArray()
         )
         .build();

      final SSLConnectionSocketFactory sslSocketFactory = 
         SSLConnectionSocketFactoryBuilder.create()
                .setSslContext(sslContext)
                .build();

      final HttpClientConnectionManager cm = 
         PoolingHttpClientConnectionManagerBuilder.create()
                .setSSLSocketFactory(sslSocketFactory)
                .build();

      final HttpClient httpClient = HttpClients.custom()
         .setConnectionManager(cm)
         .evictExpiredConnections()
         .build();

      return builder
         .requestFactory(() -> 
            new HttpComponentsClientHttpRequestFactory(httpClient))
         .build();
   }
}

The client credentials are taken from configuration settings under the client.ssl key. Here is the @ConfigurationProperties class used the RestTemplateBuilder in the previous step.

@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {

   private String keyStore;
   private String keyStorePassword;
   private String trustStore;
   private String trustStorePassword;

   // GETTERS AND SETTERS ...

}

Here’s the configuration for the secure-caller inside application.yml file. The same as for the secure-callme we expose the REST endpoint over HTTPS.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

client.url: https://${HOST}:8443/callme
client.ssl:
  key-store: ${CLIENT_CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CLIENT_CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}

The secure-caller app calls GET /callme exposed by the secure-callme app using customized RestTemplate.

@RestController
public class SecureCallerController {

   RestTemplate restTemplate;

   @Value("${client.url}")
   String clientUrl;

   public SecureCallerController(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @GetMapping("/caller")
   public String call() {
      return "I'm `secure-caller`! calling... " +
             restTemplate.getForObject(clientUrl, String.class);
   }

}

Generate and Renew Certificates on Kubernetes with Cert Manager

With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-clusterissuer
spec:
  selfSigned: {}

After that, we can generate certificates. Here’s the cert-manager Certificate object for the secure-callme app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy to Always.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores: # (1)
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef: # (2)
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme
  secretName: secure-callme-cert
  commonName: secure-callme # (3)
  duration: 1h # (4)
  renewBefore: 5m  # (5)

The Certificate object for secure-caller is very similar. The only difference is in the CN field. We will use the port-forward option during the test, so I’ll set the domain name to localhost (1).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-caller-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - localhost
    - secure-caller
  secretName: secure-caller-cert
  commonName: localhost # (1)
  duration: 1h
  renewBefore: 5m

After applying both manifests we can display a list of Certificates. Each of them is related to the Secret with the same name:

$ kubectl get certificate
NAME                 READY   SECRET               AGE
secure-caller-cert   True    secure-caller-cert   1m
secure-callme-cert   True    secure-callme-cert   1m

Here are the details of the secure-callme-cert Secret. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store and server.ssl.key-store properties). There is also a certificate (tls.crt), a private key (tls.key), and CA (ca.crt).

$ kubectl describe secret secure-callme-cert
Name:         secure-callme-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: secure-callme
              cert-manager.io/certificate-name: secure-callme-cert
              cert-manager.io/common-name: secure-callme
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: ss-clusterissuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:          550 bytes
keystore.jks:    1029 bytes
tls.crt:         550 bytes
tls.key:         227 bytes
truststore.jks:  422 bytes

Deploy and Reload Apps on Kubernetes

Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme app.

First of all, we need to reload the app each time the secure-callme-cert changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload (1). The annotation should contain the name of Secret, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH env variable (2). We are mounting the whole secure-callme-cert Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme
  annotations:
    # (1)
    secret.reloader.stakater.com/reload: "secure-callme-cert" 
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme
    spec:
      containers:
        - image: piomin/secure-callme
          name: secure-callme
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH # (2)
              value: /opt/secret
          volumeMounts:
            - mountPath: /opt/secret # (3)
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert # (4)

The password to the key store files is available inside the jks-password-secret:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

There is also the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme
  name: secure-callme
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme
  type: ClusterIP

Now, go to the secure-callme directory and just run the following command:

$ skaffold run

The Deployment manifest of the secure-caller app is a little bit more complicated. The same as before we need to reload the app on Secret change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller
  # (1)
  annotations:
    secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller
    spec:
      containers:
        - image: piomin/secure-caller
          name: secure-caller
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            # (2)
            - name: CERT_PATH
              value: /opt/secret
            # (3)
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        # (5)
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Then, go to the secure-caller directory and deploy the app. This time we enable port-forward to easily test the app locally.

$ skaffold run --port-forward

Let’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.

$ kubectl get deploy 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
my-cert-manager              1/1     1            1           1h
my-cert-manager-cainjector   1/1     1            1           1h
my-cert-manager-webhook      1/1     1            1           1h
my-reloader-reloader         1/1     1            1           1h
secure-caller                1/1     1            1           1h
secure-callme                1/1     1            1           1h

Testing Renew of Certificates on Kubernetes

The secure-caller app is available on the 8443 port locally, while the secure-callme app is available inside the cluster under the service name secure-callme. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.key}' | base64 --decode > tls.key

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.crt}' | base64 --decode > tls.crt

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.ca\.crt}' | base64 --decode > ca.crt

Now, we can call the GET /caller endpoint using the following curl command. Under the hood, the secure-caller calls the endpoint GET /callme exposed by the secure-callme also over HTTPS. If you did everything according to the instruction you should have the same result as below.

$ curl https://localhost:8443/caller \
  --key tls.key \
  --cert tls.crt \
  --cacert ca.crt 
I'm `secure-caller`! calling... I'm `secure-callme`!

Our certificate is valid for one hour.

Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.

Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.

Final Thoughts

Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.

The post Renew Certificates on Kubernetes with Cert Manager and Reloader appeared first on Piotr's TechBlog.