Here is a brief overview of all my published books.

Motivation

I won’t hide that this post is mainly directed at my blog subscribers and people who enjoy reading it and value my writing style. As you know, all posts and content on my blog, along with sample application repositories on GitHub, are always accessible to you for free. Over the past eight years, I have worked to publish high-quality content on my blog, and I plan to keep doing so. It is a part of my life, a significant time commitment, but also a lot of fun and a hobby.

I want to explain why I decided to write this book, why now, and why in this way. But first, a bit of background. I wrote my last book first, then my first book, over seven years ago. It focused on topics I was mainly involved with at the time, specifically Spring Boot and Spring Cloud. Since then, a lot of time has passed, and much has changed – not only in the technology itself but also a little in my personal life. Today, I am more involved in Kubernetes and container topics than, for example, Spring Cloud. For years, I have been helping various organizations transition from traditional application architectures to cloud-native models based on Kubernetes. Of course, Java remains my main area of expertise. Besides Spring Boot, I also really like the Quarkus framework. You can read a lot about both in my book on Kubernetes.

Based on my experience over the past few years, involving development teams is a key factor in the success of the Kubernetes platform within an organization. Ultimately, it is the applications developed by these teams that are deployed there. For developers to be willing to use Kubernetes, it must be easy for them to do so. That is why I persuade organizations to remove barriers to using Kubernetes and to design it in a way that makes it easier for development teams. On my blog and in this book, I aim to demonstrate how to quickly and simply launch applications on Kubernetes using frameworks such as Spring Boot and Quarkus.

It’s an unusual time to publish a book. AI agents are producing more and more technical content online. More often than not, instead of grabbing a book, people turn to an AI chatbot for a quick answer, though not always the best one. Still, a book that thoroughly introduces a topic and offers a step-by-step guide remains highly valuable.

Content of the Book

This book demonstrates that Java is an excellent choice for building applications that run on Kubernetes. In the first chapter, I’ll show you how to quickly build your application, create its image, and run it on Kubernetes without writing a single line of YAML or Dockerfile. This chapter also covers the minimum Kubernetes architecture you must understand to manage applications effectively in this environment. The second chapter, on the other hand, demonstrates how to effectively organize your local development environment to work with a Kubernetes cluster. You’ll see several options for running a distribution of your cluster locally and learn about the essential set of tools you should have. The third chapter outlines best practices for building applications on the Kubernetes platform. Most of the presented requirements are supported by simple examples and explanations of the benefits of meeting them. The fourth chapter presents the most valuable tools for the inner development loop with Kubernetes. After reading the first four chapters, you will understand the main Kubernetes components related to application management, enabling you to navigate the platform efficiently. You’ll also learn to leverage Spring Boot and Quarkus features to adapt your application to Kubernetes requirements.

In the following chapters, I will focus on the benefits of migrating applications to Kubernetes. The first area to cover is security. Chapter five discusses mechanisms and tools for securing applications running in a cluster. Chapter six describes Spring and Quarkus projects that enable native integration with the Kubernetes API from within applications. In chapter seven, you’ll learn about the service mesh tool and the benefits of using it to manage HTTP traffic between microservices. Chapter eight addresses the performance and scalability of Java applications in a Kubernetes environment. Chapter Eight demonstrates how to design a CI/CD process that runs entirely within the cluster, leveraging Kubernetes-native tools for pipeline building and the GitOps approach. This book also covers AI. In the final, ninth chapter, you’ll learn how to run a simple Java application that integrates with an AI model deployed on Kubernetes.

Publication

I decided to publish my book on Leanpub. Leanpub is a platform for writing, publishing, and selling books, especially popular among technical content authors. I previously published a book with Packt, but honestly, I was alone during the writing process. Leanpub is similar but offers several key advantages over publishers like Packt. First, it allows you to update content collaboratively with readers and keep it current. Even though my book is finished, I don’t rule out adding more chapters, such as on AI on Kubernetes. I also look forward to your feedback and plan to improve the content and examples in the repository continuously. Overall, this has been another exciting experience related to publishing technical content.

And when you buy such a book, you can be sure that most of the royalties go to me as the author, unlike with other publishers, where most of the royalties go to them as promoters. So, I’m looking forward to improving my book with you!

Conclusion

My book aims to bring together all the most interesting elements surrounding Java application development on Kubernetes. It is intended not only for developers but also for architects and DevOps teams who want to move to the Kubernetes platform.

The post A Book: Hands-On Java with Kubernetes appeared first on Piotr's TechBlog.

We will do the same thing as described in that article, but fully with the GitOps approach applied by Argo CD. The main goal here is to integrate Postgres with the Vault database secrets engine to generate database credentials dynamically and initialize the DB schema for the sample Spring Boot app. In order to achieve these goals, we are going to install two Kubernetes operators: Atlas and Vault Config. Atlas is a tool for managing the database schema as code. Its Kubernetes Operator allows us to define the schema and apply it to our database using the CRD objects. The Vault Config Operator provided by the Red Hat Community of Practice does a very similar thing but for Hashicorp Vault.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions

How It Works

Before we start, let’s describe our sample scenario. Thanks to the database secrets engine Vault integrates with Postgres and generates its credentials dynamically based on configured roles. On the other hand, our sample Spring Boot app integrates with Vault and uses its database engine to authenticate against Postgres. All the aspects of that scenario are managed in the GitOps style. Argo CD installs Vault, Postgres, and additional operators on Kubernetes via their Helm charts. Then, it applies all the required CRD objects to configure both Vault and Postgres. We keep the whole configuration in a single Git repository in the form of YAML manifests.

Argo CD prepares the configuration on Vault and creates a table on Postgres for the sample Spring Boot app. Our app integrates with Vault through the Spring Cloud Vault project. It also uses Spring Data JPA to interact with the database. Here’s the illustration of our scenario.

Install Argo CD on Kubernetes

Traditionally, we need to start our GitOps exercise by installing Argo CD on the Kubernetes cluster. Of course, we can do it using the Helm chart. In the first step, we need to add the following repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

We will add one parameter to the argo-cm ConfigMap to ignore the MutatingWebhookConfiguration kind. This step is not necessary. It allows us to ignore the specific resource generated by one of the Helm charts used in the further steps. Thanks to that we will have everything in Argo CD in the “green” color Here’s the Helm values.yaml file with the required configuration:

configs:
  cm:
    resource.exclusions: |
      - apiGroups:
        - admissionregistration.k8s.io
        kinds:
        - MutatingWebhookConfiguration
        clusters:
        - "*"

Now, we can install the Argo CD in the argocd namespace using the configuration previously defined in the values.yml file:

$ helm install argo-cd argo/argo-cd \
    --version 6.7.8 \
    -n argo \
    --create-namespace

That’s not all. Since the Atlas operator is available in the OCI-type Helm repository, we need to apply the following Secret in the argocd namespace. By default, Argo CD doesn’t allow the OCI-type repo, so we need to include the enableOCI parameter in the definition.

apiVersion: v1
kind: Secret
metadata:
  name: ghcr-io-helm-oci
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: ariga
  url: ghcr.io/ariga
  enableOCI: "true"
  type: helm

Let’s take a look at the list of repositories in the Argo CD UI dashboard. You should see the “Successful” connection status.

Prepare Configuration Manifests for Argo CD

Config Repository Structure

Let me first explain the structure of our Git config repository. The additional configuration is stored in the apps directory. It includes the CRD objects required to initialize the database schema or Vault engines. In the bootstrap directory, we keep the values.yaml file for each Helm chart managed by Argo CD. It’s all that we need. The bootstrap-via-appset/bootstrap.yaml contains the definition of Argo CD ApplicationSet we need to apply to the Kubernetes cluster. This ApplicationSet will generate all required Argo CD applications responsible for installing the charts and creating CRD objects.

.
├── apps
│   ├── postgresql
│   │   ├── database.yaml
│   │   ├── policies.yaml
│   │   ├── roles.yaml
│   │   └── schema.yaml
│   └── vault
│       └── job.yaml
├── bootstrap
│   ├── values
│   │   ├── atlas
│   │   │   └── values.yaml
│   │   ├── cert-manager
│   │   │   └── values.yaml
│   │   ├── postgresql
│   │   │   └── values.yaml
│   │   ├── vault
│   │   │   └── values.yaml
│   │   └── vault-config-operator
│   │       └── values.yaml
└── bootstrap-via-appset
    └── bootstrap.yaml

Bootstrap with the Argo CD ApplicationSet

Let’s take a look at the ApplicationSet. It’s pretty interesting (I hope :)). I’m using here some relatively new Argo CD features like multiple sources (Argo CD 2.6) or application sets template patch (Argo CD 2.10). We need to generate an Argo CD Application per each tool we want to install on Kubernetes (1). In the generators section, we define parameters for Vault, PostgreSQL, Atlas Operator, Vault Config Operator, and Cert Manager (which is required by the Vault Config Operator). In the templatePatch section, we prepare a list of source repositories used by each Argo CD Application (2). There is always a Helm chart repo, which refers to our Git repository containing dedicated values.yaml files. For the Vault and PostgreSQL charts, we include another source containing CRDs or additional Kubernetes objects. We will discuss it later.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bootstrap-config
  namespace: argocd
spec:
  goTemplate: true
  generators:
  - list:
      elements:
        - chart: vault
          name: vault
          repo: https://helm.releases.hashicorp.com
          revision: 0.27.0
          namespace: vault
          postInstall: true
        - chart: postgresql
          name: postgresql
          repo: https://charts.bitnami.com/bitnami
          revision: 12.12.10
          namespace: default
          postInstall: true
        - chart: cert-manager
          name: cert-manager
          repo: https://charts.jetstack.io
          revision: v1.14.4
          namespace: cert-manager
          postInstall: false
        - chart: vault-config-operator
          name: vault-config-operator
          repo: https://redhat-cop.github.io/vault-config-operator
          revision: v0.8.25
          namespace: vault-config-operator
          postInstall: false
        - chart: charts/atlas-operator
          name: atlas
          repo: ghcr.io/ariga
          revision: 0.4.2
          namespace: atlas
          postInstall: false
  template:
    metadata:
      name: '{{.name}}'
      annotations:
        argocd.argoproj.io/sync-wave: "1"
    spec:
      syncPolicy:
        automated: {}
        syncOptions:
          - CreateNamespace=true
      destination:
        namespace: '{{.namespace}}'
        server: https://kubernetes.default.svc
      project: default
  templatePatch: |
    spec:
      sources:
        - repoURL: '{{ .repo }}'
          chart: '{{ .chart }}'
          targetRevision: '{{ .revision }}'
          helm:
            valueFiles:
              - $values/bootstrap/values/{{ .name }}/values.yaml
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          ref: values
        {{- if .postInstall }}
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          path: apps/{{ .name }}
        {{- end }}

Once we apply the bootstrap-config ApplicationSet to the argocd namespace, all the magic just happens. You should see five applications in the Argo CD UI dashboard. All of them are automatically synchronized (Argo CD autoSync enabled) to the cluster. It does the whole job. Now, let’s analyze step-by-step what we have to put in that configuration.

The Argo CD ApplicationSet generates five applications for installing all required tools. Here’s the Application generated for installing Vault with Helm charts and applying an additional configuration stored in the apps/vault directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
spec:
  destination:
    namespace: vault
    server: https://kubernetes.default.svc
  project: default
  sources:
    - chart: vault
      helm:
        valueFiles:
          - $values/bootstrap/values/vault/values.yaml
      repoURL: https://helm.releases.hashicorp.com
      targetRevision: 0.27.0
    - ref: values
      repoURL: https://github.com/piomin/kubernetes-config-argocd.git
      targetRevision: HEAD
    - path: apps/vault
      repoURL: https://github.com/piomin/kubernetes-config-argocd.git
      targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Configure Vault on Kubernetes

Customize Helm Charts

Let’s take a look at the Vault values.yaml file. We run it in the development mode (single, in-memory node, no unseal needed). We will also enable the UI dashboard.

server:
  dev:
    enabled: true
ui:
  enabled: true

With the parameters visible above Argo CD installs Vault in the vault namespace. Here’s a list of running pods:

$ kubectl get po -n vault
NAME                                    READY   STATUS      RESTARTS      AGE
vault-0                                 1/1     Running     0            1h
vault-agent-injector-7f7f68d457-fvsd2   1/1     Running     0            1h

It also exposes Vault API under the 8200 port in the vault Kubernetes Service.

$ kubectl get svc -n vault
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
vault                      ClusterIP   10.110.69.159    <none>        8200/TCP,8201/TCP   21h
vault-agent-injector-svc   ClusterIP   10.111.24.183    <none>        443/TCP             21h
vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   21h
vault-ui                   ClusterIP   10.110.160.239   <none>        8200/TCP            21h

For the Vault Config Operator, we need to override the default address of Vault API to vault.vault.svc:8200 (an a). In order to do that, we need to set the VAULT_ADDR env variable in the values.yaml file. We also disable Prometheus monitoring and enable integration with Cert Manager. Thanks to “cert-manager” we don’t need to generate any certificates or keys manually.

enableMonitoring: false
enableCertManager: true
env:
  - name: VAULT_ADDR
    value: http://vault.vault:8200

Enable Vault Config Operator

The Vault Config Operator needs to authenticate against Vault API using Kubernetes Authentication. So we need to configure a root Kubernetes Authentication mount point and role. Then we can create more roles or other Vault objects via the operator. Here’s the Kubernetes Job responsible for configuring Kubernetes mount point and role. It uses the Vault image and the vault CLI available inside that image. As you see, it creates the vault-admin role allowed in the default namespace.

apiVersion: batch/v1
kind: Job
metadata:
  name: vault-admin-initializer
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  template:
    spec:
      containers:
        - name: vault-admin-initializer
          image: hashicorp/vault:1.15.2
          env:
            - name: VAULT_ADDR
              value: http://vault.vault.svc:8200
          command:
            - /bin/sh
            - -c
            - |
              export VAULT_TOKEN=root
              sleep 10
              vault auth enable kubernetes
              vault secrets enable database
              vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              vault write auth/kubernetes/role/vault-admin bound_service_account_names=default bound_service_account_namespaces=default policies=vault-admin ttl=1h
              vault policy write vault-admin - <<EOF
                path "/*" {
                  capabilities = ["create", "read", "update", "delete", "list","sudo"]
                }          
              EOF
      restartPolicy: Never

Argo CD applies such a Job after installing the Vault chart.

$ kubectl get job -n vault
NAME                      COMPLETIONS   DURATION   AGE
vault-admin-initializer   1/1           15s        1h

Configure Vault via CRDs

Once a root Kubernetes authentication is ready, we can proceed to the CRD object creation. In the first step, we create objects responsible for configuring a connection to the Postgres database. In the DatabaseSecretEngineConfig we set the connection URL, credentials, and the name of a Vault plugin used to interact with the database (postgresql-database-plugin). We also define a list of allowed roles (postgresql-default-role). In the next step, we define the postgresql-default-role DatabaseSecretEngineRole object. Of course, the name of the role should be the same as the name passed in the allowedRoles list in the previous step. The role defines a target database connection name in Vault and the SQL statement for creating new users with privileges.

kind: DatabaseSecretEngineConfig
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: postgresql-database-config
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  allowedRoles:
    - postgresql-default-role
  authentication:
    path: kubernetes
    role: vault-admin
  connectionURL: 'postgresql://{{username}}:{{password}}@postgresql.default:5432?sslmode=disable'
  path: database
  pluginName: postgresql-database-plugin
  rootCredentials:
    passwordKey: postgres-password
    secret:
      name: postgresql
  username: postgres
---
apiVersion: redhatcop.redhat.io/v1alpha1
kind: DatabaseSecretEngineRole
metadata:
  name: postgresql-default-role
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  creationStatements:
    - CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}"; GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO "{{name}}";
  maxTTL: 10m0s
  defaultTTL: 1m0s
  authentication:
    path: kubernetes
    role: vault-admin
  dBName: postgresql-database-config
  path: database

Once Argo CD applies both DatabaseSecretEngineConfig and DatabaseSecretEngineRole objects, we can verify it works fine by generating database credentials using the vault read command. We need to pass the name of the previously created role (postgresql-default-role). Our sample app will do the same thing but through the Spring Cloud Vault module.

Finally, we can create a policy and role for our sample Spring Boot. The policy requires only the privilege to generate new credentials:

kind: Policy
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: database-creds-view
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  authentication:
    path: kubernetes
    role: vault-admin
  policy: |
    path "database/creds/default" {
      capabilities = ["read"]
    }

Now, we have everything to proceed to the last step in this section. We need to create a Vault role with the Kubernetes authentication method dedicated to our sample app. In this role, we set the name and location of the Kubernetes ServiceAccount and the name of the Vault policy created in the previous step.

kind: KubernetesAuthEngineRole
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: database-engine-creds-role
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  authentication:
    path: kubernetes
    role: vault-admin
  path: kubernetes
  policies:
    - database-creds-view
  targetServiceAccounts:
    - default
  targetNamespaces:
    targetNamespaces:
      - default

Managing Postgres Schema with Atlas Operator

Finally, we can proceed to the last step in the configuration part. We will use the AtlasSchema CRD object to configure the database schema for our sample app. The object contains two sections: credentials and schema. In the credentials section, we refer to the PostgreSQL Secret to obtain a password. In the schema section, we create the person table with the id primary key.

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasSchema
metadata:
  name: sample-spring-cloud-vault
  annotations:
    argocd.argoproj.io/sync-wave: "4"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  credentials:
    scheme: postgres
    host: postgresql.default
    user: postgres
    passwordFrom:
      secretKeyRef:
        key: postgres-password
        name: postgresql
    database: postgres
    port: 5432
    parameters:
      sslmode: disable
  schema:
    sql: |
      create table person (
        id serial primary key,
        name varchar(255),
        gender varchar(255),
        age int,
        external_id int
      );

Here’s the corresponding app @Entity model class in the sample Spring Boot app.

@Entity
public class Person {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String name;
    private int age;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    private Integer externalId;   
    
   // GETTERS AND SETTERS ...
   
}

Once Argo CD applies the AtlasSchema object, we can verify its status. As you see, it has been successfully executed on the target database.

We can log in to the database using psql CLI and verify that the person table exists in the postgres database:

Run Sample Spring Boot App

Dependencies

For this demo, I created a simple Spring Boot application. It exposes REST API and connects to the PostgreSQL database. It uses Spring Data JPA to interact with the database. Here are the most important dependencies of our app in the Maven pom.xml:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>
<dependency>
  <groupId>org.postgresql</groupId>
  <artifactId>postgresql</artifactId>
  <scope>runtime</scope>
</dependency>

The first of them enables bootstrap.yml processing on the application startup. The third one includes Spring Cloud Vault Database engine support.

Integrate with Vault using Spring Cloud Vault

The only thing we need to do is to provide the right configuration settings. Here’s the minimal set of the required dependencies to make it work without any errors. The following configuration is provided in the bootstrap.yml file:

spring:
  application:
    name: sample-db-vault
  datasource:
    url: jdbc:postgresql://postgresql:5432/postgres #(1)
  jpa:
    hibernate:
      ddl-auto: update
  cloud:
    vault:
      config.lifecycle: #(2)
        enabled: true
        min-renewal: 10s
        expiry-threshold: 30s
      kv.enabled: false #(3)
      uri: http://vault.vault:8200 #(4)
      authentication: KUBERNETES #(5)
      postgresql: #(6)
        enabled: true
        role: postgresql-default-role
        backend: database
      kubernetes: #(7)
        role: database-engine-creds-role

Let’s analyze the configuration visible above in the details:

(1) Firstly, we need to set the database connection URL without any credentials. Our application uses standard properties for authentication against the database (spring.datasource.username and spring.datasource.password). Thanks to that, we don’t need to do anything else

(2) As you probably remember, the maximum TTL for the database lease is 10 minutes. We enable lease renewal every 30 seconds. Just for the demo purpose. You will see that Spring Cloud Vault will create new credentials in PostgreSQL every 30 seconds, and the application still works without any errors

(3) Vault KV is not needed here, since I’m using only the database engine

(4) The application is going to be deployed in the default namespace, while Vault is running in the vault namespace. So, the address of Vault should include the namespace name

(5) (7) Our application uses the Kubernetes authentication method to access Vault. We just need to set the role name, which is database-engine-creds-role. All other settings should be left with the default values

(6) We also need to enable postgres database backend support. The name of the backend in Vault is database and the name of the Vault role used for that engine is postgresql-default-role.

Run the App on Kubernetes

Finally, we can run our sample app on Kubernetes by applying the following YAML manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-deployment
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: piomin/sample-app:1.0-gitops
          ports:
            - containerPort: 8080
      serviceAccountName: default
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  type: ClusterIP
  selector:
    app: sample-app
  ports:
  - port: 8080

Our app exposes REST API under the /persons path. We can easily test it with curl after enabling port forwarding as shown below:

$ kubectl port-forward svc/sample-app 8080:8080
$ curl http://localhost:8080/persons

Final Thoughts

This article proves that we can effectively configure and manage tools like Postgres database or Hashicorp Vault on Kubernetes with Argo CD. The database schema or Vault configuration can be stored in the Git repository in the form of YAML manifests thanks to Atlas and Vault Config Kubernetes operators. Argo CD applies all required CRDs automatically, which results in the integration between Vault, Postgres, and our sample Spring Boot app.

The post GitOps on Kubernetes for Postgres and Vault with Argo CD appeared first on Piotr's TechBlog.

Our architecture consists of two Spring Boot microservices, an API gateway built on top of Spring Cloud Gateway, and a Keycloak authorization server. Spring Cloud Gateway acts here as an OAuth2 Client and OAuth2 Resource Server. For any incoming request, it verifies an access token before forwarding traffic to the downstream services. It initializes an authorization code flow procedure with Keycloak for any unauthenticated request. Our scenario needs to include the communication between internal microservices. They are both hidden behind the API gateway. The caller app invokes an endpoint exposed by the callme app. The HTTP client used in that communication has to use the access token sent by the gateway.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the oauth directory. You will find two Spring Boot microservices there: callme and caller. Of course, there is the gateway app built on top of Spring Cloud Gateway. After that, you should just follow my instructions. Let’s begin.

Run and Configure Keycloak

We are running Keycloak as a Docker container. By default, Keycloak exposes API and a web console on the port 8080. We also need to set an admin username and password with environment variables. Here’s the command used to run the Keycloak container:

$ docker run -d --name keycloak -p 8080:8080 \
    -e KEYCLOAK_ADMIN=admin \
    -e KEYCLOAK_ADMIN_PASSWORD=admin \
    quay.io/keycloak/keycloak:23.0.7 start-dev

Once the container starts, we can go to the UI admin console available under the http://localhost:8080/admin address. We will create a new realm. The name is that realm is demo. Instead of creating the required things manually, we can import the JSON resource file that contains the whole configuration of the realm. You can find such a resource file in my GitHub repository here: oauth/gateway/src/test/resources/realm-export.json. However, in the next parts of that section, we will use the Keycloak dashboard to create objects step by step. In case you import the configuration from the JSON resource file, you can just skip to the next section.

Then, we need to add a single OpenID Connect client to the demo realm. The name of our client is spring-with-test-scope. We should enable client authentication and put the right address in the “Valid redirect URIs” field (it can be the wildcard for testing purposes).

We need to save the name of the client and its secret. Those two settings have to be set on the application side.

Then, let’s create a new client scope with the TEST name.

Then, we have to add the TEST to the spring-with-test-scope client scopes.

We also need to create a user to authenticate against Keycloak. The name of our user is spring. In order to set the password, we need to switch to the “Credentials” tab. For my user, I choose the Spring_123 password.

Once we finish with the configuration, we can export it to the JSON file (the same file we can use when creating a new realm). Such a file will be useful later, for building automated tests with Testcontainers.

Unfortunately, Keycloak doesn’t export realm users to the file. Therefore, we need to add the following JSON to the users section in the exported file.

{
  "username": "spring",
  "email": "piotr.minkowski@gmail.com",
  "firstName": "Piotr",
  "lastName": "Minkowski",
  "enabled": true,
  "credentials": [
    {
      "type": "password",
      "value": "Spring_123"
    }
  ],
  "realmRoles": [
    "default-roles-demo",
    "USER"
  ]
}

Create Spring Cloud Gateway with OAuth2 Support and Keycloak

As I mentioned before, our gateway app will act as an OAuth2 Client and OAuth2 Resource Server. In that case, we include both the Spring Boot Auth2 Client Starter and the spring-security-oauth2-resource-server dependency. We also need to include the spring-security-oauth2-jose to decode JWT tokens automatically. Of course, we need to include the Spring Cloud Gateway Starter. Finally, we add dependencies for automated testing with JUnit. We will use Testcontainers to run the Keycloak container during the JUnit test. It can be achieved with the com.github.dasniko:testcontainers-keycloak dependency.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-oauth2-client</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-resource-server</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-jose</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-test</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>com.github.dasniko</groupId>
  <artifactId>testcontainers-keycloak</artifactId>
  <version>3.2.0</version>
 <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>junit-jupiter</artifactId>
  <version>1.19.6</version>
  <scope>test</scope>
</dependency>

Let’s begin with the Spring Security configuration. First, we need to annotate the Configuration bean with @EnableWebFluxSecurity. That’s because Spring Cloud Gateway uses the reactive version of the Spring web module. The oauth2Login() method is responsible for redirecting an unauthenticated request to the Keycloak login page. On the other hand, the oauth2ResourceServer() method verifies an access token before forwarding traffic to the downstream services.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {

    @Bean
    public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
        http.authorizeExchange(auth -> auth.anyExchange().authenticated())
                .oauth2Login(withDefaults())
                .oauth2ResourceServer((oauth2) -> oauth2.jwt(Customizer.withDefaults()));
        http.csrf(ServerHttpSecurity.CsrfSpec::disable);
        return http.build();
    }

}

That’s not all. We also need to provide several configuration settings with the spring.security.oauth2 prefix. The Spring OAuth2 Resource Server module will use the Keycloak JWKS endpoint to verify incoming JWT tokens. In the Spring OAuth2 Client section, we need to provide the address of the Keycloak issuer realm. Of course, we also need to provide the Keycloak client credentials, choose the authorization grant type and scope.

spring.security.oauth2:
  resourceserver:
    jwt:
      jwk-set-uri: http://localhost:8080/realms/demo/protocol/openid-connect/certs
  client:
    provider:
      keycloak:
        issuer-uri: http://localhost:8080/realms/demo
    registration:
      spring-with-test-scope:
        provider: keycloak
        client-id: spring-with-test-scope
        client-secret: IWLSnakHG8aNTWNaWuSj0a11UY4lzxd9
        authorization-grant-type: authorization_code
        scope: openid

The gateway exposes a single HTTP endpoint by itself. It uses OAuth2AuthorizedClient bean to return the current JWT access token.

@SpringBootApplication
@RestController
public class GatewayApplication {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(GatewayApplication.class);

   public static void main(String[] args) {
      SpringApplication.run(GatewayApplication.class, args);
   }

   @GetMapping(value = "/token")
   public Mono<String> getHome(@RegisteredOAuth2AuthorizedClient OAuth2AuthorizedClient authorizedClient) {
      return Mono.just(authorizedClient.getAccessToken().getTokenValue());
   }

}

That’s all about OAuth2 configuration in that section. We also need to configure routing on the gateway in the Spring application.yml file. Spring Cloud Gateway can forward OAuth2 access tokens downstream to the services it is proxying using the TokenRelay GatewayFilter. It is possible to set it as a default filter for all incoming requests. Our gateway forwards traffic to both our callme and caller microservices. I’m not using any service discovery in that scenario. By default, the callme app listens on the 8040 port, while the caller app on the 8020 port.

spring:
  application:
    name: gateway
  cloud:
    gateway:
      default-filters:
        - TokenRelay=
      routes:
        - id: callme-service
          uri: http://localhost:8040
          predicates:
            - Path=/callme/**
        - id: caller-service
          uri: http://localhost:8020
          predicates:
            - Path=/caller/**

Verify Tokens in Microservices with OAuth2 Resource Server

The list of dependencies for the callme and caller is pretty similar. They are exposing HTTP endpoints using the Spring Web module. Since the caller app uses the WebClient bean we also need to include the Spring WebFlux dependency. Once again, we need to include the Spring OAuth2 Resource Server module and the spring-security-oauth2-jose dependency for decoding JWT tokens.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-resource-server</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-jose</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-webflux</artifactId>
</dependency>

Here’s the configuration of the app security. This time we need to use the @EnableWebSecurity annotation since we have a Spring Web module. The oauth2ResourceServer() method verifies an access token with the Keyclock JWKS endpoint.

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http.authorizeHttpRequests(authorize -> authorize.anyRequest().authenticated())
                .oauth2ResourceServer((oauth2) -> oauth2.jwt(Customizer.withDefaults()));
        return http.build();
    }
}

Here’s the OAuth2 Resource Server configuration for Keycloak in the Spring application.yml file:

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          jwk-set-uri: http://localhost:8080/realms/demo/protocol/openid-connect/certs

Let’s take a look at the implementation of the REST controller class. It is a single ping method. That method may be accessed only by the client with the TEST scope. It returns a list of assigned scopes taken from the Authentication bean.

@RestController
@RequestMapping("/callme")
public class CallmeController {

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        SecurityContext context = SecurityContextHolder.getContext();
        Authentication authentication = context.getAuthentication();
        return "Scopes: " + authentication.getAuthorities();
    }
}

This method can be invoked directly by the external client through the API gateway. However, also the caller app calls that endpoint inside its own “ping” endpoint implementation.

@RestController
@RequestMapping("/caller")
public class CallerController {

    private WebClient webClient;

    public CallerController(WebClient webClient) {
        this.webClient = webClient;
    }

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        SecurityContext context = SecurityContextHolder.getContext();
        Authentication authentication = context.getAuthentication();

        String scopes = webClient
                .get()
                .uri("http://localhost:8040/callme/ping")
                .retrieve()
                .bodyToMono(String.class)
                .block();
        return "Callme scopes: " + scopes;
    }
}

If the WebClient calls the endpoint exposed by the second microservice, it also has to propagate the bearer token. We can easily achieve it with the ServletBearerExchangeFilterFunction as shown below. Thanks to that Spring Security will look up the current Authentication and extract the AbstractOAuth2Token credential. Then, it will propagate that token in the Authorization header automatically.

@SpringBootApplication
public class CallerApplication {

    public static void main(String[] args) {
        SpringApplication.run(CallerApplication.class, args);
    }

    @Bean
    public WebClient webClient() {
        return WebClient.builder()
                .filter(new ServletBearerExchangeFilterFunction())
                .build();
    }
    
}

Testing with Running Applications

We can run all three Spring Boot apps using the same Maven command. Let’s begin with the gateway app:

$ cd oauth/gateway
$ mvn spring-boot:run

Once we run the first app, we can check out the logs if everything works fine. Here are the logs generated by the gateway app. As you see, it listens on the 8060 port.

After that, we can run e.g. the caller app.

$ cd oauth/caller
$ mvn spring-boot:run

It listens on the 8020 port.

Of course, the order of starting apps doesn’t matter. As the last one, we can run the callme app.

$ cd oauth/callme
$ mvn spring-boot:run

Now, let’s call the caller app endpoint through the gateway. In that case, we need to go to the http://localhost:8060/caller/ping URL. The gateway app will redirect us to the Keycloak login page. We need to sign in there with the spring user and Spring_123 password.

After we sign in, everything happens automatically. Spring Cloud Gateway obtains the access token from Keycloak and then sends it to the downstream service. Once the caller app receives the request, it invokes the callme app using the WebClient instance. Here’s the result:

We can easily get the access token using the endpoint GET /token exposed by the gateway app.

Now, we can perform a similar call as before, but with the curl command. We need to copy the token string and put it inside the Authorization header as a bearer token.

$ curl http://localhost:8060/callme/ping \
    -H "Authorization: Bearer <TOKEN>" -v

Here’s my result:

Now, let’s do a similar thing, but in a fully automated way with JUnit and Testcontainers.

Spring OAuth2 with Keycloak Testcontainers

We need to switch to the gateway module once again. We will implement tests that run the API gateway app, connect it to the Keycloak instance, and route the authorized traffic in the target endpoint. Here’s the @RestController in the src/test/java directory that simulates the callme app endpoint:

@RestController
@RequestMapping("/callme")
public class CallmeController {

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        return "Hello!";
    }
}

Here’s the required configuration to run the tests. We are starting the gateway app on the 8060 port and using the WebTestClient instance for calling it. In order to automatically configure Keycloak we will import the demo realm configuration stored in the realm-export.json. Since Testcontainers use random port numbers we need to override some Spring OAuth2 configuration settings. We also override the Spring Cloud Gateway route, to forward the traffic to our test implementation of the callme app controller instead of the real service. That’s all. We can proceed to the tests implementation.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class GatewayApplicationTests {

   static String accessToken;

   @Autowired
   WebTestClient webTestClient;

   @Container
   static KeycloakContainer keycloak = new KeycloakContainer()
            .withRealmImportFile("realm-export.json")
            .withExposedPorts(8080);

   @DynamicPropertySource
   static void registerResourceServerIssuerProperty(DynamicPropertyRegistry registry) {
      registry.add("spring.security.oauth2.client.provider.keycloak.issuer-uri",
                () -> keycloak.getAuthServerUrl() + "/realms/demo");
      registry.add("spring.security.oauth2.resourceserver.jwt.jwk-set-uri",
                () -> keycloak.getAuthServerUrl() + "/realms/demo/protocol/openid-connect/certs");
      registry.add("spring.cloud.gateway.routes[0].uri",
                () -> "http://localhost:8060");
      registry.add("spring.cloud.gateway.routes[0].id", () -> "callme-service");
      registry.add("spring.cloud.gateway.routes[0].predicates[0]", () -> "Path=/callme/**");
   }

   // TEST IMPLEMENTATION ...

}

Here’s our first test. Since it doesn’t contain any token it should be redirected into the Keycloak authorization mechanism.

@Test
@Order(1)
void shouldBeRedirectedToLoginPage() {
   webTestClient.get().uri("/callme/ping")
             .exchange()
             .expectStatus().is3xxRedirection();
}

In the second test, we use the WebClient instance to interact with the Keycloak container. We need to authenticate against Kecloak with the spring user and the spring-with-test-scope client. Keycloak will generate and return an access token. We will save its value for the next test.

@Test
@Order(2)
void shouldObtainAccessToken() throws URISyntaxException {
   URI authorizationURI = new URIBuilder(keycloak.getAuthServerUrl() + "/realms/demo/protocol/openid-connect/token").build();
   WebClient webclient = WebClient.builder().build();
   MultiValueMap<String, String> formData = new LinkedMultiValueMap<>();
   formData.put("grant_type", Collections.singletonList("password"));
   formData.put("client_id", Collections.singletonList("spring-with-test-scope"));
   formData.put("username", Collections.singletonList("spring"));
   formData.put("password", Collections.singletonList("Spring_123"));

   String result = webclient.post()
                .uri(authorizationURI)
                .contentType(MediaType.APPLICATION_FORM_URLENCODED)
                .body(BodyInserters.fromFormData(formData))
                .retrieve()
                .bodyToMono(String.class)
                .block();
   JacksonJsonParser jsonParser = new JacksonJsonParser();
   accessToken = jsonParser.parseMap(result)
                .get("access_token")
                .toString();
   assertNotNull(accessToken);
}

Finally, we run a similar test as in the first step. However, this time, we provide an access token inside the Authorization header. The expected response is 200 OK and the “Hello!” payload, which is returned by the test instance of the CallmeController bean.

@Test
@Order(3)
void shouldReturnToken() {
   webTestClient.get().uri("/callme/ping")
                .header("Authorization", "Bearer " + accessToken)
                .exchange()
                .expectStatus().is2xxSuccessful()
                .expectBody(String.class).isEqualTo("Hello!");
}

Let’s run all the tests locally. As you see, they are all successfully finished.

Final Thoughts

After publishing my previous article about Spring Cloud Gateway and Keycloak I received a lot of comments and questions with a request for some clarifications. I hope that this article answers some of them. We focused more on automation and service-to-service communication than just on the OAuth2 support in the Spring Cloud Gateway. We considered a case where a gateway acts as the OAuth2 client and resource server at the same time. Finally, we used Testcontainers to verify our scenario with Spring Cloud Gateway and Keycloak.

The post Microservices with Spring Cloud Gateway, OAuth2 and Keycloak appeared first on Piotr's TechBlog.

It is not my first article about Azure and Spring Cloud. As a preparation for that exercise, it is worth reading the article to familiarize yourself with some interesting features of Spring Cloud Azure. It describes an integration with Azure Spring Apps, Cosmos DB, and App Configuration services. On the other hand, if you are interested in CI/CD for Spring Boot apps you can refer to the following article about Azure DevOps and Terraform.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. The Spring Boot apps used in the article are located in the serverless directory. After you go to that directory you should just follow my further instructions.

Architecture

In this exercise, we will prepare two sample Spring Boot apps (aka functions) account-function and customer-function. Then, we will deploy them on the Azure Function service. Our apps do not communicate with each other directly but through the Azure Event Hubs service. Event Hubs is a cloud-native data streaming service compatible with the Apache Kafka API. After adding a new customer the customer-function sends an event to the Azure Event Hubs using the Spring Cloud Stream binder. The account-function app receives the event through the Azure Event Hubs trigger. Also, the customer-function is exposed to the external client through the Azure HTTP trigger. Here’s the diagram of our architecture:

Prerequisites

There are some prerequisites before you start the exercise. You need to install JDK17+ and Maven on your local machine. You also need to have an account on Azure and az CLI to interact with that account. Once you install the az CLI and log in to Azure you can execute the following command for verification:

$ az account show

If you would like to test Azure Functions locally, you need to install Azure Functions Core Tools. You can find detailed installation instructions in Microsoft Docs here. For macOS, there are three required commands to run:

$ brew tap azure/functions
$ brew install azure-functions-core-tools@4
$ brew link --overwrite azure-functions-core-tools@4

Create Resources on Azure

Before we proceed with the source code, we need to create several required resources on the Azure cloud. In the first step, we will prepare a resource group for all required objects. The name of the group is spring-cloud-serverless. The location depends on your preferences. For me it is eastus.

$ az group create -l eastus -n spring-cloud-serverless

In the next step, we need to create a storage account. The Azure Function service requires it, but we will also use that account during the local development with Azure Functions Core Tools.

$ az storage account create -n pminkowsserverless \
     -g spring-cloud-serverless \
     -l eastus \
     --sku Standard_LRS

In order to run serverless apps on Azure with e.g. Spring Cloud Function, we need to create the Azure Function App instances. Of course, we use the previously created resource group and storage account. The name of my Function App instances are pminkows-account-function and pminkows-customer-function. We can also set a default OS type (Linux), functions version (4), and a runtime stack (Java) for each Function App.

$ az functionapp create -n pminkows-customer-function \
     -c eastus \
     --os-type Linux \
     --functions-version 4 \
     -g spring-cloud-serverless \
     --runtime java \
     --runtime-version 17.0 \
     -s pminkowsserverless

$ az functionapp create -n pminkows-account-function \
     -c eastus \
     --os-type Linux \
     --functions-version 4 \
     -g spring-cloud-serverless \
     --runtime java \
     --runtime-version 17.0 \
     -s pminkowsserverless

Then, we have to create the Azure Event Hubs namespace. The name of my namespace is spring-cloud-serverless. The same as before I choose the East US location and the spring-cloud-serverless resource group. We can also set the pricing tier (Standard) and the upper limit of throughput units when the AutoInflate option is enabled.

$ az eventhubs namespace create -n spring-cloud-serverless \
     -g spring-cloud-serverless \
     --location eastus \
     --sku Standard \
     --maximum-throughput-units 1 \
     --enable-auto-inflate true

Finally, we have to create topics on Event Hubs. Of course, they have to be assigned to the previously created spring-cloud-serverless Event Hubs namespace. The names of our topics are accounts and customers. The number of partitions is irrelevant in this exercise.

$ az eventhubs eventhub create -n accounts \
     -g spring-cloud-serverless \
     --namespace-name spring-cloud-serverless \
     --cleanup-policy Delete \
     --partition-count 3

$ az eventhubs eventhub create -n customers \
     -g spring-cloud-serverless \
     --namespace-name spring-cloud-serverless \
     --cleanup-policy Delete \
     --partition-count 3

Now, let’s switch to the Azure Portal. Find the spring-cloud-serverless resource group. You should have the same list of resources inside this group as shown below. It means that our environment is ready and we can proceed to the source code.

App Dependencies

Firstly, we need to declare the dependencyManagement section inside the Maven pom.xml for three projects used in the app implementation: Spring Boot, Spring Cloud, and Spring Cloud Azure.

<properties>
  <java.version>17</java.version>
  <spring-boot.version>3.1.4</spring-boot.version>
  <spring-cloud-azure.version>5.7.0</spring-cloud-azure.version>
  <spring-cloud.version>2022.0.4</spring-cloud.version>
  <maven.compiler.release>${java.version}</maven.compiler.release>
  <maven.compiler.source>${java.version}</maven.compiler.source>
  <maven.compiler.target>${java.version}</maven.compiler.target>
</properties>

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>${spring-cloud.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    <dependency>
      <groupId>com.azure.spring</groupId>
      <artifactId>spring-cloud-azure-dependencies</artifactId>
      <version>${spring-cloud-azure.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-dependencies</artifactId>
      <version>${spring-boot.version}</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

<build>
  <plugins>
    <plugin>
      <groupId>org.springframework.boot</groupId>
       <artifactId>spring-boot-maven-plugin</artifactId>
      <executions>
       <execution>
          <goals>
           <goal>repackage</goal>
          </goals>
         </execution>
       </executions>
    </plugin>
  </plugins>
</build>

Here’s the list of required Maven dependencies. We need to include the spring-cloud-function-context library to enable Spring Cloud Functions. In order to integrate with the Azure Functions service, we need to include the spring-cloud-function-adapter-azure extension. Our apps also send messages to the Azure Event Hubs service through the dedicated Spring Cloud Stream binder.

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-function-adapter-azure</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-stream-binder-eventhubs</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-function-context</artifactId>
  </dependency>
</dependencies>

Create Azure Serverless Apps with Spring Cloud

Expose Azure Function as HTTP endpoint

Let’s begin with the customer-function. To simplify the app we will use an in-memory H2 for storing data. Each time a new customer is added, it is persisted in the in-memory database using Spring Data JPA extension. Here’s our entity class:

@Entity
public class Customer {
   @Id
   @GeneratedValue(strategy = GenerationType.IDENTITY)
   private Long id;
   private String name;
   private int age;
   private String status;

   // GETTERS AND SETTERS
}

Here’s the Spring Data repository interface for the Customer entity:

public interface CustomerRepository extends ListCrudRepository<Customer, Long> {
}

Let’s proceed to the Spring Cloud Functions implementation. The CustomerInternalFunctions bean defines two functions. The addConsumer function (1) persists a new customer in the database and sends the event about it to the Azure Event Hubs topic. It uses the Spring Cloud Stream StreamBridge bean for interacting with Event Hubs. It also returns the persisted Customer entity as a response. The second function changeStatus (2) doesn’t return any data, it just needs to react to the incoming event and change the status of the particular customer. That event is sent by the account-function app after generating an account for a newly created customer. The important information here is that those are just Spring Cloud functions. For now, they have nothing to do with Azure Functions.

@Service
public class CustomerInternalFunctions {

   private static final Logger LOG = LoggerFactory
      .getLogger(CustomerInternalFunctions.class);

   private StreamBridge streamBridge;
   private CustomerRepository repository;

   public CustomerInternalFunctions(StreamBridge streamBridge, 
                                    CustomerRepository repository) {
      this.streamBridge = streamBridge;
      this.repository = repository;
   }

   // (1)
   @Bean
   public Function<Customer, Customer> addCustomer() {
      return c -> {
         Customer newCustomer = repository.save(c);
         streamBridge.send("customers-out-0", newCustomer);
         LOG.info("New customer added: {}", c);
         return newCustomer;
      };
   }

   // (2)
   @Bean
   public Consumer<Account> changeStatus() {
      return account -> {
         Customer customer = repository.findById(account.getCustomerId())
                .orElseThrow();
         customer.setStatus(Customer.CUSTOMER_STATUS_ACC_ACTIVE);
         repository.save(customer);
         LOG.info("Customer activated: id={}", customer.getId());
      };
   }

}

We also need to provide several configuration properties in the Spring Boot application.properties file. We should set the Azure Event Hubs connection URL and the name of the target topic. Since our app uses Spring Cloud Stream only for sending events we should also turn off the autodiscovery of functional beans as messaging bindings.

spring.cloud.azure.eventhubs.connection-string = ${EVENT_HUBS_CONNECTION_STRING}
spring.cloud.stream.bindings.customers-out-0.destination = customers
spring.cloud.stream.function.autodetect = false

In order to expose the functions on Azure, we will take advantage of the Spring Cloud Function Azure Adapter. It includes several Azure libraries to our Maven dependencies. It can invoke the Spring Cloud Functions directly or through the lookup approach with the FunctionCatalog bean (1). The method has to be annotated with @FunctionName, which defines the name of the function in Azure (2). In order to expose that function over HTTP, we need to define the Azure @HttpTrigger (3). The trigger exposes the function as the POST endpoint and doesn’t require any authorization. Our method receives the request through the HttpRequestMessage object. Then, it invokes the Spring Cloud function addCustomer by name using the FunctionCatalog bean (4).

// (1)
@Autowired
private FunctionCatalog functionCatalog;

@FunctionName("add-customer") // (2)
public Customer addCustomerFunc(
        // (3)
        @HttpTrigger(name = "req",
                     methods = { HttpMethod.POST },
                     authLevel = AuthorizationLevel.ANONYMOUS)
        HttpRequestMessage<Optional<Customer>> request,
        ExecutionContext context) {
   Customer c = request.getBody().orElseThrow();
   context.getLogger().info("Request: {}" + c);
   // (4)
   Function<Customer, Customer> function = functionCatalog
      .lookup("addCustomer");
   return function.apply(c);
}

Integrate Function with Azure Data Hubs Trigger

Let’s switch to the account-function app directory inside our Git repository. The same as customer-function it uses an in-memory H2 database for storing customer accounts. Here’s our Account entity class:

@Entity
public class Account {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String number;
    private Long customerId;
    private int balance;

   // GETTERS AND SETTERS ...
}

Inside the account-function app, there is a single Spring Cloud Function addAccount. It generates a new 16-digit account number for each new customer. Then, it saves the account in the database and sends the event to the Azure Event Hubs with Spring Cloud Stream StreamBridge bean.

@Service
public class AccountInternalFunctions {

   private static final Logger LOG = LoggerFactory
      .getLogger(AccountInternalFunctions.class);

   private StreamBridge streamBridge;
   private AccountRepository repository;

   public AccountInternalFunctions(StreamBridge streamBridge, 
                                   AccountRepository repository) {
      this.streamBridge = streamBridge;
      this.repository = repository;
   }

   @Bean
   public Function<Customer, Account> addAccount() {
      return customer -> {
         String n = RandomStringUtils.random(16, false, true);
         Account a = new Account(n, customer.getId(), 0);
         a = repository.save(a);
         boolean b = streamBridge.send("accounts-out-0", a);
         LOG.info("New account added: {}", a);
         return a;
      };
   }

}

The same as for customer-function we need to provide some configuration settings for Spring Cloud Stream. However, instead of the customers topic, this time we are sending events to the accounts topic.

spring.cloud.azure.eventhubs.connection-string = ${EVENT_HUBS_CONNECTION_STRING}
spring.cloud.stream.bindings.accounts-out-0.destination = accounts
spring.cloud.stream.function.autodetect = false

The account-function app is not exposed as the HTTP endpoint. We want to trigger the function in reaction to the event delivered to the Azure Event Hubs topic. The name of our Azure function is new-customer (1). It receives the Customer event from the customers topic thanks to the @EventHubTrigger annotation (2). This annotation also defines a property name, that contains the address of the Azure Event Hubs namespace (EVENT_HUBS_CONNECTION_STRING). I’ll show you later how to set such a property for our function in Azure. Once, the new-customer function is triggered, it invokes the Spring Cloud Function addAccount (3).

@Autowired
private FunctionCatalog functionCatalog;

// (1)
@FunctionName("new-customer")
public void newAccountEventFunc(
        // (2)
        @EventHubTrigger(eventHubName = "customers",
                         name = "newAccountTrigger",
                         connection = "EVENT_HUBS_CONNECTION_STRING",
                         cardinality = Cardinality.ONE)
        Customer event,
        ExecutionContext context) {
   context.getLogger().info("Event: " + event);
   // (3)
   Function<Customer, Account> function = functionCatalog
      .lookup("addAccount");
   function.apply(event);
}

At the end of this section, let’s switch back once again to the customer-function app. It fires on the event sent by the new-customer function to the accounts topic. Therefore we are using the @EventHubTrigger annotation once again.

@FunctionName("activate-customer")
public void activateCustomerEventFunc(
          @EventHubTrigger(eventHubName = "accounts",
                 name = "changeStatusTrigger",
                 connection = "EVENT_HUBS_CONNECTION_STRING",
                 cardinality = Cardinality.ONE)
          Account event,
          ExecutionContext context) {
   context.getLogger().info("Event: " + event);
   Consumer<Account> consumer = functionCatalog.lookup("changeStatus");
   consumer.accept(event);
}

All our functions are ready. Now, we can proceed to the deployment phase.

Running Azure Functions Locally with Maven

Before we deploy our functions on Azure, we can run and test them locally. I assume you have already the Azure Functions Core Tools according to the “Prerequisites” section. Our function still needs to connect with some services on the cloud like Azure Event Hubs or the storage account. The address to the Azure Event Hubs should be set as the EVENT_HUBS_CONNECTION_STRING app property or environment variable. In order to find the Event Hubs connection string, we should switch to the Azure Portal and find the spring-cloud-serverless namespace. Then, we need to go to the “Shared access policies” menu item and click the “RootManagedSharedAccessKey” policy. The connection string value is available in the “Connection string-primary key” field.

We also need to obtain the connection credentials to the Azure storage account. In your storage account, you should find the “Access keys” section and copy the value from the “Connection string” field.

After that, we can create the local.settings.json file in each app’s root directory. Alternatively, we can just set the environment variables AzureWebJobsStorage and EVENT_HUBS_CONNECTION_STRING.

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": <YOUR_ACCOUNT_STORAGE_CONNECTION_STRING>,
    "FUNCTIONS_WORKER_RUNTIME": "java",
    "EVENT_HUBS_CONNECTION_STRING": <YOUR_EVENT_HUBS_CONNECTION_STRING>
  }
}

Once you place your credentials in the local.settings.json file, you can build the app with Maven.

$ mvn clean package

After that, you can use the Maven plugin included in the spring-cloud-function-adapter-azure module. In order to run the function, you need to execute the following command:

$ mvn azure-functions:run 

Here’s the command output for the customer-function app. As you see, it contains two Azure functions: add-customer (HTTP) and activate-customer (Event Hub Trigger). You can test the function by invoking the http://localhost:7071/api/add-customer URL.

Here’s the command output for the account-function app. As you see, it contains a single function new-customer activated through the Event Hub trigger.

Deploy Spring Cloud Serverless on Azure Functions

Let’s take a look at the pminkows-customer-function Azure Function App before we deploy our first app there. The http://pminkows-customer-function.azurewebsites.net base URL will precede all the HTTP endpoint URLs for our functions. We should remember the name of the automatically generated service plan (EastUSLinuxDynamicPlan).

Before deploying our Spring Boot app we should create the host.json file with the following content:

{
  "version": "2.0",
  "extensionBundle": {
    "id": "Microsoft.Azure.Functions.ExtensionBundle",
    "version": "[4.*, 5.0.0)"
  }
}

Then, we should add the azure-functions-maven-plugin Maven plugin to our pom.xml. In the configuration section, we have to set the name of the Azure Function App instance (pminkows-customer-function), the target resource group (spring-cloud-serverless), the region (eastus), the service plan (EastUSLinuxDynamicPlan), and the location of the host.json file. We also need to set the connection string to the Azure Event Hubs inside the EVENT_HUBS_CONNECTION_STRING app property. So before running the build, you should export the value of that environment variable.

<plugin>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>azure-functions-maven-plugin</artifactId>
  <version>1.30.0</version>
  <configuration>
    <appName>pminkows-customer-function</appName>
    <resourceGroup>spring-cloud-serverless</resourceGroup>
    <region>eastus</region>
    <appServicePlanName>EastUSLinuxDynamicPlan</appServicePlanName>
    <hostJson>${project.basedir}/src/main/resources/host.json</hostJson>
    <runtime>
      <os>linux</os>
      <javaVersion>17</javaVersion>
    </runtime>
    <appSettings>
      <property>
        <name>FUNCTIONS_EXTENSION_VERSION</name>
        <value>~4</value>
      </property>
      <property>
        <name>EVENT_HUBS_CONNECTION_STRING</name>
        <value>${EVENT_HUBS_CONNECTION_STRING}</value>
      </property>
    </appSettings>
  </configuration>
  <executions>
    <execution>
      <id>package-functions</id>
      <goals>
        <goal>package</goal>
      </goals>
    </execution>
  </executions>
</plugin>

Let’s begin with the customer-function app. Before deploying the app we need to build it first with the mvn clean package command. Once you do it you can run your first function Azure with the following command:

$ mvn azure-functions:deploy

Here’s my output after running that command. As you see, the function add-customer is exposed under the http://pminkows-customer-function.azurewebsites.net/api/add-customer URL.

We can come back to the pminkows-customer-function Azure Function App in the portal. According to the expectations, two functions are running there. The activate-customer function is triggered by the Azure Event Hub.

Let’s switch to the account-function directory. We will deploy it to the Azure pminkows-account-function function. Once again we need to build the app with mvn clean package command, and then deploy it using the mvn azure-functions:deploy command. Here’s the output. There are no HTTP triggers defined, but just a single function triggered by the Azure Event Hub.

Here are the details of the pminkows-account-function Function App in the Azure Portal.

Invoke Azure Functions

Finally, we can test our functions by calling the following endpoint using e.g. curl. Let’s repeat the similar command several times with different data:

$ curl https://pminkows-customer-function.azurewebsites.net/api/add-customer \ 
    -d "{\"name\":\"Test\",\"age\":33}" \
    -H "Content-Type: application/json"

After that, we should switch to the Azure Portal. Go to the pminkows-customer-function details and click the link “Invocation and more” on the add-customer function.

You will be redirected to the Azure Monitor statistics for that function. Azure Monitor displays a list of invocations with statuses.

We can click one of the records from the invocation history to see the details.

As you probably remember, the add-customer function sends messages to Azure Event Hubs. On the other hand, we can also verify how the new-customer function in the pminkows-account-function consumes and handles those events.

Final Thoughts

This article gives you a comprehensive guide on how to build and run Spring Cloud serverless apps on Azure Functions. It explains the concept of the triggers in Azure Functions and shows the integration with Azure Event Hubs. Finally, it shows how to run such functions locally and then monitor them on Azure after deployment.

The post Serverless on Azure with Spring Cloud Function appeared first on Piotr's TechBlog.

If you need an introduction to Spring Cloud read my article about microservices with Spring Boot 3 and Spring Cloud available here. It is worth at least taking a look at the Spring Cloud Azure docs for a basic understanding of the main concepts.

Architecture

Our architecture is pretty simple. As I mentioned before, we have a single Spring Boot app (account-service in the diagram) that runs on Azure and connects to Cosmos DB. It exposes some REST endpoints for adding, deleting, or searching accounts backed by Cosmos DB. It also stores the whole required configuration (like Cosmos DB address and credentials) in the Azure App Configuration service. The app is managed by the Azure Spring Apps service. Here’s the diagram illustrating our architecture.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. The Spring Boot app used in the article is located in the microservices/account-service directory. After you go to that directory you should just follow my further instructions.

There are some prerequisites before you start the exercise. You need to install JDK17+ and Maven on your local machine. You also need to have an account on Azure and az CLI to interact with that account. In order to deploy the app on Azure we will use azure-spring-apps-maven-plugin, which requires az CLI.

Dependencies

Firstly, let’s take a look at the list of required Maven dependencies. Of course, we need to add the Spring Boot Web starter to enable REST support through the Spring MVC module. In order to integrate with Cosmos DB, we will use the Spring Data repositories. Spring Cloud Azure provides a dedicated starter spring-cloud-azure-starter-data-cosmos for it. The spring-cloud-azure-starter-actuator module is optional. It will enable a health indicator for Cosmos DB in the /actuator/health endpoint. After that, we will include the starter providing integration with the Azure App Configuration service. Finally, we can add the Springdoc OpenAPI project responsible for generating REST API documentation.

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter-data-cosmos</artifactId>
  </dependency>
  <dependency>
    <groupId>com.azure.spring</groupId>
    <artifactId>spring-cloud-azure-starter-appconfiguration-config</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.2.0</version>
  </dependency>
</dependencies>

Spring Cloud with Azure Cosmos DB

After including the Spring Data module with Cosmos DB support we may define the model class. The Account class contains three String fields: id (primary key), number, and customerId (partition key). The partition key is responsible for dividing data into distinct subsets called logical partitions. The model must be annotated with @Container. The containerName parameter inside the annotation corresponds to the name of the Cosmos DB container created in Azure.

@Container(containerName = "accounts")
public class Account {
   @Id
   @GeneratedValue
   private String id;
   private String number;
   @PartitionKey
   private String customerId;

   // GETTERS AND SETTERS ...
}

Now, let’s prepare our environment in Azure. After logging in with the az login CLI command we create the resource group for our exercise. The name of the group is sample-spring-cloud. The location depends on your preferences. For me it is eastus.

$ az group create -l eastus -n sample-spring-cloud

Then, we are going to create a new Azure Cosmos DB database account. The name of my account is sample-pminkows-cosmosdb. It is placed inside our sample-spring-cloud resource group. I’ll leave the default values in all other parameters. But you can consider overriding some parameters to decrease the instance cost. For example, we can set the Local backup redundancy type using the --backup-redundancy parameter.

$ az cosmosdb create -n sample-pminkows-cosmosdb -g sample-spring-cloud

Once we enable a database account we can create a database instance. The name of our database is sampled. Of course, it has to be placed in the previously created sample-pminkows-cosmosdb Cosmos DB account.

$ az cosmosdb sql database create \
    -a sample-pminkows-cosmosdb \
    -n sampledb \
    -g sample-spring-cloud

Finally, we need to create a container inside our database. The name of the container should be the same as the value of the containerName field declared in the model class. We also have to set the partition key path. As you probably remember, we are using the customerId field in the Account class for that.

$ az cosmosdb sql container create \
    -a sample-pminkows-cosmosdb \
    -g sample-spring-cloud \
    -n accounts \
    -d sampledb \
    -p /customerId

Everything is ready on the Azure side. Let’s back for a moment to the source code. In order to interact with the database, we will create the Spring Data repository interface. It has to extend the CosmosRepository interface provided within Spring Cloud Azure. It defines one additional method for searching by the customerId field.

public interface AccountRepository extends CosmosRepository<Account, String> {
   List<Account> findByCustomerId(String customerId);
}

Finally, we can create @RestController with the endpoints implementation. It injects and uses the AccountRepository bean.

@RestController
@RequestMapping("/accounts")
public class AccountController {

   private final static Logger LOG = LoggerFactory
      .getLogger(AccountController.class);
   private final AccountRepository repository;

   public AccountController(AccountRepository repository) {
      this.repository = repository;
   }

   @PostMapping
   public Account add(@RequestBody Account account) {
      LOG.info("add: {}", account.getNumber());
      return repository.save(account);
   }

   @GetMapping("/{id}")
   public Account findById(@PathVariable String id) {
      LOG.info("findById: {}", id);
      return repository.findById(id).orElseThrow();
   }

   @GetMapping
   public List<Account> findAll() {
      List<Account> accounts = new ArrayList<>();
      repository.findAll().forEach(accounts::add);
      return accounts;
   }

   @GetMapping("/customer/{customerId}")
   public List<Account> findByCustomerId(@PathVariable String customerId) {
      LOG.info("findByCustomerId: {}", customerId);
      return repository.findByCustomerId(customerId);
   }
}

Azure App Configuration with Spring Cloud

Once we finish the app implementation, we can run it and connect with Cosmos DB. Of course, we need to set the connection URL and credentials. Let’s switch to the Azure Portal. We need to find the “Azure Cosmos DB” service in the main menu. Then click your database account. You will see the address of the endpoint as shown below. You should also see the previously created container in the “Containers” section.

In order to obtain the connection key, we need to go to the “Data Explorer” item in the left-side menu. Then choose the “Connect” tile. You will find the key in the target window.

We could easily set all the required connection parameters using the spring.cloud.azure.cosmos.* properties. However, I would like to store all the configuration settings on Azure. Spring Cloud comes with built-in support for Azure App Configuration service. We have already included the required Spring Cloud starter. So now, we need to enable the Azure App Configuration service and put our properties into the store. Here’s the command for creating an App Configuration under the sample-spring-cloud-config name:

$ az appconfig create \
    -g sample-spring-cloud \
    -n sample-spring-cloud-config \
    -l eastus \
    --sku Standard

Once we create the App Configuration we can put our configuration settings in the key/value form. By default, Spring Cloud Azure is loading configurations that start with the key /application/. We need to add three Spring Cloud properties: spring.cloud.azure.cosmos.key, spring.cloud.azure.cosmos.database, and spring.cloud.azure.cosmos.endpoint.

$ az appconfig kv set \
    -n sample-spring-cloud-config \
    --key /application/spring.cloud.azure.cosmos.key \
    --value <YOUR_PRIMARY_KEY>

$ az appconfig kv set \
    -n sample-spring-cloud-config \
    --key /application/spring.cloud.azure.cosmos.database \
    --value sampledb

$ az appconfig kv set \
    -n sample-spring-cloud-config \
    --key /application/spring.cloud.azure.cosmos.endpoint \
    --value <YOUR_ENDPOINT_URI>

Let’s switch to the Azure Portal to check the configuration settings. We need to find the “App Configuration” service in the main dashboard. Then go to the sample-spring-cloud-config details and choose the “Configuration explorer” menu item. You should have all your application properties prefixed by the /application/. I also overrode some Spring Actuator settings to enable health check details and additional management endpoints.

That’s all. Now, we are ready to run our app. We just need to connect it to the Azure App Configuration service. In order to do that, we need to obtain its connection endpoint and credentials. You can go to the “Access keys” menu item in the “Settings” section. Then you should copy the value from the “Connection string” field as shown below. Alternatively, you can obtain the same information by executing the following CLI command: az appconfig credential list --name sample-spring-cloud-config.

Let’s save the value inside the APP_CONFIGURATION_CONNECTION_STRING environment variable. After that, we just need to create the Spring bootstrap.properties file in the src/main/resources directory containing the spring.cloud.azure.appconfiguration.stores[0].connection-string property.

spring.cloud.azure.appconfiguration.stores[0].connection-string=${APP_CONFIGURATION_CONNECTION_STRING}

Running Spring Boot App Locally

Finally, we can run our sample Spring Boot app. For now, we will just run it locally. As a result, it will connect to the Azure App Configuration and Cosmos DB deployed on the cloud. We can execute the following Maven command to start the app:

$ mvn clean spring-boot:run

Once you start the app you should see that it loads property sources from the Azure store:

If everything works fine your app is loading settings from Azure App Configuration and connects to the Cosmos DB instance:

Once you start the app you can access it under the 8080 local port. The Swagger UI is available under the /swagger-ui.html path:

We can some data using e.g. the curl command as shown below:

$ curl -X 'POST' 'http://localhost:8080/accounts' \
    -H 'Content-Type: application/json' \
    -d '{"number": "1234567893","customerId": "1"}'
{"id":"5301e9dd-0556-40b7-9ea3-96975492f00c","number":"1234567893","customerId":"1"}

Then, we can e.g. find accounts owned by a particular customer:

$ curl http://localhost:8080/accounts/customer/1

We can also delete an existing account by calling the DELETE /account/{id} endpoint. In that case, I received the HTTP 404 Not Found error. Interesting?

Let’s see what happened. If you take a look at the implementation of AccountController you will find the method for the DELETE endpoint, right? In the meantime, I added one method annotated with @FeatureGate. This annotation is provided by Spring Cloud Azure. The following fragment of code shows the usage of feature management with Azure App Configuration. In fact, I’m using the “Feature Gate” functionality, which allows us to call the endpoint only if a feature is enabled on the Azure side. The name of our feature is delete-account.

@DeleteMapping("/{id}")
@FeatureGate(feature = "delete-account")
public void deleteById(@PathVariable String id) {
   repository.deleteById(id);
}

Now, the only thing we need to do is to add a new feature to the sample-spring-cloud-config App Configuration.

$ az appconfig feature set -n sample-spring-cloud-config --feature test-2

Let’s switch to the Azure Portal. You should go to the “Feature manager” menu item in the “Operations” section. As you see, by default the feature flag is disabled. It means the feature is not active and the endpoint is disabled.

You can enable the feature by clicking the checkbox button and then restart the app. After that, the DELETE endpoint should be available.

Deploy Spring Cloud App on Azure

We can deploy our sample app to Azure in several different ways. I’ll choose the service dedicated especially to Spring Boot – Azure Spring Apps.

The installation from Azure Portal is pretty straightforward. I won’t get into the details. The name of our instance (cluster) is sample-spring-cloud-apps. We don’t need to know anything more to be able to deploy our app there.

Azure provides several Maven plugins for deploying apps. For Azure Spring Apps we should use azure-spring-apps-maven-plugin. We need to set the Azure Spring Apps instance in the clusterName parameter. The name of our app is account-service. We should also choose SKU and set the Azure subscription ID (loaded from the SUBSCRIPTION environment variable). In the deployment section, we need to define the required resources (RAM and CPU), number of running instances, Java version, and a single environment variable containing the connection string to the Azure App Configuration instance.

<plugin>
  <groupId>com.microsoft.azure</groupId>
  <artifactId>azure-spring-apps-maven-plugin</artifactId>
  <version>1.19.0</version>
  <configuration>
    <subscriptionId>${env.SUBSCRIPTION}</subscriptionId>
    <resourceGroup>sample-spring-cloud</resourceGroup>
    <clusterName>sample-spring-cloud-apps</clusterName>
    <sku>Consumption</sku>
    <appName>account-service</appName>
    <isPublic>true</isPublic>
    <deployment>
      <cpu>0.5</cpu>
      <memoryInGB>1</memoryInGB>
      <instanceCount>1</instanceCount>
      <runtimeVersion>Java 17</runtimeVersion>
      <environment>
        <APP_CONFIGURATION_CONNECTION_STRING>
          ${env.APP_CONFIGURATION_CONNECTION_STRING}
        </APP_CONFIGURATION_CONNECTION_STRING>
      </environment>
      <resources>
        <resource>
          <directory>target/</directory>
          <includes>
            <include>*.jar</include>
          </includes>
        </resource>
      </resources>
    </deployment>
  </configuration>
</plugin>

Then we need to build the app and deploy it on Azure Spring Apps with the following command:

$ mvn clean package azure-spring-apps:deploy

You should have a similar result as shown below:

Does the name of the instance sound familiar? Under the hood it’s Kubernetes. The Azure Spring Apps service uses Azure Container Apps for running containers. On the other hand, Azure Container Apps is hosted on the Kubernetes cluster. But these are the details. What is important here – our app has already been deployed on Azure.

We can display the account-service app details. The app is exposed under the public URL. We just need to copy the link.

Let’s take a look at the configuration section. As you see it contains the connection string to the App Configuration endpoint.

We can display the Swagger UI and perform some test calls.

Final Thoughts

That’s all in this article, but I’m planning to create several others about Spring Boot and Azure soon! Azure seems to be a friendly platform for the Spring Boot developer I showed you how to easily integrate your Spring Boot app with the most popular Azure services like Cosmos DB. We also covered such topics as configuration management and feature flags (gates) with the App Configuration service. Finally, we deployed the app on… Kubernetes through the Azure Spring Apps service

The post Getting Started with Spring Cloud Azure appeared first on Piotr's TechBlog.

If you are looking for a quick intro to the Protocol Buffers with Java you can read my article available here. It was written more than 5 years ago. However, I updated it in recent days. I also updated the repository with code examples to the latest Spring Boot 3 and Java 17.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. It contains four apps. Two of them account-service and customer-service are related to my previous article, which introduces Protocol Buffers with Java. For the current article, please refer to another two apps account-service-grpc and customer-service-grpc. They are pretty similar to the corresponding apps but use our third-party Spring Boot and gRPC communication instead of REST. Also, they need to use Spring Boot 2, because our third-party starter still doesn’t support Spring Boot 3. Anyway, once you clone the repository just follow my instructions

Generate Model Classes and Services for gRPC

In the first step, we will generate model classes and gRPC services using the .proto manifests. We need to include some additional Protobuf schemas to use the google.protobuf.* package (1). Our gRPC service will provide methods for searching accounts using various criteria and a single method for adding a new account (2). Those methods will use primitives from the google.protobuf.* package and model classes defined inside the .proto file as messages. There are two messages defined. The Account message represents a single model class. It contains three fields: id, number, and customer_id (3). The Accounts message contains a list of Account objects (4).

syntax = "proto3";

package model;

option java_package = "pl.piomin.services.grpc.account.model";
option java_outer_classname = "AccountProto";

// (1)
import "empty.proto";
import "wrappers.proto";

// (2)
service AccountsService {
  rpc FindByNumber(google.protobuf.StringValue) returns (Account) {}
  rpc FindByCustomer(google.protobuf.Int32Value) returns (Accounts) {}
  rpc FindAll(google.protobuf.Empty) returns (Accounts) {}
  rpc AddAccount(Account) returns (Account) {}
}

// (3)
message Account {
  int32 id = 1;
  string number = 2;
  int32 customer_id = 3;
}

// (4)
message Accounts {
  repeated Account account = 1;
}

As you probably remember, there are two sample Spring Boot apps. Let’s take a look at the .proto schema for the second app customer-service-grpc. It is a little bit more complicated than the previous definition. Our gRPC service will also provide several methods for searching objects and a single method for adding a new customer (1). The customer-service-grpc is communicating with the account-service-grpc app, so we need to generate Account and Accounts messages (2). Of course, you may create an additional interface module with generated Protobuf classes and share it across both our sample apps. Finally, we have to define our model classes. The Customer class contains three primitive fields id, pesel, name, the enum type, and a list of accounts assigned to the particular customer (3). There is also the Customers message containing a list of Customer objects (4).

syntax = "proto3";

package model;

option java_package = "pl.piomin.services.grpc.customer.model";
option java_outer_classname = "CustomerProto";

import "empty.proto";
import "wrappers.proto";

// (1)
service CustomersService {
  rpc FindByPesel(google.protobuf.StringValue) returns (Customer) {}
  rpc FindById(google.protobuf.Int32Value) returns (Customer) {}
  rpc FindAll(google.protobuf.Empty) returns (Customers) {}
  rpc AddCustomer(Customer) returns (Customer) {}
}

// (2)
message Account {
  int32 id = 1;
  string number = 2;
  int32 customer_id = 3;
}

message Accounts {
  repeated Account account = 1;
}

// (3)
message Customer {
  int32 id = 1;
  string pesel = 2;
  string name = 3;
  CustomerType type = 4;
  repeated Account accounts = 5;
  enum CustomerType {
    INDIVIDUAL = 0;
    COMPANY = 1;
  }
}

// (4)
message Customers {
  repeated Customer customers = 1;
}

In order to generate Java classes from .proto schemas, we will use the Maven plugin. You can between some available plugins for that. My choice fell on the protoc-jar-maven-plugin plugin. In the configuration, we need to override the default location of our .proto schema to src/main/proto. We also need to include additional Protobuf schemas we used in the .proto manifest using the includeDirectories tag. Those manifests are located inside the src/main/proto-imports directory. The output target directory is src/main/generated. By default, the plugin does not generate gRPC services. In order to enable it we need to include the outputTarget with the grpc-java type. For generating classes we will use the protoc-gen-grpc-java library.

<plugin>
  <groupId>com.github.os72</groupId>
  <artifactId>protoc-jar-maven-plugin</artifactId>
  <version>3.11.4</version>
  <executions>
    <execution>
      <phase>generate-sources</phase>
      <goals>
        <goal>run</goal>
      </goals>
      <configuration>
        <addProtoSources>all</addProtoSources>
        <includeMavenTypes>direct</includeMavenTypes>
        <outputDirectory>src/main/generated</outputDirectory>
        <inputDirectories>
          <include>src/main/proto</include>
        </inputDirectories>
        <includeDirectories>
          <include>src/main/proto-imports</include>
        </includeDirectories>
        <outputTargets>
           <outputTarget>
             <type>java</type>
             <outputDirectory>src/main/generated</outputDirectory>
           </outputTarget>
           <outputTarget>
             <type>grpc-java</type>
             <pluginArtifact>io.grpc:protoc-gen-grpc-java:1.57.2</pluginArtifact>  
             <outputDirectory>src/main/generated</outputDirectory>
          </outputTarget>
        </outputTargets>
      </configuration>
    </execution>
  </executions>
</plugin>

We will also attach the generated Java code under the src/main/generated as a source directory with the build-helper-maven-plugin Maven plugin.

<plugin>
  <groupId>org.codehaus.mojo</groupId>
  <artifactId>build-helper-maven-plugin</artifactId>
  <executions>
    <execution>
      <id>add-source</id>
      <phase>generate-sources</phase>
      <goals>
        <goal>add-source</goal>
      </goals>
      <configuration>
        <sources>
          <source>src/main/generated</source>
        </sources>
      </configuration>
    </execution>
  </executions>
</plugin>

Once you execute the mvn clean package command Maven will generate the required Java classes. Here’s the final structure of directories in the account-service-grpc app after generating the Java classes.

$ tree
.
├── pom.xml
└── src
    ├── main
    │   ├── generated
    │   │   └── pl
    │   │       └── piomin
    │   │           └── services
    │   │               └── grpc
    │   │                   └── account
    │   │                       └── model
    │   │                           ├── AccountProto.java
    │   │                           └── AccountsServiceGrpc.java
    │   ├── java
    │   │   └── pl
    │   │       └── piomin
    │   │           └── services
    │   │               └── grpc
    │   │                   └── account
    │   │                       ├── AccountApplication.java
    │   │                       ├── repository
    │   │                       │   └── AccountRepository.java
    │   │                       └── service
    │   │                           └── AccountsService.java
    │   ├── proto
    │   │   └── account.proto
    │   ├── proto-imports
    │   │   ├── empty.proto
    │   │   └── wrappers.proto
    │   └── resources
    └── test
        └── java
            └── pl
                └── piomin
                    └── services
                        └── grpc
                            └── account
                                └── AccountServicesTests.java

Using the gRPC Spring Boot Starter

Once we generate the required Protobuf model classes and gRPC stubs we can proceed to the implementation. In the first step, we need to include the following Spring Boot starter:

<dependency>
  <groupId>net.devh</groupId>
  <artifactId>grpc-server-spring-boot-starter</artifactId>
  <version>2.14.0.RELEASE</version>
</dependency>

Then we have to create the gRPC service implementation class. It needs to extend the AccountsServiceImplBase generated based on the .proto declaration. We also need to annotate the whole class with the @GrpcService (1). After that, we will override all the methods exposed over gRPC. Our service is using a simple in-memory repository (2). Each method provides a parameter object and the io.grpc.stub.StreamObserver class used for returning the responses in a reactive way (3) (4).

@GrpcService // (1)
public class AccountsService extends AccountsServiceGrpc.AccountsServiceImplBase {

    @Autowired
    AccountRepository repository; // (2)

    @Override
    public void findByNumber(StringValue request, StreamObserver<AccountProto.Account> responseObserver) { // (3)
        AccountProto.Account a = repository.findByNumber(request.getValue());
        responseObserver.onNext(a); # (4)
        responseObserver.onCompleted();
    }

    @Override
    public void findByCustomer(Int32Value request, StreamObserver<AccountProto.Accounts> responseObserver) {
        List<AccountProto.Account> accounts = repository.findByCustomer(request.getValue());
        AccountProto.Accounts a = AccountProto.Accounts.newBuilder().addAllAccount(accounts).build();
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }

    @Override
    public void findAll(Empty request, StreamObserver<AccountProto.Accounts> responseObserver) {
        List<AccountProto.Account> accounts = repository.findAll();
        AccountProto.Accounts a = AccountProto.Accounts.newBuilder().addAllAccount(accounts).build();
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }

    @Override
    public void addAccount(AccountProto.Account request, StreamObserver<AccountProto.Account> responseObserver) {
        AccountProto.Account a = repository.add(request.getCustomerId(), request.getNumber());
        responseObserver.onNext(a);
        responseObserver.onCompleted();
    }
}

Here’s the AccountRepository implementation:

public class AccountRepository {

    List<AccountProto.Account> accounts;
    AtomicInteger id;

    public AccountRepository(List<AccountProto.Account> accounts) {
        this.accounts = accounts;
        this.id = new AtomicInteger();
        this.id.set(accounts.size());
    }

    public List<AccountProto.Account> findAll() {
        return accounts;
    }

    public List<AccountProto.Account> findByCustomer(int customerId) {
        return accounts.stream().filter(it -> it.getCustomerId() == customerId).toList();
    }

    public AccountProto.Account findByNumber(String number) {
        return accounts.stream()
                .filter(it -> it.getNumber().equals(number))
                .findFirst()
                .orElseThrow();
    }

    public AccountProto.Account add(int customerId, String number) {
        AccountProto.Account a = AccountProto.Account.newBuilder()
                .setId(id.incrementAndGet())
                .setCustomerId(customerId)
                .setNumber(number)
                .build();
        return a;
    }

}

We are adding some test data on startup. Here’s our app main class:

@SpringBootApplication
public class AccountApplication {

    public static void main(String[] args) {
        SpringApplication.run(AccountApplication.class, args);
    }

    @Bean
    AccountRepository repository() {
        List<AccountProto.Account> accounts = new ArrayList<>();
        accounts.add(AccountProto.Account.newBuilder().setId(1).setCustomerId(1).setNumber("111111").build());
        accounts.add(AccountProto.Account.newBuilder().setId(2).setCustomerId(2).setNumber("222222").build());
        accounts.add(AccountProto.Account.newBuilder().setId(3).setCustomerId(3).setNumber("333333").build());
        accounts.add(AccountProto.Account.newBuilder().setId(4).setCustomerId(4).setNumber("444444").build());
        accounts.add(AccountProto.Account.newBuilder().setId(5).setCustomerId(1).setNumber("555555").build());
        accounts.add(AccountProto.Account.newBuilder().setId(6).setCustomerId(2).setNumber("666666").build());
        accounts.add(AccountProto.Account.newBuilder().setId(7).setCustomerId(2).setNumber("777777").build());
        return new AccountRepository(accounts);
    }
}

Before starting the app, we will also include Spring Boot Actuator to expose some metrics related to gRPC. We will expose under a different port than the gRPC service, so we also need to include the Spring Boot Web starter:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>

In the application.yml file we should enable the metrics endpoint:

spring.application.name: account-service-grpc

management.endpoints.web.exposure.include: metrics
management.endpoint.metrics.enabled: true

By default, the gRPC services are available under the 9090 port. We can override that number using the grpc.server.port property. Set the port to 0 to use a free random port. Let’s start our sample app:

Calling gRPC Services

We can use the grpcurl CLI tool to call the gRPC services exposed by our sample app. By default, the gRPC server will be started on a port 9090 using PLAINTEXT mode. In order to print a list of available services we need to execute the following command:

$ grpcurl --plaintext localhost:9090 list
grpc.health.v1.Health
grpc.reflection.v1alpha.ServerReflection
model.AccountsService

Then, let’s print the list of methods exposed by the model.AccountService:

$ grpcurl --plaintext localhost:9090 list model.AccountsService
model.AccountsService.AddAccount
model.AccountsService.FindAll
model.AccountsService.FindByCustomer
model.AccountsService.FindByNumber

We can also print the details about each method by using the describe keyword in the command:

$ grpcurl --plaintext localhost:9090 describe model.AccountsService.FindByNumber
model.AccountsService.FindByNumber is a method:
rpc FindByNumber ( .google.protobuf.StringValue ) returns ( .model.Account );

Now, let’s call the endpoint described with the command visible above. The name of our method is model.AccountsService.FindByNumber. We are also setting the input string parameter with the 111111 value.

$ grpcurl --plaintext -d '"111111"' localhost:9090 model.AccountsService.FindByNumber
{
  "id": 1,
  "number": "111111",
  "customer_id": 1
}

After that, we can take a look at the model.AccountsService.FindByNumber gRPC method. It takes an integer as the input parameter and returns a list of objects.

$ grpcurl --plaintext -d '1' localhost:9090 model.AccountsService.FindByCustomer
{
  "account": [
    {
      "id": 1,
      "number": "111111",
      "customer_id": 1
    },
    {
      "id": 5,
      "number": "555555",
      "customer_id": 1
    }
  ]
}

Finally, we can call the method for adding a new account. It takes the JSON object as the input parameter. Then it will return a newly created Account object with the incremented id field.

$ grpcurl --plaintext -d '{"customer_id": 6, "number": "888888"}' localhost:9090 model.AccountsService.AddAccount
{
  "id": 8,
  "number": "888888",
  "customer_id": 6
}

The gRPC Spring Boot starter adds three additional metrics to the Actuator.

We can display a number of requests per a gRPC method. Here’s the request and response for the FindByNumber method.

We can also display an average processing time per method as shown below.

Testing the gRPC Services

In the previous section, we ran the app and tested gRPC services manually using the grpcurl CLI tool. However, we can also implement unit or integrated tests based on the Spring Boot Test module. We will create the integration test for our app with the gRPC client. In order to do that, we need to include the following three dependencies in Maven pom.xml:

<dependency>
  <groupId>io.grpc</groupId>
  <artifactId>grpc-testing</artifactId>
  <version>1.51.0</version>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-test</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>net.devh</groupId>
  <artifactId>grpc-client-spring-boot-starter</artifactId>
  <version>2.14.0.RELEASE</version>
  <scope>test</scope>
</dependency>

In the test implementation visible below, we need to enable an “in-process” server (1) and disable an external server (2). Then we have to configure the client to connect to the “in-process server” (3). We will use the gRPC client already generated during the Maven build. It is available as the AccountsServiceBlockingStub class. We just to inject and annotate it properly with the @GrpcClient (4). After that, we may use the client stub to call our gRPC services (5).

@SpringBootTest(properties = {
        "grpc.server.inProcessName=test", // (1)
        "grpc.server.port=-1", // (2)
        "grpc.client.inProcess.address=in-process:test" // (3)
})
@DirtiesContext
public class AccountServicesTests {

    @GrpcClient("inProcess") // (4)
    AccountsServiceGrpc.AccountsServiceBlockingStub service;

    @Test
    void shouldFindAll() {
        AccountProto.Accounts a = service.findAll(Empty.newBuilder().build()); // (5)
        assertNotNull(a);
        assertFalse(a.getAccountList().isEmpty());
    }

    @Test
    void shouldFindByCustomer() {
        AccountProto.Accounts a = service.findByCustomer(Int32Value.newBuilder().setValue(1).build());
        assertNotNull(a);
        assertFalse(a.getAccountList().isEmpty());
    }

    @Test
    void shouldFindByNumber() {
        AccountProto.Account a = service.findByNumber(StringValue.newBuilder().setValue("111111").build());
        assertNotNull(a);
        assertNotEquals(0, a.getId());
    }

    @Test
    void shouldAddAccount() {
        AccountProto.Account a = AccountProto.Account.newBuilder()
                .setNumber("123456")
                .setCustomerId(10)
                .build();

        a = service.addAccount(a);
        assertNotNull(a);
        assertNotEquals(0, a.getId());
    }

}

Here are the results of our tests:

Communication Between gRPC Microservices

In this section, we will switch to the customer-service-grpc app. The same as with the previous app, we need to generate the classes and gRPC service stubs with the Maven command mvn clean package. The service implementation is also similar to the account-service-grpc. However, this time, we use a client to call the external gRPC method. Here’s the implementation of the @GrpcService. As you see, we are injecting the AccountClient bean and then using it to call the gRPC method exposed by the account-service-grpc app (1). Then we use the client bean to find the account assigned to the particular customer (2).

@GrpcService
public class CustomersService extends CustomersServiceGrpc.CustomersServiceImplBase {

    @Autowired
    CustomerRepository repository;
    @Autowired
    AccountClient accountClient; // (1)

    @Override
    public void findById(Int32Value request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.findById(request.getValue());
        CustomerProto.Accounts a = accountClient.getAccountsByCustomerId(c.getId()); // (2)
        List<CustomerProto.Account> l = a.getAccountList();
        c = CustomerProto.Customer.newBuilder(c).addAllAccounts(l).build();
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void findByPesel(StringValue request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.findByPesel(request.getValue());
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void findAll(Empty request, StreamObserver<CustomerProto.Customers> responseObserver) {
        List<CustomerProto.Customer> customerList = repository.findAll();
        CustomerProto.Customers c = CustomerProto.Customers.newBuilder().addAllCustomers(customerList).build();
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }

    @Override
    public void addCustomer(CustomerProto.Customer request, StreamObserver<CustomerProto.Customer> responseObserver) {
        CustomerProto.Customer c = repository.add(request.getType(), request.getName(), request.getPesel());
        responseObserver.onNext(c);
        responseObserver.onCompleted();
    }
}

Now, let’s take a look at the implementation of the AccountClient class. We use the generated client stub to call the external gRPC method (1). Pay attention to the value inside inside the annotation. It is the name of our client.

@Service
public class AccountClient {

    private static final Logger LOG = LoggerFactory.getLogger(AccountClient.class);

    @GrpcClient("account-service-grpc") // (1)
    AccountsServiceGrpc.AccountsServiceBlockingStub stub;

    public CustomerProto.Accounts getAccountsByCustomerId(int customerId) {
        try {
            return stub.findByCustomer(Int32Value.newBuilder().setValue(customerId).build());
        } catch (final StatusRuntimeException e) {
            LOG.error("Error in communication", e);
            return null;
        }
    }
}

The last thing we need to do is to provide the address of the target service. Fortunately, the gRPC Spring Boot supports service discovery with Spring Cloud. We will use Eureka as a discovery server. Therefore both our sample apps need to include the Spring Cloud Eureka client.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>

We also need to add the dependencyManagement section in the pom.xml containing the version of Spring Cloud we use.

<dependencyManagement>
  <dependencies>
    <dependency>
     <groupId>org.springframework.cloud</groupId>
       <artifactId>spring-cloud-dependencies</artifactId>
       <version>2021.0.8</version>
       <type>pom</type>
       <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

In order to avoid port conflicts with the account-service-grpc we will override the default gRPC and HTTP (Actuator) ports. We also need to provide several configuration settings for the @GrpcClient. First of all, we should have the same name as set inside the @GrpcClient annotation in the AccountClient class. The client communicates over the plaintext protocol and reads the address of the target service from the discovery server under the name set in the discovery:/// field.

server.port: 8081
grpc.server.port: 9091

grpc:
  client:
    account-service-grpc:
      address: 'discovery:///account-service-grpc'
      enableKeepAlive: true
      keepAliveWithoutCalls: true
      negotiationType: plaintext

Finally, we can run a discovery server and our two sample microservices. The Eureka server is available in our repository inside the discovery-server directory. Once you run you can go to the UI dashboard available under the http://localhost:8761 address.

Then run both our sample Spring Boot gRPC microservices. You can run all the apps using the following Maven command:

$ mvn spring-boot:run

Finally, let’s call the customer-service-grpc method that communicates with account-service-grpc. We use the grpcurl tool once again. As you see it returns a list of accounts inside the Customer object:

Final Thoughts

The gRPC Spring Boot Starter provides several useful features that simplify developer life. We can easily create services with @GrpcService, clients with @GrpcClient, or integrate gRPC with Spring Boot Actuator metrics and Spring Cloud discovery. However, there are also some drawbacks. The library is not very actively developed. There are about 2-3 releases per year, and there is still no support for Spring Boot 3. If you are looking for a more actively developed starter (but less popular) you can try the following one (I found out about it thanks to the discussion on Reddit after releasing the article).

The post Introduction to gRPC with Spring Boot appeared first on Piotr's TechBlog.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.

Firstly, let’s discuss our repository. It contains five apps. There are three microservices (employee-service, department-service, organization-service) communicating with each other through the REST client and connecting to the Mongo database. There is also the API gateway (gateway-service) created with the Spring Cloud Gateway project. Finally, the admin-service directory contains the Spring Boot Admin app used for monitoring all other apps. You can easily deploy all the apps from the source code using a single Skaffold command. If you run the following command from the repository root directory it will build the images with Jib Maven Plugin and deploy all apps on your Kubernetes cluster:

$ skaffold run

On the other hand, you can go to the particular app directory and deploy only it using exactly the same command. All the required Kubernetes YAML manifests for each app are placed inside the k8s directories. There is also a global configuration with e.g. Mongo deployment in the project root k8s directory. Here’s the structure of our sample repo:

How It Works

In our sample architecture, we will use Spring Cloud Kubernetes Config for injecting configuration via ConfigMap and Secret and Spring Cloud Kubernetes Discovery for inter-service communication with the OpenFeign client. All our apps are running within the same namespace, but we could as well deploy them across several different namespaces and handle communication between them with OpenFeign. The only thing we should do in that case is to set the property spring.cloud.kubernetes.discovery.all-namespaces to true. For more details, you can refer to the following article.

In front of our services, there is an API gateway. It is a separate app, but we could as well install it on Kubernetes using the native CRD integration. For more details, you can refer to the following post on the Spring blog. In our case, this is a standard Spring Boot 3 app that just includes and uses the Spring Cloud Gateway module. It also uses Spring Cloud Kubernetes Discovery together with Spring Cloud OpenFeign to locate and call the downstream services. Here’s the diagram that illustrates our architecture.

Using Spring Cloud Kubernetes Config

I’ll describe implementation details by the example of department-service. It exposes some REST endpoints but also calls the endpoints exposed by the employee-service. Besides the standard modules, we need to include Spring Cloud Kubernetes in the Maven dependencies. Here, we have to decide if we use the Fabric8 client or the Kubernetes Java Client. Personally, I have an experience with Fabric8, so I’ll use the spring-cloud-starter-kubernetes-fabric8-all starter to include both config and discovery modules.

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-kubernetes-fabric8-all</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-mongodb</artifactId>
</dependency>

As you see our app is connecting to the Mongo database. Let’s provide connection details and credentials required by the app. In the k8s directory, you will find the configmap.yaml file. It contains the address of Mongo and the database name. Those properties are injected into the pod as the application.properties file. And now the most important thing. The name of the ConfigMap has to be the same as the name of our app. The name of the Spring Boot is indicated by the spring.application.name property.

kind: ConfigMap
apiVersion: v1
metadata:
  name: department
data:
  application.properties: |-
    spring.data.mongodb.host: mongodb
    spring.data.mongodb.database: admin
    spring.data.mongodb.authentication-database: admin

In the current case, the name of the app is department. Here’s the application.yml file inside the app:

spring:
  application:
    name: department

The same naming rule applies to Secret. We are keeping sensitive data like the username and password to the Mongo database inside the following Secret. You can also find that content inside the secret.yaml file in the k8s directory.

kind: Secret
apiVersion: v1
metadata:
  name: department
data:
  spring.data.mongodb.password: UGlvdF8xMjM=
  spring.data.mongodb.username: cGlvdHI=
type: Opaque

Now, let’s proceed to the Deployment manifest. We will clarify the two first points here later. Spring Cloud Kubernetes requires special privileges on Kubernetes to interact with the master API (1). We don’t have to provide a tag for the image – Skaffold will handle it (2). In order to enable loading properties from ConfigMap we need to set the spring.config.import=kubernetes: property (a new way) or set the property spring.cloud.bootstrap.enabled to true (the old way). Instead of using properties directly, we will set the corresponding environment variables on the Deployment (3). By default, consuming secrets through the API is not enabled for security reasons. In order to enable it, we will set the SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI environment variable to true (4).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: department
  labels:
    app: department
spec:
  replicas: 1
  selector:
    matchLabels:
      app: department
  template:
    metadata:
      labels:
        app: department
    spec:
      serviceAccountName: spring-cloud-kubernetes # (1)
      containers:
      - name: department
        image: piomin/department # (2)
        ports:
        - containerPort: 8080
        env:
          - name: SPRING_CLOUD_BOOTSTRAP_ENABLED # (3)
            value: "true"
          - name: SPRING_CLOUD_KUBERNETES_SECRETS_ENABLEAPI # (4)
            value: "true"

Using Spring Cloud Kubernetes Discovery

We have already included the Spring Cloud Kubernetes Discovery module in the previous section using the spring-cloud-starter-kubernetes-fabric8-all starter. In order to provide a declarative REST client we will also include the Spring Cloud OpenFeign module:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

Now, we can declare the @FeignClient interface. The important thing here is the name of a discovered service. It should be the same as the name of the Kubernetes Service defined for the employee-service app.

@FeignClient(name = "employee")
public interface EmployeeClient {

    @GetMapping("/department/{departmentId}")
    List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId);

    @GetMapping("/department-with-delay/{departmentId}")
    List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId);
}

Here’s the Kubernetes Service manifest for the employee-service app. The name of the service is employee (1). The label spring-boot is set for Spring Boot Admin discovery purposes (2). You can find the following YAML in the employee-service/k8s directory.

apiVersion: v1
kind: Service
metadata:
  name: employee # (1)
  labels:
    app: employee
    spring-boot: "true" # (2)
spec:
  ports:
    - port: 8080
      protocol: TCP
  selector:
    app: employee
  type: ClusterIP

Just to clarify – here’s the implementation of the employee-service API methods called by the OpenFeign client in the department-service.

@RestController
public class EmployeeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(EmployeeController.class);
	
    @Autowired
    EmployeeRepository repository;

    // ... other endpoints implementation 

    @GetMapping("/department/{departmentId}")
    public List<Employee> findByDepartment(@PathVariable("departmentId") String departmentId) {
        LOGGER.info("Employee find: departmentId={}", departmentId);
        return repository.findByDepartmentId(departmentId);
    }

    @GetMapping("/department-with-delay/{departmentId}")
    public List<Employee> findByDepartmentWithDelay(@PathVariable("departmentId") String departmentId) throws InterruptedException {
        LOGGER.info("Employee find: departmentId={}", departmentId);
        Thread.sleep(2000);
        return repository.findByDepartmentId(departmentId);
    }
	
}

That’s all that we have to do. Now, we can just call the endpoint using the OpenFeign client from department-service. For example on the “delayed” endpoint, we can use Spring Cloud Circuit Breaker with Resilience4J.

@RestController
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(DepartmentController.class);

    DepartmentRepository repository;
    EmployeeClient employeeClient;
    Resilience4JCircuitBreakerFactory circuitBreakerFactory;

    public DepartmentController(
        DepartmentRepository repository, 
        EmployeeClient employeeClient,
        Resilience4JCircuitBreakerFactory circuitBreakerFactory) {
            this.repository = repository;
            this.employeeClient = employeeClient;
            this.circuitBreakerFactory = circuitBreakerFactory;
    }

    @GetMapping("/{id}/with-employees-and-delay")
    public Department findByIdWithEmployeesAndDelay(@PathVariable("id") String id) {
        LOGGER.info("Department findByIdWithEmployees: id={}", id);
        Department department = repository.findById(id).orElseThrow();
        CircuitBreaker circuitBreaker = circuitBreakerFactory.create("delayed-circuit");
        List<Employee> employees = circuitBreaker.run(() ->
                employeeClient.findByDepartmentWithDelay(department.getId()));
        department.setEmployees(employees);
        return department;
    }

    @GetMapping("/organization/{organizationId}/with-employees")
    public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") String organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        List<Department> departments = repository.findByOrganizationId(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Testing with Fabric8 Kubernetes

We have already finished the implementation of our service. All the Kubernetes YAML manifests are prepared and ready to deploy. Now, the question is – can we easily test that everything works fine before we proceed to the deployment on the real cluster? The answer is – yes. Moreover, we can choose between several tools. Let’s begin with the simplest option – Kubernetes mock server. In order to use it, we to include an additional Maven dependency:

<dependency>
  <groupId>io.fabric8</groupId>
  <artifactId>kubernetes-server-mock</artifactId>
  <version>6.7.1</version>
  <scope>test</scope>
</dependency>

Then, we can proceed to the test. In the first step, we need to provide several test annotations. Inside @SpringBootTest we should simulate the Kubernetes platform with spring.main.cloud-platform property set to KUBERNETES (1). Normally Spring Boot is able to autodetect if it is running on Kubernetes. In that case, we need “trick him”, because we are just simulating the API, not running the test on Kubernetes. We also need to enable the old way of ConfigMap injection with the spring.cloud.bootstrap.enabled=true property.

Once we annotate the test method with @EnableKubernetesMockClient (2) we can use an auto-configured static instance of the Fabric8 KubernetesClient (3). During the test Fabric8 library runs a web server that mocks all the API requests sent by the client. By the way, we are using Testcontainers for running Mongo (4). In the next step, we are creating the ConfigMap that injects Mongo connection settings into the Spring Boot app (5). Thanks to the Spring Cloud Kubernetes Config it is automatically loaded by the app and the app is able to connect the Mongo database on the dynamically generated port.

Spring Cloud Kubernetes comes with auto-configured Fabric8 KubernetesClient. We need to force it to connect to the mock API server. Therefore we should override kubernetes.master property used by the Fabric8 KubernetesClient into the master URL taken from the test “mocked” instance (6). Finally, we can just implement test methods in the standard way.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {
                "spring.main.cloud-platform=KUBERNETES",
                "spring.cloud.bootstrap.enabled=true"}) // (1)
@EnableKubernetesMockClient(crud = true) // (2)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesMockTest {

    private static final Logger LOG = LoggerFactory
        .getLogger(EmployeeKubernetesMockTest.class);

    static KubernetesClient client; // (3)

    @Container // (4)
    static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");

    @BeforeAll
    static void setup() {

        ConfigMap cm = client.configMaps()
                .create(buildConfigMap(mongodb.getMappedPort(27017)));
        LOG.info("!!! {}", cm); // (5)

        // (6)
        System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY, 
            client.getConfiguration().getMasterUrl());
        System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, "true");
        System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, "default");
    }

    private static ConfigMap buildConfigMap(int port) {
        return new ConfigMapBuilder().withNewMetadata()
                .withName("employee").withNamespace("default")
                .endMetadata()
                .addToData("application.properties",
                        """
                        spring.data.mongodb.host=localhost
                        spring.data.mongodb.port=%d
                        spring.data.mongodb.database=test
                        spring.data.mongodb.authentication-database=test
                        """.formatted(port))
                .build();
    }

    @Autowired
    TestRestTemplate restTemplate;

    @Test
    @Order(1)
    void addEmployeeTest() {
        Employee employee = new Employee("1", "1", "Test", 30, "test");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(2)
    void addAndThenFindEmployeeByIdTest() {
        Employee employee = new Employee("1", "2", "Test2", 20, "test2");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
        employee = restTemplate
                .getForObject("/{id}", Employee.class, employee.getId());
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(3)
    void findAllEmployeesTest() {
        Employee[] employees =
                restTemplate.getForObject("/", Employee[].class);
        assertEquals(2, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByDepartmentTest() {
        Employee[] employees =
                restTemplate.getForObject("/department/1", Employee[].class);
        assertEquals(1, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByOrganizationTest() {
        Employee[] employees =
                restTemplate.getForObject("/organization/1", Employee[].class);
        assertEquals(2, employees.length);
    }

}

Now, after running the tests we can take a look at the logs. As you see, our test is loading properties from the employee ConfigMap.

Finally, it is able to successfully connect Mongo on the dynamic port and run all the tests against that instance.

Testing with Testcontainers on k3s

As I mentioned before, there are several tools we can use for testing with Kubernetes. This time we will see how to do it with Testcomntainers. We have already used it in the previous section for running the Mongo database. But there is also the Testcontainers module for Rancher’s k3s Kubernetes distribution. Currently, it is in the incubating state, but it doesn’t bother us to try it. In order to use it in the project we need to include the following Maven dependency:

<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>k3s</artifactId>
  <scope>test</scope>
</dependency>

Here’s the implementation of the same tests as in the previous section, but this time with the k3s container. We don’t have to create any mocks. Instead, we will create the K3sContainer object (1). Before running the tests we need to create and initialize KubernetesClient. Testcontainers K3sContainer provides the getKubeConfigYaml() method for getting kubeconfig data. With the Fabric8 Config object we can initialize the client from that kubeconfig (2) (3). After that, we will create the ConfigMap with Mongo connection details (4). Finally, we have to override the master URL for Spring Cloud Kubernetes auto-configured Fabric8 client. In comparison to the previous section, we also need to set Kubernetes client certificates and keys (5).

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT,
        properties = {
                "spring.main.cloud-platform=KUBERNETES",
                "spring.cloud.bootstrap.enabled=true"})
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class EmployeeKubernetesTest {

   private static final Logger LOG = LoggerFactory
      .getLogger(EmployeeKubernetesTest.class);

   @Container
   static MongoDBContainer mongodb = new MongoDBContainer("mongo:5.0");
   @Container
   static K3sContainer k3s = new K3sContainer(DockerImageName
      .parse("rancher/k3s:v1.21.3-k3s1")); // (1)

   @BeforeAll
   static void setup() {
      Config config = Config
         .fromKubeconfig(k3s.getKubeConfigYaml()); // (2)
      DefaultKubernetesClient client = new 
         DefaultKubernetesClient(config); // (3)

      ConfigMap cm = client.configMaps().inNamespace("default")
         .create(buildConfigMap(mongodb.getMappedPort(27017)));
      LOG.info("!!! {}", cm); // (4)

      System.setProperty(Config.KUBERNETES_MASTER_SYSTEM_PROPERTY, 
         client.getConfiguration().getMasterUrl());
      
      // (5) 
      System.setProperty(Config.KUBERNETES_CLIENT_CERTIFICATE_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getClientCertData());
      System.setProperty(Config.KUBERNETES_CA_CERTIFICATE_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getCaCertData());
       System.setProperty(Config.KUBERNETES_CLIENT_KEY_DATA_SYSTEM_PROPERTY,
         client.getConfiguration().getClientKeyData());
      System.setProperty(Config.KUBERNETES_TRUST_CERT_SYSTEM_PROPERTY, 
         "true");
      System.setProperty(Config.KUBERNETES_NAMESPACE_SYSTEM_PROPERTY, 
         "default");
    }

    private static ConfigMap buildConfigMap(int port) {
        return new ConfigMapBuilder().withNewMetadata()
                .withName("employee").withNamespace("default")
                .endMetadata()
                .addToData("application.properties",
                        """
                        spring.data.mongodb.host=localhost
                        spring.data.mongodb.port=%d
                        spring.data.mongodb.database=test
                        spring.data.mongodb.authentication-database=test
                        """.formatted(port))
                .build();
    }

    @Autowired
    TestRestTemplate restTemplate;

    @Test
    @Order(1)
    void addEmployeeTest() {
        Employee employee = new Employee("1", "1", "Test", 30, "test");
        employee = restTemplate.postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(2)
    void addAndThenFindEmployeeByIdTest() {
        Employee employee = new Employee("1", "2", "Test2", 20, "test2");
        employee = restTemplate
           .postForObject("/", employee, Employee.class);
        assertNotNull(employee);
        assertNotNull(employee.getId());
        employee = restTemplate
                .getForObject("/{id}", Employee.class, employee.getId());
        assertNotNull(employee);
        assertNotNull(employee.getId());
    }

    @Test
    @Order(3)
    void findAllEmployeesTest() {
        Employee[] employees =
                restTemplate.getForObject("/", Employee[].class);
        assertEquals(2, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByDepartmentTest() {
        Employee[] employees =
                restTemplate.getForObject("/department/1", Employee[].class);
        assertEquals(1, employees.length);
    }

    @Test
    @Order(3)
    void findEmployeesByOrganizationTest() {
        Employee[] employees =
                restTemplate.getForObject("/organization/1", Employee[].class);
        assertEquals(2, employees.length);
    }

}

Run Spring Kubernetes Apps on Minikube

In this exercise, I’m using Minikube, but you can as well use any other distribution like Kind or k3s. Spring Cloud Kubernetes requires additional privileges on Kubernetes to be able to interact with the master API. So, before running the apps we will create the spring-cloud-kubernetes ServiceAccount with the required privileges. Our role needs to have access to the configmaps, pods, services, endpoints and secrets. If we do not enable discovery across all namespaces (the spring.cloud.kubernetes.discovery.all-namespaces property), it can be Role within the namespace. Otherwise, we should create a ClusterRole.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spring-cloud-kubernetes
  namespace: default
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spring-cloud-kubernetes
  namespace: default
rules:
  - apiGroups: [""]
    resources: ["configmaps", "pods", "services", "endpoints", "secrets"]
    verbs: ["get", "list", "watch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: spring-cloud-kubernetes
  namespace: default
subjects:
  - kind: ServiceAccount
    name: spring-cloud-kubernetes
    namespace: default
roleRef:
  kind: ClusterRole
  name: spring-cloud-kubernetes

Of course, you don’t have to apply the manifests visible above by yourself. As I mentioned at the beginning of the article, there is a skaffold.yaml file in the repository root directory file that contains the whole configuration. It runs manifests with Mongo Deployment (1) and with privileges (2) together with all the services.

apiVersion: skaffold/v4beta5
kind: Config
metadata:
  name: sample-spring-microservices-kubernetes
build:
  artifacts:
    - image: piomin/admin
      jib:
        project: admin-service
    - image: piomin/department
      jib:
        project: department-service
        args:
          - -DskipTests
    - image: piomin/employee
      jib:
        project: employee-service
        args:
          - -DskipTests
    - image: piomin/gateway
      jib:
        project: gateway-service
    - image: piomin/organization
      jib:
        project: organization-service
        args:
          - -DskipTests
  tagPolicy:
    gitCommit: {}
manifests:
  rawYaml:
    - k8s/mongodb-*.yaml # (1)
    - k8s/privileges.yaml # (2)
    - admin-service/k8s/*.yaml
    - department-service/k8s/*.yaml
    - employee-service/k8s/*.yaml
    - gateway-service/k8s/*.yaml
    - organization-service/k8s/*.yaml

All we need to do it to deploy all the apps by executing the following skaffold command:

$ skaffold dev

Once we will do it we can display a list of running s pods:

kubectl get pod
NAME                            READY   STATUS    RESTARTS   AGE
admin-5f8c8498f-vtstx           1/1     Running   0          2m38s
department-746774879b-llrdn     1/1     Running   0          2m38s
employee-5bbf6b765f-7hsv7       1/1     Running   0          2m37s
gateway-578cb64558-m9n7f        1/1     Running   0          2m37s
mongodb-7f68b8b674-dbfnb        1/1     Running   0          2m38s
organization-5688c58656-bv8n6   1/1     Running   0          2m37s

We can also display a list of services. Some of them, like admin or gateway, are exposed as NodePort. Thanks to that we can easily access them outside of our Kubernetes cluster.

kubectl get svc
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
admin          NodePort    10.101.220.141   <none>        8080:31368/TCP   3m53s
department     ClusterIP   10.108.144.90    <none>        8080/TCP         3m52s
employee       ClusterIP   10.99.75.2       <none>        8080/TCP         3m52s
gateway        NodePort    10.96.7.237      <none>        8080:31518/TCP   3m52s
kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP          38h
mongodb        ClusterIP   10.108.198.233   <none>        27017/TCP        3m53s
organization   ClusterIP   10.107.102.26    <none>        8080/TCP         3m52s

Let’s obtain the Minikube IP address on our local machine:

$ minikube ip

Now, we can use that IP address to access e.g. Spring Boot Admin Server on the target port. For me its 31368. Spring Boot Admin should successfully discover all three microservices and connect to the /actuator endpoints exposed by that apps.

We can go to the details of each Spring Boot app. As you depatment-service is running on my local Minikube.

Once you stop the skaffold dev command, all the apps and configured will be removed from your Kubernetes cluster.

Final Thoughts

If you are running only Spring Boot apps on your Kubernetes cluster, Spring Cloud Kubernetes is an interesting option. It allows us to easily integrate with Kubernetes discovery, config maps, and secrets. Thanks to that we can take advantage of other Spring Cloud components like load balancer, circuit breaker, etc. However, if you are running apps written in different languages and frameworks, and using language-agnostic tools like service mesh (Istio, Linkerd), Spring Cloud Kubernetes may not be the best choice.

The post Spring Cloud Kubernetes with Spring Boot 3 appeared first on Piotr's TechBlog.

In general, we will cover the following topics in this article:

• Using Spring Boot 3 in cloud-native development
• Provide service discovery for all microservices with Spring Cloud Netflix Eureka. Anticipating your questions – yes, Eureka is still there. It’s the last of Netflix microservices components still available in Spring Cloud
• Spring Cloud OpenFeign in inter-service communication
• Distributed configuration with Spring Cloud Config
• API Gateway pattern with Spring Cloud Gateway including a global OpenAPI documentation with the Springdoc project
• Collecting traces with Micrometer OpenTelemetry and Zipkin

Fortunately, the migration from Spring Boot 2 to 3 is not a painful process. You can even check it out in my example repository, which was originally written in Spring Boot 2. The list of changes is not large. However, times have changed during the last five years… And we will begin our considerations from that point.

Running Environment

Here are the results of my quick 1-day voting poll run on Twitter. I assume that those results are meaningful since around 900 people voted. As you probably expect, currently, the first-choice platform for running your Spring Boot microservices is Kubernetes. I don’t have a survey conducted five years ago, but the results would probably be significantly different. Even if you had Kubernetes in your organization 5 years ago, you were probably starting a migration of your apps or at least it was in progress. Of course, there might be some exceptions, but I’m thinking about the vast majority.

You could migrate to Kubernetes during that time, but also Kubernetes ecosystem has changed a lot. There are many useful tools and platform services you may easily integrate with your apps. We can at least mention Kubernetes native solutions like service mesh (e.g. Istio) or serverless (e.g. Knative). The main question here is: if I’m running microservices on Kubernetes are Spring Cloud components still relevant? The answer is: in most cases no. Of course, you can still use Eureka for service discovery, Spring Cloud Config for a distributed configuration, or Spring Cloud Gateway for the API gateway pattern. However, you can easily replace them with Kubernetes built-in mechanisms and additional platform services.

To conclude, this article is not aimed at Kubernetes users. It shows how to easily run microservices architecture anywhere. If you are looking for staff mainly related to Kubernetes you can read my articles about the best practices for Java apps and microservices there.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.

Before we proceed to the source code, let’s take a look at the following diagram. It illustrates the architecture of our sample system. We have three independent Spring Boot 3 microservices, which register themself in service discovery, fetch properties from the configuration service, and communicate with each other. The whole system is hidden behind the API gateway. Our Spring Boot 3 microservices send traces to the Zipkin instance using the Micrometer OTEL project.

Currently, the newest version of Spring Cloud is 2022.0.1. This version of spring-cloud-dependencies should be declared as a BOM for dependency management.

<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>org.springframework.cloud</groupId>
      <artifactId>spring-cloud-dependencies</artifactId>
      <version>2022.0.1</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

Step 1: Configuration Server with Spring Cloud Config

To enable Spring Cloud Config feature for an application, we should first include spring-cloud-config-server to your project dependencies.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-config-server</artifactId>
</dependency>

Then enable running the embedded configuration server during application boot use @EnableConfigServer annotation.

@SpringBootApplication
@EnableConfigServer
public class ConfigApplication {

   public static void main(String[] args) {
      new SpringApplicationBuilder(ConfigApplication.class).run(args);
   }

}

By default Spring Cloud Config Server stores the configuration data inside the Git repository. We will change that behavior by activating the native mode. In this mode, Spring Cloud Config Server reads property sources from the classpath. We place all the YAML property files inside src/main/resources/config. Here’s the config server application.yml file. It activates the native mode and overrides a default port to 8088.

server:
  port: 8088
spring:
  profiles:
    active: native

The YAML filename will be the same as the name of the service. For example, the YAML file of discovery-service is located here: src/main/resources/config/discovery-service.yml. Besides a default profile, we will also define the custom docker profile. Therefore the name of the config file will contain the docker suffix. On the default profile, we are connecting services through localhost with dynamically assigned ports. So, the typical configuration file for the default profile will look like that:

server:
  port: 0

eureka:
  client:
    serviceUrl:
      defaultZone: http://localhost:8061/eureka/

Here’s the typical configuration file for the default profile:

server:
  port: 8080

eureka:
  client:
    serviceUrl:
      defaultZone: http://discovery-service:8061/eureka/

In order to connect the config server on the client side we need to include the following module in Maven dependencies:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-config</artifactId>
</dependency>

Depending on the running environment (localhost or docker) we need to provide different addresses for the config server:

spring:
  config:
    import: "optional:configserver:http://config-service:8088"
    activate:
      on-profile: docker
---
spring:
  application:
    name: discovery-service
  config:
    import: "optional:configserver:http://localhost:8088"

Step 2: Discovery Server with Spring Cloud Netflix Eureka

Of course, you can replace Eureka with any other discovery server supported by Spring Cloud. It can be Consul, Alibaba Nacos, or Zookeeper. The best way to run the Eureka server is just to embed it into the Spring Boot app. In order to do that, we first need to include the following Maven dependency:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-netflix-eureka-server</artifactId>
</dependency>

Then we need to set the @EnableEurekaServer annotation on the main class.

@SpringBootApplication
@EnableEurekaServer
public class DiscoveryApplication {

   public static void main(String[] args) {
      new SpringApplicationBuilder(DiscoveryApplication.class).run(args);
   }

}

There is nothing new with that. As I already mentioned, the configuration files, discovery-service.yml or discovery-service-docker.yml, should be placed inside config-service module. We have changed Eureka’s running port from the default value (8761) to 8061. For the standalone Eureka instance, we have to disable registration and omit to fetch the registry. We just want to activate a single-node, demo discovery server.

server:
  port: 8061

eureka:
  instance:
    hostname: localhost
  client:
    registerWithEureka: false
    fetchRegistry: false
    serviceUrl:
      defaultZone: http://${eureka.instance.hostname}:${server.port}/eureka/

Once you have successfully started the application you may visit Eureka Dashboard available under the address http://localhost:8061/.

Step 3: Build Apps with Spring Boot 3 and Spring Cloud

Let’s take a look at a list of required Maven modules for our microservices. Each app has to get a configuration from the config-service and needs to register itself in the discovery-service. It also exposes REST API, automatically generates API documentation, and export tracing info to the Zipkin instance. We use the springdoc-openapi v2 library dedicated to Spring Boot 3. It generates documentation in both JSON and YAML formats available under the v3/api-docs path (or /v3/api-docs.yaml for the YAML format). In order to export traces to the Zipkin server, we will include the opentelemetry-exporter-zipkin module.

<dependencies>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-api</artifactId>
    <version>2.0.2</version>
  </dependency>
</dependencies>

For the apps that call other services, we also need to include a declarative REST client. We will use Spring Cloud OpenFeign.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-openfeign</artifactId>
</dependency>

OpenFeign client automatically integrates with the service discovery. We just to set the name under which it is registered in Eureka inside the @FeingClient annotation. In order to create a client, we need to define an interface containing all the endpoints it has to call.

@FeignClient(name = "employee-service")
public interface EmployeeClient {

   @GetMapping("/organization/{organizationId}")
   List<Employee> findByOrganization(@PathVariable("organizationId") Long organizationId);
	
}

During the demo, we will send all the traces to Zipkin. It requires setting the value of the probability parameter to 1.0. In order to override the default URL of Zipkin we need to use the management.zipkin.tracing.endpoint property.

management:
  tracing:
    sampling:
      probability: 1.0
  zipkin:
    tracing:
      endpoint: http://zipkin:9411/api/v2/spans

Here’s the implementation of the @RestController in department-service. It injects the repository bean to interact with the database, and the Feign client bean to communicate with employee-service. The rest of the code is pretty simple.

@RestController
public class DepartmentController {

  private static final Logger LOGGER = LoggerFactory.getLogger(DepartmentController.class);

  DepartmentRepository repository;
  EmployeeClient employeeClient;

  public DepartmentController(DepartmentRepository repository, EmployeeClient employeeClient) {
    this.repository = repository;
    this.employeeClient = employeeClient;
  }

  @PostMapping("/")
  public Department add(@RequestBody Department department) {
    LOGGER.info("Department add: {}", department);
    return repository.add(department);
  }
	
  @GetMapping("/{id}")
  public Department findById(@PathVariable("id") Long id) {
    LOGGER.info("Department find: id={}", id);
    return repository.findById(id);
  }
	
  @GetMapping("/")
  public List<Department> findAll() {
    LOGGER.info("Department find");
    return repository.findAll();
  }
	
  @GetMapping("/organization/{organizationId}")
  public List<Department> findByOrganization(@PathVariable("organizationId") Long organizationId) {
    LOGGER.info("Department find: organizationId={}", organizationId);
    return repository.findByOrganization(organizationId);
  }
	
  @GetMapping("/organization/{organizationId}/with-employees")
  public List<Department> findByOrganizationWithEmployees(@PathVariable("organizationId") Long organizationId) {
    LOGGER.info("Department find: organizationId={}", organizationId);
    List<Department> departments = repository.findByOrganization(organizationId);
    departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
    return departments;
  }
	
}

As you see there are almost no differences in the app implementation between Spring Boot 2 and 3. The only thing you would have to do is to change all the javax.persistence to the jakarta.persistance.

Step 4: API Gateway with Spring Cloud Gateway

A gateway-service is the last app in our microservices architecture with Spring Boot 3. Beginning from Spring Boot 2 Spring Cloud Gateway replaced Netflix Zuul. We can also install it on Kubernetes using, for example, the Helm chart provided by VMWare Tanzu.

We will create a separate application with the embedded gateway. In order to do that we need to include Spring Cloud Gateway Starter in the Maven dependencies. Since our gateway has to interact with discovery and config services, it also includes Eureka Client Starter and Spring Cloud Config Starter. We don’t want to use it just as a proxy to the downstream services, but also we expose there OpenAPI documentation generated by all the apps. Since Spring Cloud Gateway is built on top of Spring WebFlux, we need to include Springdoc starters dedicated to that project.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>
<dependency>
  <groupId>org.springdoc</groupId>
  <artifactId>springdoc-openapi-starter-webflux-api</artifactId>
  <version>2.0.2</version>
</dependency>
<dependency>
  <groupId>org.springdoc</groupId>
  <artifactId>springdoc-openapi-starter-webflux-ui</artifactId>
  <version>2.0.2</version>
</dependency>

In order to expose OpenAPI documentation from multiple v3/api-docs endpoints we need to use the GroupedOpenApi object. It should provide a way to switch between documentation generated by employee-service, department-service and organization-service. Those services run on dynamic addresses (or at least random ports). In that case, we will use the RouteDefinitionLocator bean to grab the current URL of each service. Then we just need to filter a list of routes to find only those related to our three microservices. Finally, we create the GroupedOpenApi containing a service name and path.

@SpringBootApplication
public class GatewayApplication {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(GatewayApplication.class);

   public static void main(String[] args) {
      SpringApplication.run(GatewayApplication.class, args);
   }

   @Autowired
   RouteDefinitionLocator locator;

   @Bean
   public List<GroupedOpenApi> apis() {
      List<GroupedOpenApi> groups = new ArrayList<>();
      List<RouteDefinition> definitions = locator
         .getRouteDefinitions().collectList().block();
      assert definitions != null;
      definitions.stream().filter(routeDefinition -> routeDefinition
         .getId()
         .matches(".*-service"))
         .forEach(routeDefinition -> {
            String name = routeDefinition.getId()
               .replaceAll("-service", "");
            groups.add(GroupedOpenApi.builder()
               .pathsToMatch("/" + name + "/**").group(name).build());
         });
      return groups;
   }

}

Here’s the configuration of gateway-service. We should enable integration with the discovery server by setting the property spring.cloud.gateway.discovery.locator.enabled to true. Then we may proceed to define the route rules. We use the Path Route Predicate Factory for matching the incoming requests, and the RewritePath GatewayFilter Factory for modifying the requested path to adapt it to the format exposed by downstream services. The uri parameter specifies the name of the target service registered in the discovery server. For example, organization-service is available on the gateway under the /organization/** path thanks to the predicate Path=/organization/**, and the rewrite path from /organization/** to the /**.

spring:
  output:
    ansi:
      enabled: always
  cloud:
    gateway:
      discovery:
        locator:
          enabled: true
      routes:
      - id: employee-service
        uri: lb://employee-service
        predicates:
        - Path=/employee/**
        filters:
        - RewritePath=/employee/(?<path>.*), /$\{path}
      - id: department-service
        uri: lb://department-service
        predicates:
        - Path=/department/**
        filters:
        - RewritePath=/department/(?<path>.*), /$\{path}
      - id: organization-service
        uri: lb://organization-service
        predicates:
        - Path=/organization/**
        filters:
        - RewritePath=/organization/(?<path>.*), /$\{path}
      - id: openapi
        uri: http://localhost:${server.port}
        predicates:
        - Path=/v3/api-docs/**
        filters:
        - RewritePath=/v3/api-docs/(?<path>.*), /$\{path}/v3/api-docs

springdoc:
  swagger-ui:
    urls:
      - name: employee
        url: /v3/api-docs/employee
      - name: department
        url: /v3/api-docs/department
      - name: organization
        url: /v3/api-docs/organization

As you see above, we are also creating a dedicated route for Springdoc OpenAPI. It rewrites the path for the /v3/api-docs context to serve it properly in the Swagger UI.

Step 5: Running Spring Boot 3 Microservices

Finally, we can run all our microservices. With the current configuration in the repository, you can start them directly on your laptop or with Docker containers.

Option 1: Starting directly on the laptop

In total, we have 6 apps to run: 3 microservices, a discovery server, a config server, and a gateway. We also need to run Zipkin to collect and store traces from communication between the services. In the first step, we should start the config-service. We can use Spring Boot Maven plugin for that. Just go to the config-service directory and the following command. It is exposed on the 8088 port.

$ mvn spring-boot:run

We should repeat the same step for all the other apps. The discovery-service is listening on the 8061 port, while the gateway-service on the 8060 port. Microservices will start on the dynamically generated port number thanks to the server.port=0 property in config. In the final step, we can run Zipkin using its Docker container with the following command:

$ docker run -d --name zipkin -p 9411:9411 openzipkin/zipkin

Option 2: Build images and run them with Docker Compose

In the first step, we will build the whole Maven project and Docker images for all the apps. I created a profile build-image that needs to be activated to build images. It mostly uses the build-image step provided by the Spring Boot Maven Plugin. However, for config-service and discovery-service I’m using Jib because it is built on top of the base image with curl installed. For both these services Docker compose needs to verify health checks before starting other containers.

$ mvn clean package -Pbuild-image

The docker-compose.yml is available in the repository root directory. The whole file is visible below. We need to run config-service before all other apps since it provides property sources. Secondly, we should start discovery-service. In both these cases, we are defining a health check that tests the HTTP endpoint using curl inside the container. Once we start and verify config-service and discovery-service we may run gateway-service and all the microservices. All the apps are running with the docker Spring profile activated thanks to the SPRING_PROFILES_ACTIVE environment variable. It corresponds to the spring.profiles.active param that may be defined in configuration properties.

version: "3.7"
services:
  zipkin:
    container_name: zipkin
    image: openzipkin/zipkin
    ports:
      - "9411:9411"
  config-service:
    image: piomin/config-service:1.1-SNAPSHOT
    ports:
      - "8088:8088"
    healthcheck:
      test: curl --fail http://localhost:8088/employee/docker || exit 1
      interval: 5s
      timeout: 2s
      retries: 3
  discovery-service:
    image: piomin/discovery-service:1.1-SNAPSHOT
    ports:
      - "8061:8061"
    depends_on:
      config-service:
        condition: service_healthy
    links:
      - config-service
    healthcheck:
      test: curl --fail http://localhost:8061/eureka/v2/apps || exit 1
      interval: 4s
      timeout: 2s
      retries: 3
    environment:
      SPRING_PROFILES_ACTIVE: docker
  employee-service:
    image: piomin/employee-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  department-service:
    image: piomin/department-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - employee-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  organization-service:
    image: piomin/organization-service:1.2-SNAPSHOT
    ports:
      - "8080"
    depends_on:
      discovery-service:
        condition: service_healthy
    links:
      - config-service
      - discovery-service
      - employee-service
      - department-service
      - zipkin
    environment:
      SPRING_PROFILES_ACTIVE: docker
  gateway-service:
    image: piomin/gateway-service:1.1-SNAPSHOT
    ports:
      - "8060:8060"
    depends_on:
      discovery-service:
        condition: service_healthy
    environment:
      SPRING_PROFILES_ACTIVE: docker
    links:
      - config-service
      - discovery-service
      - employee-service
      - department-service
      - organization-service
      - zipkin

Finally, let’s run all the apps using Docker Compose:

$ docker-compose up

Try it out

Once you start all the apps you can perform some test calls to the services through the gateway-service. It listening on the 8060 port. There is some test data automatically generated during startup. You can call the following endpoint to test all the services and communication between them:

$ curl http://localhost:8060/employee/
$ curl http://localhost:8060/department/organization/1
$ curl http://localhost:8060/department/organization/1/with-employees
$ curl http://localhost:8060/organization/
$ curl http://localhost:8060/organization/1/with-departments

Here are the logs generated by the apps during the calls visible above:

Let’s display Swagger UI exposed on the gateway. You can easily switch between contexts for all three microservices as you see below:

We can go to the Zipkin dashboard to verify the collected traces:

Final Thoughts

Treat this article as a quick guide to the most common components related to microservices with Spring Boot 3. I focused on showing you some new features since my last article on this topic. You could read how to implement tracing with Micrometer OpenTelemetry, generate API docs with Springdoc, or build Docker images with Spring Boot Maven Plugin.

The post Microservices with Spring Boot 3 and Spring Cloud appeared first on Piotr's TechBlog.

There are some significant changes in the approach to observability between Spring Boot 2 and 3. Tracing will no longer be part of Spring Cloud through the Spring Cloud Sleuth project. The core of that project has been moved to Micrometer Tracing. You can read more about the reasons and future plans in this post on the Spring blog.

The main goal of that article is to give you a simple receipt for how to enable observability for your microservices written in Spring Boot using a new approach. In order to simplify our exercise, we will use a fully managed Grafana instance in their cloud. We will build a very basic architecture with two microservices running locally. Let’s take a moment to discuss our architecture in great detail.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. It contains several tutorials. You need to go to the inter-communication directory. After that, you should just follow my instructions.

Spring Boot Observability Architecture

There are two applications: inter-callme-service and inter-caller-service. The inter-caller-service app calls the HTTP endpoint exposed by the inter-callme-service app. We run two instances of inter-callme-service. We will also configure a static load balancing between those two instances using Spring Cloud Load Balancer. All the apps will expose Prometheus metrics using Spring Boot Actuator and the Micrometer project. For tracing, we are going to use Open Telemetry with Micrometer Tracing and OpenZipkin. In order to send all the data including logs, metrics, and traces from our local Spring Boot instances to the cloud, we have to use Grafana Agent.

On the other hand, there is a stack responsible for collecting and visualizing all the data. As I mentioned before we will leverage Grafana Cloud for that. It is a very comfortable way since we don’t have to install and configure all the required tools. First of all, Grafana Cloud offers a ready instance of Prometheus responsible for collecting metrics. We also need a log aggregation tool for storing and querying logs from our apps. Grafana Cloud offers a preconfigured instance of Loki for that. Finally, we have a distributed tracing backend through the Grafana Tempo. Here’s the visualization of our whole architecture.

Enable Metrics and Tracing with Micrometer

In order to export metrics in Prometheus format, we need to include the micrometer-registry-prometheus dependency. For tracing context propagation we should add the micrometer-tracing-bridge-otel module. We should also export tracing spans using one of the formats supported by Grafana Tempo. It will be OpenZipkin through the opentelemetry-exporter-zipkin dependency.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-registry-prometheus</artifactId>
</dependency>
<dependency>
  <groupId>io.micrometer</groupId>
  <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>

We need to use the latest available version of Spring Boot 3. Currently, it is 3.0.0-RC1. As a release candidate that version is available in the Spring Milestone repository.

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>3.0.0-RC1</version>
  <relativePath/>
</parent>

One of the more interesting new features in Spring Boot 3 is the support for Prometheus exemplars. Exemplars are references to data outside of the metrics published by an application. They allow linking metrics data to distributed traces. In that case, the published metrics contain a reference to the traceId. In order to enable exemplars for the particular metrics, we need to expose percentiles histograms. We will do that for http.server.requests metric (1). We will also send all the traces to Grafana Cloud by setting the sampling probability to 1.0 (2). Finally, just to verify it works properly we print traceId and spanId in the log line (3).

spring:
  application:
    name: inter-callme-service
  output.ansi.enabled: ALWAYS

management:
  endpoints.web.exposure.include: '*'
  metrics.distribution.percentiles-histogram.http.server.requests: true # (1)
  tracing.sampling.probability: 1.0 # (2)

logging.pattern.console: "%clr(%d{HH:mm:ss.SSS}){blue} %clr(%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]){yellow} %clr(:){red} %clr(%m){faint}%n" # (3)

The inter-callme-service exposes the POST endpoint that just returns the message in the reversed order. We don’t need to add here anything, just standard Spring Web annotations.

@RestController
@RequestMapping("/callme")
class CallmeController(private val repository: ConversationRepository) {

   private val logger: Logger = LoggerFactory
       .getLogger(CallmeController::class.java)

   @PostMapping("/call")
   fun call(@RequestBody request: CallmeRequest) : CallmeResponse {
      logger.info("In: {}", request)
      val response = CallmeResponse(request.id, request.message.reversed())
      repository.add(Conversation(request = request, response = response))
      return response
   }

}

Load Balancing with Spring Cloud

In the endpoint exposed by the inter-caller-service we just call the endpoint from inter-callme-service. We use Spring RestTemplate for that. You can also declare Spring Cloud OpenFeign client, but it seems it does not currently support Micrometer Tracing out-of-the-box.

@RestController
@RequestMapping("/caller")
class CallerController(private val template: RestTemplate) {

   private val logger: Logger = LoggerFactory
      .getLogger(CallerController::class.java)

   private var id: Int = 0

   @PostMapping("/send/{message}")
   fun send(@PathVariable message: String): CallmeResponse? {
      logger.info("In: {}", message)
      val request = CallmeRequest(++id, message)
      return template.postForObject("http://inter-callme-service/callme/call",
         request, CallmeResponse::class.java)
   }

}

In this exercise, we will use a static client-side load balancer that distributes traffic across two instances of inter-callme-service. Normally, you would integrate Spring Cloud Load Balancer with service discovery based e.g. on Eureka. However, I don’t want to complicate our demo with external components in the architecture. Assuming we are running inter-callme-service on 55800 and 55900 here’s the load balancer configuration in the application.yml file:

spring:
  application:
    name: inter-caller-service
  cloud:
    loadbalancer:
      cache:
        ttl: 1s
      instances:
        - name: inter-callme-service
          servers: localhost:55800, localhost:55900

Since there is no built-in static load balancer implementation we need to add some code. Firstly, we have to inject configuration properties into the Spring bean.

@Configuration
@ConfigurationProperties("spring.cloud.loadbalancer")
class LoadBalancerConfigurationProperties {

   val instances: MutableList<ServiceConfig> = mutableListOf()

   class ServiceConfig {
      var name: String = ""
      var servers: String = ""
   }

}

Then we need to create a bean that implements the ServiceInstanceListSupplier interface. It just returns a list of ServiceInstance objects that represents all defined static addresses.

class StaticServiceInstanceListSupplier(private val properties: LoadBalancerConfigurationProperties,
                                        private val environment: Environment): 
   ServiceInstanceListSupplier {

   override fun getServiceId(): String = environment
      .getProperty(LoadBalancerClientFactory.PROPERTY_NAME)!!

   override fun get(): Flux<MutableList<ServiceInstance>> {
      val serviceConfig: LoadBalancerConfigurationProperties.ServiceConfig? =
                properties.instances.find { it.name == serviceId }
      val list: MutableList<ServiceInstance> =
                serviceConfig!!.servers.split(",", ignoreCase = false, limit = 0)
                        .map { StaticServiceInstance(serviceId, it) }.toMutableList()
      return Flux.just(list)
   }

}

Finally, we need to enable Spring Cloud Load Balancer for the app and annotate RestTemplate with @LoadBalanced.

@SpringBootApplication
@LoadBalancerClient(value = "inter-callme-service", configuration = [CustomCallmeClientLoadBalancerConfiguration::class])
class InterCallerServiceApplication {

    @Bean
    @LoadBalanced
    fun template(builder: RestTemplateBuilder): RestTemplate = 
       builder.build()

}

Here’s the client-side load balancer configuration. We are providing our custom StaticServiceInstanceListSupplier implementation as a default ServiceInstanceListSupplier. Then we set RandomLoadBalancer as a default implementation of a load-balancing algorithm.

class CustomCallmeClientLoadBalancerConfiguration {

   @Autowired
   lateinit var properties: LoadBalancerConfigurationProperties

   @Bean
   fun clientServiceInstanceListSupplier(
      environment: Environment, context: ApplicationContext):     
      ServiceInstanceListSupplier {
        val delegate = StaticServiceInstanceListSupplier(properties, environment)
        val cacheManagerProvider = context
           .getBeanProvider(LoadBalancerCacheManager::class.java)
        return if (cacheManagerProvider.ifAvailable != null) {
            CachingServiceInstanceListSupplier(delegate, cacheManagerProvider.ifAvailable)
        } else delegate
    }

   @Bean
   fun loadBalancer(environment: Environment, 
      loadBalancerClientFactory: LoadBalancerClientFactory):
            ReactorLoadBalancer<ServiceInstance> {
        val name: String? = environment.getProperty(LoadBalancerClientFactory.PROPERTY_NAME)
        return RandomLoadBalancer(
           loadBalancerClientFactory
              .getLazyProvider(name, ServiceInstanceListSupplier::class.java),
            name!!
        )
    }
}

Testing Observability with Spring Boot

Let’s see how it works. In the first step, we are going to run two instances of inter-callme-service. Since we set a static value of the listen port we need to override the property server.port for each instance. We can do it with the env variable SERVER_PORT. Go to the inter-communication/inter-callme-service directory and run the following commands:

$ SERVER_PORT=55800 mvn spring-boot:run
$ SERVER_PORT=55900 mvn spring-boot:run

Then, go to the inter-communication/inter-caller-service directory and run a single instance on the default port 8080:

$ mvn spring-boot:run

Then, let’s call our endpoint POST /caller/send/{message} several times with parameters, for example:

$ curl -X POST http://localhost:8080/caller/call/hello1

Here are the logs from inter-caller-service with the highlighted value of the traceId parameter:

Let’s take a look at the logs from inter-callme-service. As you see the traceId parameter is the same as the traceId for that request on the inter-caller-service side.

Here are the logs from the second instance of inter-callme-service:

ou could also try the same exercise with Spring Cloud OpenFeign. It is configured and ready to use. However, for me, it didn’t propagate the traceId parameter properly. Maybe, it is the case with the current non-GA versions of Spring Boot and Spring Cloud.

Let’s verify another feature – Prometheus exemplars. In order to do that we need to call the /actuator/prometheus endpoint with the Accept header that is asking for the OpenMetrics format. This is the same header Prometheus will use to scrape the metrics.

$ curl -H 'Accept: application/openmetrics-text; version=1.0.0' http://localhost:8080/actuator/prometheus

As you see several metrics for the result contain traceId and spanId parameters. These are our exemplars.

Install and Configure Grafana Agent

Our sample apps are ready. Now, the main goal is to send all the collected observables to our account on Grafana Cloud. There are various ways of sending metrics, logs, and traces to the Grafana Stack. In this article, I will show you how to use the Grafana Agent for that. Firstly, we need to install it. You can find detailed installation instructions depending on your OS here. Since I’m using macOS I can do it with Homebrew:

$ brew install grafana-agent

Before we start the agent, we need to prepare a configuration file. The location of that file also depends on your OS. For me it is $(brew --prefix)/etc/grafana-agent/config.yml. The configuration YAML manifests contain information on how we want to collect and send metrics, traces, and logs. Let’s begin with the metrics. Inside the scrape_configs section, we need to set a list of endpoints for scraping (1) and a default path (2). Inside the remote_write section, we have to pass our Grafana Cloud instance auth credentials (3) and URL (4). By default, Grafana Agent does not send exemplars. Therefore we need to enable it with the send_exemplars property (5).

metrics:
  configs:
    - name: integrations
      scrape_configs:
        - job_name: agent
          static_configs:
            - targets: ['127.0.0.1:55800','127.0.0.1:55900','127.0.0.1:8080'] # (1)
          metrics_path: /actuator/prometheus # (2)
      remote_write:
        - basic_auth: # (3)
            password: <YOUR_GRAFANA_CLOUD_TOKEN>
            username: <YOUR_GRAFANA_CLOUD_USER>
          url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push # (4)
          send_exemplars: true # (5)
  global:
    scrape_interval: 60s

You can find all information about your instance of Prometheus in the Grafana Cloud dashboard.

In the next step, we prepare a configuration for collecting and sending logs to Grafana Loki. The same as before we need to set auth credentials (1) and the URL (2) of our Loki instance. The most important thing here is to pass the location of log files (3).

logs:
  configs:
    - clients:
        - basic_auth: # (1)
            password: <YOUR_GRAFANA_CLOUD_TOKEN>
            username: <YOUR_GRAFANA_CLOUD_USER>
          # (2)
          url: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
      name: springboot
      positions:
        filename: /tmp/positions.yaml
      scrape_configs:
        - job_name: springboot-json
          static_configs:
            - labels:
                __path__: ${HOME}/inter-*/spring.log # (3)
                job: springboot-json
              targets:
                - localhost
      target_config:
        sync_period: 10s

By default, Spring Boot logs only to the console and does not write log files. In our case, the Grafana Agent reads log lines from output files. In order to write log files, we need to set a logging.file.name or logging.file.path property. Since there are two instances of inter-callme-service we need to distinguish somehow their log files. We will use the server.port property for that. The logs inside files are stored in JSON format.

logging:
  pattern:
    console: "%clr(%d{HH:mm:ss.SSS}){blue} %clr(%5p [${spring.application.name:},%X{traceId:-},%X{spanId:-}]){yellow} %clr(:){red} %clr(%m){faint}%n"
    file: "{\"timestamp\":\"%d{HH:mm:ss.SSS}\",\"level\":\"%p\",\"traceId\":\"%X{traceId:-}\",\"spanId\":\"%X{spanId:-}\",\"appName\":\"${spring.application.name}\",\"message\":\"%m\"}%n"
  file:
    path: ${HOME}/inter-callme-service-${server.port}

Finally, we will configure trace collecting. Besides auth credentials and URL of the Grafana Tempo instance, we need to enable OpenZipkin receiver (1).

traces:
  configs:
    - name: default
      remote_write:
        - endpoint: tempo-eu-west-0.grafana.net:443
          basic_auth:
            username: <YOUR_GRAFANA_CLOUD_USER>
            password: <YOUR_GRAFANA_CLOUD_TOKEN>
      # (1)
      receivers:
        zipkin:

Then, we can start the agent with the following command:

$ brew services restart grafana-agent

Grafana agent contains a Zipkin collector that listens on the default port 9411. There is also an HTTP API exposed outside the agent on the port 12345 for verifying agent status.

For example, we can use Grafana Agent HTTP API to verify how many log files it is monitoring. To do that just call the endpoint GET agent/api/v1/logs/targets. As you see, for me, it is monitoring three files. So that’s what we exactly wanted to achieve since there are two running instances of inter-callme-service and a single instance of inter-caller-service.

Visualize Spring Boot Observability with Grafana Stack

One of the most important advantages of Grafana Cloud in our exercise is that we have all the required things configured. After you navigate to the Grafana dashboard you can display a list of available data sources. As you see, there are Loki, Prometheus, and Tempo already configured.

By default, Grafana Cloud enables exemplars in the Prometheus data source. If you are running Grafana by yourself, don’t forget to enable it on your Prometheus data source.

Let’s start with the logs. We will analyze exactly the same logs as in the section “Testing Observability with Spring Boot”. We will get all the logs sent by the Grafana Agent. As you probably remember, we formatted all the logs as JSON. Therefore, we will parse them using the Json parser on the server side. Thanks to that, we would be able to filter by all the log fields. For example, we can use the traceId label a filter expression: {job="springboot-json"} | json | traceId = 1bb1d7d78a5ac47e8ebc2da961955f87.

Here’s a full list of logs without any filtering. The highlighted lines contain logs of two analyzed traces.

In the next step, we will configure Prometheus metrics visualization. Since we enabled percentile histograms for the http.server.requests metrics, we have multiple buckets represented by the http_server_requests_seconds_bucket values. A set of multiple buckets _bucket with a label le which contains a count of all samples whose values are less than or equal to the numeric value contained in the le label. We need to count histograms for 90% and 60% percent of requests. Here are our Prometheus queries:

Here’s our histogram. Exemplars are shown as green diamonds.

When you hover over the selected exemplar, you will see more details. It includes, for example, the traceId value.

Finally, the last part of our exercise. We would like to analyze the particular traces with Grafana Tempo. The only thing you need to do is to choose the grafanacloud-*-traces data source and set the value of the searched traceId. Here’s a sample result.

Final Thoughts

The first GA release of Spring Boot 3 is just around the corner. Probably one of the most important things you will have to handle during migration from Spring Boot 2 is observability. In this article, you can find a detailed description of the current Spring Boot approach. If you are interested in Spring Boot, it’s worth reading about its best practices for building microservices in this article.

The post Spring Boot 3 Observability with Grafana appeared first on Piotr's TechBlog.

Architecture

Before we start, let’s take a moment to understand our scenario for today. We have a single Kafka topic used by both our applications to exchange events. This topic consists of 10 partitions. There is also a single instance of the producer that sends events at regular intervals. We are going to scale down and scale up the number of pods for the consumer service. All the instances of the consumer service are assigned to the same Kafka consumer group. It means that only a single instance with the group may receive the particular event.

Each consumer instance has only a single receiving thread. Therefore, we can easily simulate an event processing time. We will sleep the main thread for 1 second. On the other hand, the producer will send events with a variable speed. Also, it will split the messages across all available partitions. Such behavior may result in consumer lag on partitions because Spring Cloud Stream commits offset only after handling a message. In our case, the value of lag depends on producer speed and the number of running consumer instances. To clarify let’s take a look at the diagram below.

Our goal is very simple. We need to adjust the number of consumer instances to the traffic rate generated by the producer service. The value of offset lag can’t exceed the desired threshold. If we increase the traffic rate on the producer side KEDA should scale up the number of consumer instances. Consequently, if we decrease the producer traffic rate it should scale down the number of consumer instances. Here’s the diagram with our scenario.

Use Kafka with Spring Cloud Stream

In order to use Spring Cloud Stream for Kafka, we just need to include a single dependency in Maven pom.xml:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>

After that, we can use a standard Spring Cloud Stream model. However, in the background, it integrates with Kafka through a particular binder implementation. I will not explain the details, but if you are interested please read the following article. It explains the basics at the example of RabbitMQ.

Both our applications are very simple. The producer just generates and sends events (by default in JSON format). The only thing we need to do in the code is to declare the Supplier bean. In the background, there is a single thread that generates and sends CallmeEvent every second. Each time it only increases the id field inside the message:

@SpringBootApplication
public class ProducerApp {

   private static int id = 0;

   public static void main(String[] args) {
      SpringApplication.run(ProducerApp.class, args);
   }

   @Bean
   public Supplier<CallmeEvent> eventSupplier() {
      return () -> new CallmeEvent(++id, "Hello" + id, "PING");
   }

}

We can change a default fixed delay between the Supplier ticks with the following property. Let’s say we want to send an event every 100 ms:

spring.cloud.stream.poller.fixedDelay = 100

We should also provide basic configuration like the Kafka address, topic name (if different than the name of the Supplier function), number of partitions, and a partition key. Spring Cloud Stream automatically creates topics on application startup.

spring.cloud.stream.bindings.eventSupplier-out-0.destination = test-topic
spring.cloud.stream.bindings.eventSupplier-out-0.producer.partitionKeyExpression = payload.id
spring.cloud.stream.bindings.eventSupplier-out-0.producer.partitionCount = 10
spring.kafka.bootstrap-servers = one-node-cluster.redpanda:9092

Now, the consumer application. It is also not very complicated. As I mentioned before, we will sleep the main thread inside the receiving method in order to simulate processing time.

public class ConsumerApp {

   private static final Logger LOG = LoggerFactory.getLogger(ConsumerAApp.class);

   public static void main(String[] args) {
      SpringApplication.run(ConsumerApp.class, args);
   }

   @Bean
   public Consumer<Message<CallmeEvent>> eventConsumer() {
      return event -> {
         LOG.info("Received: {}", event.getPayload());
         try {
            Thread.sleep(1000);
         } catch (InterruptedException e) { }
      };
   }

}

Finally, the configuration on the consumer side. It is important to set the consumer group and enable partitioning.

spring.cloud.stream.bindings.eventConsumer-in-0.destination = test-topic
spring.cloud.stream.bindings.eventConsumer-in-0.group = a
spring.cloud.stream.bindings.eventConsumer-in-0.consumer.partitioned = true
spring.kafka.bootstrap-servers = one-node-cluster.redpanda:9092

Now, we should deploy both applications on Kubernetes. But before we do that, let’s install Kafka and KEDA on Kubernetes.

Install Kafka on Kubernetes

To perform this part you need to install helm. Instead of Kafka directly, we can install Redpanda. It is a Kafka API compatible platform. However, the Redpanda operator requires cert-manager to create certificates for TLS communication. So, let’s install it first. We use the latest version of the cert-manager. It requires adding CRDs:

$ kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.crds.yaml

Then you need to add a new Helm repository:

$ helm repo add jetstack https://charts.jetstack.io

And finally, install it cert-manager:

$ helm install cert-manager \
   --namespace cert-manager \
   --version v1.6.1 \
   jetstack/cert-manager

Now, we can proceed to the Redpanda installation. The same as before, let’s add the Helm repository:

$ helm repo add redpanda https://charts.vectorized.io/
$ helm repo update

We can obtain the latest version of Redpanda:

$ export VERSION=$(curl -s https://api.github.com/repos/vectorizedio/redpanda/releases/latest | jq -r .tag_name)

Then, let’s add the CRDs:

$ kubectl apply -f https://github.com/vectorizedio/redpanda/src/go/k8s/config/crd

After that, we can finally install the Redpanda operator:

$ helm install \
   redpanda-operator \
   redpanda/redpanda-operator \
   --namespace redpanda-system \
   --create-namespace \
   --version $VERSION

We will install a single-node cluster in the redpanda namespace. To do that we need to apply the following manifest:

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  name: one-node-cluster
spec:
  image: "vectorized/redpanda"
  version: "latest"
  replicas: 1
  resources:
    requests:
      cpu: 1
      memory: 1.2Gi
    limits:
      cpu: 1
      memory: 1.2Gi
  configuration:
    rpcServer:
      port: 33145
    kafkaApi:
    - port: 9092
    pandaproxyApi:
    - port: 8082
    adminApi:
    - port: 9644
    developerMode: true

Once you did that, you can verify a list of pods in the redpanda namespace:

$ kubectl get pod -n redpanda                      
NAME                 READY   STATUS    RESTARTS   AGE
one-node-cluster-0   1/1     Running   0          4s

If you noticed, I have already set a Kafka bootstrap server address in the application.properties. For me, it is one-node-cluster.redpanda:9092. You can verify it using the following command:

$ kubectl get svc -n redpanda
NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                      AGE
one-node-cluster   ClusterIP   None         <none>        9644/TCP,9092/TCP,8082/TCP   23h

Install KEDA and integrate it with Kafka

The same as before we will install KEDA on Kubernetes with Helm. Let’s add the following Helm repo:

$ helm repo add kedacore https://kedacore.github.io/charts

Don’t forget to update the repository. We will install the operator in the keda namespace. Let’s create the namespace first:

$ kubectl create namespace keda

Finally, we can install the operator:

$ helm install keda kedacore/keda --namespace keda

I will run both example applications in the default namespace, so I will create a KEDA object also in this namespace. The main object responsible for configuring autoscaling with KEDA is ScaledObject. Here’s the definition:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: consumer-scaled
spec:
  scaleTargetRef:
    name: consumer-deployment # (1)
  cooldownPeriod: 30 # (2)
  maxReplicaCount:  10 # (3)
  advanced:
    horizontalPodAutoscalerConfig: # (4)
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 30
          policies:
            - type: Percent
              value: 50
              periodSeconds: 30
  triggers: # (5)
    - type: kafka
      metadata:
        bootstrapServers: one-node-cluster.redpanda:9092
        consumerGroup: a
        topic: test-topic
        lagThreshold: '5'

Let’s analyze the configuration in the details:

(1) We are setting autoscaler for the consumer application, which is deployed under the consumer-deployment name (see the next section for the Deployment manifest)

(2) We decrease the default value of the cooldownPeriod parameter from 300 seconds to 30 in order to test the scale-to-zero mechanism

(3) The maximum number of running pods is 10 (the same as the number of partitions in the topic) instead of the default 100

(4) We can customize the behavior of the Kubernetes HPA. Let’s do that for the scale-down operation. We could as well configure that for the scale-up operation. We allow to scale down 50% of current running replicas.

(5) The last and the most important part – a trigger configuration. We should set the address of the Kafka cluster, the name of the topic, and the consumer group used by our application. The lag threshold is 10. It sets the average target value of offset lags to trigger scaling operations.

Before applying the manifest containing ScaledObject we need to deploy the consumer application. Let’s proceed to the next section.

Test Autoscaling with KEDA and Kafka

Let’s deploy the consumer application first. It is prepared to be deployed with Skaffold, so you can just run the command skaffold dev from the consumer directory. Anyway, here’s the Deployment manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer-deployment
spec:
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - name: consumer
        image: piomin/consumer
        ports:
        - containerPort: 8080

Once we created it, we can also apply the KEDA ScaledObject. After creating let’s display its status with the kubectl get so command.

Ok, but… it is inactive. If you think about that’s logical since there are no incoming events in Kafka’s topic. Right? So, KEDA has performed a scale-to-zero operation as shown below:

Now, let’s deploy the producer application. For now, DO NOT override the default value of the spring.cloud.stream.poller.maxMessagesPerPoll parameter. The producer will send one event per second.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: producer
spec:
  selector:
    matchLabels:
      app: producer
  template:
    metadata:
      labels:
        app: producer
    spec:
      containers:
        - name: producer
          image: piomin/producer
          ports:
            - containerPort: 8080

After some time you can run the kubectl get so once again. Now the status in the ACTIVE column should be true. And there is a single instance of the consumer application (1 event per second sent by a producer and received by a consumer).

We can also verify offsets and lags for consumer groups on the topic partitions. Just run the command rpk group describe a inside the Redpanda container.

Now, we will change the traffic rate on the producer side. It will send 5 events per second instead of 1 event before. To do that we have to define the following property in the application.properties file.

spring.cloud.stream.poller.fixedDelay = 200

Before KEDA performs autoscaling we still have a single instance of the consumer application. Therefore, after some time the lag on partitions will exceed the desired threshold as shown below.

Once autoscaling occurs we can display a list of deployments. As you see, now there are 5 running pods of the consumer service.

Once again let’s verify the status of our Kafka consumer group. There 5 consumers on the topic partitions.

Finally, just to check it out – let’s undeploy the producer application. What happened? The consumer-deployment has been scaled down to zero.

Final Thoughts

You can use KEDA not only with Kafka. There are a lot of other options available including databases, different message brokers, or even cron. Here’s a full list of the supported tools. In this article, I showed how to use Kafka consumer offset and lag as a criterium for autoscaling with KEDA. I tried to explain this process in the detail. Hope it helps you to understand how KEDA exactly works

The post Autoscaling on Kubernetes with KEDA and Kafka appeared first on Piotr's TechBlog.