Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Quarkus Buildpacks Extension

Recently, support for Cloud Native Buildpacks in Quarkus has been significantly enhanced. Here you can access the repository containing the source code for the Paketo Quarkus buildpack. To implement this solution, add one dependency to your application.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-container-image-buildpack</artifactId>
</dependency>

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-container-image-buildpack</artifactId>
</dependency>

Next, run the build command with Maven and activate the quarkus.container-image.build parameter. Also, set the appropriate Java version needed for your application. For the sample Quarkus application in this article, the Java version is 21.

mvn clean package \
  -Dquarkus.container-image.build=true \
  -Dquarkus.buildpack.builder-env.BP_JVM_VERSION=21

mvn clean package \
  -Dquarkus.container-image.build=true \
  -Dquarkus.buildpack.builder-env.BP_JVM_VERSION=21

To build, you need Docker or Podman running. Here’s the output from the command run earlier.

As you can see, Quarkus uses, among other buildpacks, the buildpack as mentioned earlier.

The new image is now available for use.

$ docker images sample-quarkus/person-service:1.0.0-SNAPSHOT
REPOSITORY                      TAG              IMAGE ID       CREATED        SIZE
sample-quarkus/person-service   1.0.0-SNAPSHOT   e0b58781e040   45 years ago   160MB

$ docker images sample-quarkus/person-service:1.0.0-SNAPSHOT
REPOSITORY                      TAG              IMAGE ID       CREATED        SIZE
sample-quarkus/person-service   1.0.0-SNAPSHOT   e0b58781e040   45 years ago   160MB

Quarkus with OpenShift Builds Shipwright

Install the Openshift Build Operator

Now, we will move the image building process to the OpenShift cluster. OpenShift offers built-in support for creating container images directly within the cluster through OpenShift Builds, using the BuildConfig solution. For more details, please refer to my previous article. However, in this article, we explore a new technology for building container images called OpenShift Builds with Shipwright. To enable this solution on OpenShift, you need to install the following operator.

After installing this operator, you will see a new item in the “Build” menu called “Shiwright”. Switch to it, then select the “ClusterBuildStrategies” tab. There are two strategies on the list designed for Cloud Native Buildpacks. We are interested in the buildpacks strategy.

Create and Run Build with Shipwright

Finally, we can create the Shiwright Build object. It contains three sections. In the first step, we define the address of the container image repository where we will push our output image. For simplicity, we will use the internal registry provided by the OpenShift cluster itself. In the source section, we specify the repository address where the application source code is located. In the last section, we need to set the build strategy. We chose the previously mentioned buildpacks strategy for Cloud Native Buildpacks. Some parameters need to be set for the buildpacks strategy: run-image and cnb-builder-image. The cnb-builder-image indicates the name of the builder image containing the buildpacks. The run-image refers to a base image used to run the application. We will also activate the buildpacks Maven profile during the build to set the Quarkus property that switches from fast-jar to uber-jar packaging.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: buildpack-quarkus-build
spec:
  env:
    - name: BP_JVM_VERSION
      value: '21'
  output:
    image: 'image-registry.openshift-image-registry.svc:5000/builds/sample-quarkus-microservice:1.0'
  paramValues:
    - name: run-image
      value: 'paketobuildpacks/run-java-21-ubi9-base:latest'
    - name: cnb-builder-image
      value: 'paketobuildpacks/builder-jammy-java-tiny:latest'
    - name: env-vars
      values:
        - value: BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS=-Pbuildpacks
  retention:
    atBuildDeletion: true
  source:
    git:
      url: 'https://github.com/piomin/sample-quarkus-microservice.git'
    type: Git
  strategy:
    kind: ClusterBuildStrategy
    name: buildpacks

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: buildpack-quarkus-build
spec:
  env:
    - name: BP_JVM_VERSION
      value: '21'
  output:
    image: 'image-registry.openshift-image-registry.svc:5000/builds/sample-quarkus-microservice:1.0'
  paramValues:
    - name: run-image
      value: 'paketobuildpacks/run-java-21-ubi9-base:latest'
    - name: cnb-builder-image
      value: 'paketobuildpacks/builder-jammy-java-tiny:latest'
    - name: env-vars
      values:
        - value: BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS=-Pbuildpacks
  retention:
    atBuildDeletion: true
  source:
    git:
      url: 'https://github.com/piomin/sample-quarkus-microservice.git'
    type: Git
  strategy:
    kind: ClusterBuildStrategy
    name: buildpacks

Here’s the Maven buildpacks profile that sets a single Quarkus property quarkus.package.jar.type. We must change it to uber-jar, because the paketobuildpacks/builder-jammy-java-tiny builder expects a single jar instead of the multi-folder layout used by the default fast-jar format. Of course, I would prefer to use the paketocommunity/builder-ubi-base builder, which can recognize the fast-jar format. However, at this time, it does not function correctly with OpenShift Builds.

<profiles>
  <profile>
    <id>buildpacks</id>
    <activation>
      <property>
        <name>buildpacks</name>
      </property>
    </activation>
    <properties>
      <quarkus.package.jar.type>uber-jar</quarkus.package.jar.type>
    </properties>
  </profile>
</profiles>

<profiles>
  <profile>
    <id>buildpacks</id>
    <activation>
      <property>
        <name>buildpacks</name>
      </property>
    </activation>
    <properties>
      <quarkus.package.jar.type>uber-jar</quarkus.package.jar.type>
    </properties>
  </profile>
</profiles>

To start the build, you can use the OpenShift console or execute the following command:

shp build run buildpack-quarkus-build --follow

shp build run buildpack-quarkus-build --follow

We can switch to the OpenShift Console. As you can see, our build is running.

The history of such builds is available on OpenShift. You can also review the build logs.

Finally, you should see your image in the list of OpenShift internal image streams.

$ oc get imagestream
NAME                          IMAGE REPOSITORY                                                                                                    TAGS                UPDATED
sample-quarkus-microservice   default-route-openshift-image-registry.apps.pminkows.95az.p1.openshiftapps.com/builds/sample-quarkus-microservice   1.2,0.0.1,1.1,1.0   13 hours ago

$ oc get imagestream
NAME                          IMAGE REPOSITORY                                                                                                    TAGS                UPDATED
sample-quarkus-microservice   default-route-openshift-image-registry.apps.pminkows.95az.p1.openshiftapps.com/builds/sample-quarkus-microservice   1.2,0.0.1,1.1,1.0   13 hours ago

Conclusion

OpenShift Build Shipwright lets you perform the entire application image build process on the OpenShift cluster in a standardized manner. Cloud Native Buildpacks is a popular mechanism for building images without writing a Dockerfile yourself. In this case, support for Buildpacks on the OpenShift side is an interesting alternative to the Source-to-Image approach.

The post Quarkus with Buildpacks and OpenShift Builds appeared first on Piotr's TechBlog.

You can find other interesting articles on my blog if you are interested in the tools for generating Kubernetes manifests. Here’s the article that shows how to use the Dekorate library to generate Kubernetes manifests for the Spring Boot app.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. First, you need to clone the following GitHub repository. It contains several sample Java applications for a Kubernetes showcase. You must go to the “inner-dev-loop” directory, to proceed with exercise. Then you should follow my further instructions.

Prerequisites

Before we start the development, we must install some tools on our laptops. Of course, we should have Maven and at least Java 21 installed. We must also have access to the container engine (like Docker or Podman) and a Kubernetes cluster. I have everything configured on my local machine using Podman Desktop and Minikube. Finally, we need to install the Helm CLI. It can be used to deploy the Postgres database on Kubernetes using the popular Bitnami Helm chart. In summary, we need to have:

• Maven
• OpenJDK 21+
• Podman or Docker
• Kubernetes
• Helm CLI

Once we have everything in place, we can proceed to the next steps.

Create Spring Boot Application

In this exercise, we create a typical Spring Boot application that connects to the relational database and exposes REST endpoints for the basic CRUD operations. Both the application and database will run on Kubernetes. We install the Postgres database using the Bitnami Helm chart. To build and deploy the application in the Kubernetes cluster, we will use Maven and Eclipse JKube features. First, let’s take a look at the source code of our application. Here’s the list of included dependencies. It’s worth noting that Spring Boot Actuator is responsible for generating Kubernetes liveness and readiness health checks. JKube will be able to detect it and generate the required elements in the Deployment manifest.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.6.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>

Here’s our Person entity class:

@Entity
public class Person {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String firstName;
    private String lastName;
    private int age;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    private Integer externalId;

    // getters and setters
}

We use the well-known Spring Data repository pattern to implement the data access layer. Here’s our PersonRepository interface. There are the additional method to search persons by age.

public interface PersonRepository extends CrudRepository<Person, Long> {
    List<Person> findByAgeGreaterThan(int age);
}

Finally, we can implement the REST controller using the previously created PersonRepository to interact with the database.

@RestController
@RequestMapping("/persons")
public class PersonController {

    private static final Logger LOG = LoggerFactory
       .getLogger(PersonController.class);
    private final PersonRepository repository;

    public PersonController(PersonRepository repository) {
        this.repository = repository;
    }

    @GetMapping
    public List<Person> getAll() {
        LOG.info("Get all persons");
        return (List<Person>) repository.findAll();
    }

    @GetMapping("/{id}")
    public Person getById(@PathVariable("id") Long id) {
        LOG.info("Get person by id={}", id);
        return repository.findById(id).orElseThrow();
    }

    @GetMapping("/age/{age}")
    public List<Person> getByAgeGreaterThan(@PathVariable("age") int age) {
        LOG.info("Get person by age={}", age);
        return repository.findByAgeGreaterThan(age);
    }

    @DeleteMapping("/{id}")
    public void deleteById(@PathVariable("id") Long id) {
        LOG.info("Delete person by id={}", id);
        repository.deleteById(id);
    }

    @PostMapping
    public Person addNew(@RequestBody Person person) {
        LOG.info("Add new person: {}", person);
        return repository.save(person);
    }
    
}

Here’s the full list of configuration properties. The database name and connection credentials are configured through environment variables: DATABASE_NAME, DATABASE_USER, and DATABASE_PASS. We should enable the exposure of the Kubernetes liveness and readiness health checks. After that, we include the database component status in the readiness probe.

spring:
  application:
    name: inner-dev-loop
  datasource:
    url: jdbc:postgresql://person-db-postgresql:5432/${DATABASE_NAME}
    username: ${DATABASE_USER}
    password: ${DATABASE_PASS}
  jpa:
    hibernate:
      ddl-auto: create
    properties:
      hibernate:
        show_sql: true
        format_sql: true

management:
  info.java.enabled: true
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint.health:
    show-details: always
    group:
      readiness:
        include: db
    probes:
      enabled: true

Install Postgres on Kubernetes

We begin our interaction with Kubernetes from the database installation. We use the Bitnami Helm chart for that. In the first step, we must add the Bitnami repository:

helm repo add bitnami https://charts.bitnami.com/bitnami 

Then, we can install the Postgres chart under the person-db name. During the installation, we create the spring user and the database under the same name.

helm install person-db bitnami/postgresql \
   --set auth.username=spring \
   --set auth.database=spring

Postgres is accessible inside the cluster under the person-db-postgresql name.

$ kubectl get svc
NAME                      TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
person-db-postgresql      ClusterIP   10.96.115.19   <none>        5432/TCP   23s
person-db-postgresql-hl   ClusterIP   None           <none>        5432/TCP   23s

Helm chart generates a Kubernetes Secret under the same name person-db-postgresql. The password for the spring user is automatically generated during installation. We can retrieve that password from the password field.

$ kubectl get secret person-db-postgresql -o yaml
apiVersion: v1
data:
  password: UkRaalNYU3o3cA==
  postgres-password: a1pBMFFuOFl3cQ==
kind: Secret
metadata:
  annotations:
    meta.helm.sh/release-name: person-db
    meta.helm.sh/release-namespace: demo
  creationTimestamp: "2024-10-03T14:39:19Z"
  labels:
    app.kubernetes.io/instance: person-db
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: postgresql
    app.kubernetes.io/version: 17.0.0
    helm.sh/chart: postgresql-16.0.0
  name: person-db-postgresql
  namespace: demo
  resourceVersion: "61646"
  uid: 00b0cf7e-8521-4f53-9c69-cfd3c942004c
type: Opaque

JKube with Spring Boot in Action

With JKube, we can build a Spring Boot app image and deploy it on Kubernetes with a single command without creating any YAML or Dockerfile. To use Eclipse JKube, we must include the org.eclipse.jkube:kubernetes-maven-plugin plugin to the Maven pom.xml. The plugin configuration contains values for two environment variables DATABASE_USER and DATABASE_NAME required by our Spring Boot application. We also set the memory and CPU request for the Deployment, which is obviusly a good practice.

<plugin>
    <groupId>org.eclipse.jkube</groupId>
    <artifactId>kubernetes-maven-plugin</artifactId>
    <version>1.17.0</version>
    <configuration>
        <resources>
            <controller>
                <env>
                    <DATABASE_USER>spring</DATABASE_USER>
                    <DATABASE_NAME>spring</DATABASE_NAME>
                </env>
                <containerResources>
                    <requests>
                        <memory>256Mi</memory>
                        <cpu>200m</cpu>
                    </requests>
                </containerResources>
            </controller>
        </resources>
    </configuration>
</plugin>

We can use the resource fragments to generate a more advanced YAML manifest with properties not covered by the plugin’s XML fields. Such a fragment of the YAML manifest must be placed in the src/main/jkube directory. In our case, the password to the database must be injected from the Kubernetes person-db-postgresql Secret generated by the Bitnami Helm chart. Here’s the fragment of Deployment YAML in the deployment.yml file:

spec:
  template:
    spec:
      containers:
        - env:
          - name: DATABASE_PASS
            valueFrom:
              secretKeyRef:
                key: password
                name: person-db-postgresql

If we want to build the image and generate Kubernetes manifests without applying them to the cluster we can use the goals k8s:build and k8s:resource during the Maven build.

mvn clean package -DskipTests k8s:build k8s:resource

Let’s take a look at the logs from the k8s:build phase. JKube reads the image group from the last part of the Maven group ID and replaces the version that contains the -SNAPSHOT suffix with the latest tag.

Here are the logs from k8s:resource phase. As you see, JKube reads the Spring Boot management.health.probes.enabled configuration property and includes /actuator/health/liveness and /actuator/health/readiness endpoints as the probes.

Here’s the Deployment object generated by the JKube plugin for our Spring Boot application.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    jkube.eclipse.org/scm-url: https://github.com/spring-projects/spring-boot/inner-dev-loop
    jkube.eclipse.org/scm-tag: HEAD
    jkube.eclipse.org/git-commit: 92b2e11f7ddb134323133aee0daa778135500113
    jkube.eclipse.org/git-url: https://github.com/piomin/kubernetes-quickstart.git
    jkube.eclipse.org/git-branch: master
  labels:
    app: inner-dev-loop
    provider: jkube
    version: 1.0-SNAPSHOT
    group: pl.piomin
    app.kubernetes.io/part-of: pl.piomin
    app.kubernetes.io/managed-by: jkube
    app.kubernetes.io/name: inner-dev-loop
    app.kubernetes.io/version: 1.0-SNAPSHOT
  name: inner-dev-loop
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: inner-dev-loop
      provider: jkube
      group: pl.piomin
      app.kubernetes.io/name: inner-dev-loop
      app.kubernetes.io/part-of: pl.piomin
      app.kubernetes.io/managed-by: jkube
  template:
    metadata:
      annotations:
        jkube.eclipse.org/scm-url: https://github.com/spring-projects/spring-boot/inner-dev-loop
        jkube.eclipse.org/scm-tag: HEAD
        jkube.eclipse.org/git-commit: 92b2e11f7ddb134323133aee0daa778135500113
        jkube.eclipse.org/git-url: https://github.com/piomin/kubernetes-quickstart.git
        jkube.eclipse.org/git-branch: master
      labels:
        app: inner-dev-loop
        provider: jkube
        version: 1.0-SNAPSHOT
        group: pl.piomin
        app.kubernetes.io/part-of: pl.piomin
        app.kubernetes.io/managed-by: jkube
        app.kubernetes.io/name: inner-dev-loop
        app.kubernetes.io/version: 1.0-SNAPSHOT
    spec:
      containers:
      - env:
        - name: DATABASE_PASS
          valueFrom:
            secretKeyRef:
              key: password
              name: person-db-postgresql
        - name: KUBERNETES_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: DATABASE_NAME
          value: spring
        - name: DATABASE_USER
          value: spring
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: piomin/inner-dev-loop:latest
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health/liveness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 180
          successThreshold: 1
        name: spring-boot
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        - containerPort: 9779
          name: prometheus
          protocol: TCP
        - containerPort: 8778
          name: jolokia
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 10
          successThreshold: 1
        securityContext:
          privileged: false

In order to deploy the application to Kubernetes, we need to add the k8s:apply to the previously executed command.

mvn clean package -DskipTests k8s:build k8s:resource k8s:apply

After that, JKube applies the generated YAML manifests to the cluster.

We can verify a list of running applications by displaying a list of pods:

$ kubectl get pod
NAME                              READY   STATUS    RESTARTS   AGE
inner-dev-loop-5cbcf7dfc6-wfdr6   1/1     Running   0          17s
person-db-postgresql-0            1/1     Running   0          106m

It is also possible to display the application logs by executing the following command:

mvn k8s:log

Here’s the output after running the mvn k8s:log command:

We can also do other things, like to undeploy the app from the Kubernetes cluster.

Final Thoughts

Eclipse JKube simplifies Spring Boot deployment on the Kubernetes cluster. Except for the presented features, it also provides mechanisms for the inner development loop with the k8s:watch and k8s:remote-dev goals.

The post Spring Boot on Kubernetes with Eclipse JKube appeared first on Piotr's TechBlog.

Assuming that we are running many Java apps on Kubernetes, we should interested in the tool that helps to automatically gather data generated by Java Flight Recorder. Here comes Cryostat. It allows us to securely manage JFR recordings for the containerized Java workloads. With the built-in discovery mechanism, it can detect all the apps that expose JFR data. Depending on the use case, we can store and analyze recordings directly on the Kubernetes cluster Cryostat Dashboard or export recorded data to perform a more in-depth analysis.

If you are interested in more topics related to Java apps on Kubernetes, you can take a look at some other posts on my blog. The following article describes a list of best practices for running Java apps Kubernetes. You can also read e.g. on how to resize CPU limit to speed up Java startup on Kubernetes here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you need to go to the callme-service directory. After that, you should just follow my instructions. Let’s begin.

Install Cryostat on Kubernetes

In the first step, we install Cryostat on Kubernetes using its operator. In order to use and manage operators on Kubernetes, we should have the Operator Lifecycle Manager (OLM) installed on the cluster. The operator-sdk binary provides a command to easily install and uninstall OLM:

$ operator-sdk olm install

Once the OLM is running on our cluster, we can proceed to the Cryostat installation. We can find the required YAML manifest with the Subscription declaration in the Operator Hub. Let’s just apply the manifest to the target with the following command:

$ kubectl create -f https://operatorhub.io/install/cryostat-operator.yaml

By default, this operator will be installed in the operators namespace and will be usable from all namespaces in the cluster. After installation, we can verify if the operator works fine by executing the following command:

$ kubectl get csv -n operators

In order to simplify the Cryostat installation process, we can use OpenShift. With OpenShift we don’t need to install OLM, since it is already there. We just need to find the “Red Hat build of Cryostat” operator in the Operator Hub and install it using OpenShift Console. By default, the operator is available in the openshift-operators namespace.

Then, let’s create a namespace dedicated to running Cryostat and our sample app. The name of the namespace is demo-jfr.

$ kubectl create ns demo-jfr

Cryostat recommends using a cert-manager for traffic encryption. In our exercise, we disable that integration for simplification purposes. However, in the production environment, you should install “cert-manager” unless you do not use another solution for encrypting traffic. In order to run Cryostat in the selected namespace, we need to create the Cryostat object. The parameter spec.enableCertManager should be set to false.

apiVersion: operator.cryostat.io/v1beta1
kind: Cryostat
metadata:
  name: cryostat-sample
  namespace: demo-jfr
spec:
  enableCertManager: false
  eventTemplates: []
  minimal: false
  reportOptions:
    replicas: 0
  storageOptions:
    pvc:
      annotations: {}
      labels: {}
      spec: {}
  trustedCertSecrets: []

If everything goes fine, you should see the following pod in the demo-jfr namespace:

$ kubectl get po -n demo-jfr
NAME                               READY   STATUS    RESTARTS   AGE
cryostat-sample-5c57c9b8b8-smzx9   3/3     Running   0          60s

Here’s a list of Kubernetes Services. The Cryostat Dashboard is exposed by the cryostat-sample Service under the 8181 port.

$ kubectl get svc -n demo-jfr
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
cryostat-sample           ClusterIP   172.31.56.83    <none>        8181/TCP,9091/TCP   70m
cryostat-sample-grafana   ClusterIP   172.31.155.26   <none>        3000/TCP            70m

We can access the Cryostat dashboard using the Kubernetes Ingress or OpenShift Route. Currently, there are no apps to monitor.

Create Sample Java App

We build a sample Java app using the Spring Boot framework. Our app exposes a single REST endpoint. As you see the endpoint implementation is very simple. The pingWithRandomDelay() method adds a random delay between 0 and 3 seconds and returns the string. However, there is one interesting thing inside that method. We are creating the ProcessingEvent object (1). Then, we call its begin method just before sleeping the thread (2). After the method is resumed we call the commit method on the ProcessingEvent object (3). In this inconspicuous way, we are generating our first custom JFR event. This event aims to monitor the processing time of our method.

@RestController
@RequestMapping("/callme")
public class CallmeController {

   private static final Logger LOGGER = LoggerFactory.getLogger(CallmeController.class);

   private Random random = new Random();
   private AtomicInteger index = new AtomicInteger();

   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping-with-random-delay")
   public String pingWithRandomDelay() throws InterruptedException {
      int r = new Random().nextInt(3000);
      int i = index.incrementAndGet();
      ProcessingEvent event = new ProcessingEvent(i); // (1)
      event.begin(); // (2)
      LOGGER.info("Ping with random delay: id={}, name={}, version={}, delay={}", i,
             buildProperties.isPresent() ? buildProperties.get().getName() : "callme-service", version, r);
      Thread.sleep(r);
      event.commit(); // (3)
      return "I'm callme-service " + version;
   }

}

Let’s switch to the ProcessingEvent implementation. Our custom event needs to extend the jdk.jfr.Event abstract class. It contains a single parameter id. We can use some additional labels to improve the event presentation in the JFR graphical tools. The event will be visible under the name set in the @Name annotation and under the category set in the @Category annotation. We also need to annotate the parameter @Label to make it visible as part of the event.

@Name("ProcessingEvent")
@Category("Custom Events")
@Label("Processing Time")
public class ProcessingEvent extends Event {
    @Label("Event ID")
    private Integer id;

    public ProcessingEvent(Integer id) {
        this.id = id;
    }

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }
}

Of course, our app will generate a lot of standard JFR events useful for profiling and monitoring. But we could also monitor our custom event.

Build App Image and Deploy on Kubernetes

Once we finish the implementation, we may build the container image of our Spring Boot app. Spring Boot comes with a feature for building container images based on the Cloud Native Buildpacks. In the Maven pom.xml you will find a dedicated profile under the build-image id. Once you activate such a profile, it will build the image using the Paketo builder-jammy-base image.

<profile>
  <id>build-image</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <image>
            <builder>paketobuildpacks/builder-jammy-base:latest</builder>
            <name>piomin/${project.artifactId}:${project.version}</name>
          </image>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>build-image</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

Before running the build we should start Docker on the local machine. After that, we should execute the following Maven command:

$ mvn clean package -Pbuild-image -DskipTests

With the build-image profile activated, Spring Boot Maven Plugin builds the image of our app. You should have a similar result as shown below. In my case, the image tag is piomin/callme-service:1.2.1.

By default, Paketo Java Buildpacks uses BellSoft Liberica JDK. With the Paketo BellSoft Liberica Buildpack, we can easily enable Java Flight Recorder for the container using the BPL_JFR_ENABLED environment variable. In order to expose data for Cryostat, we also need to enable the JMX port. In theory, we could use BPL_JMX_ENABLED and BPL_JMX_PORT environment variables for that. However, that option includes some additional configuration to the java command parameters that break the Cryostat discovery. This issue has been already described here. Therefore we will use the JAVA_TOOL_OPTIONS environment variable to set the required JVM parameters directly on the running command.

Now, let’s back to the Cryostat app discovery. Cryostat is able to automatically detect pods with a JMX port exposed. It requires the concrete configuration of the Kubernetes Service. We need to set the name of the port to jfr-jmx. In theory, we can expose JMX on any port we want, but for me anything other than 9091 caused discovery problems on Cryostat. In the Deployment definition, we have to set the BPL_JFR_ENABLED env to true, and the JAVA_TOOL_OPTIONS to -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9091.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service
  template:
    metadata:
      labels:
        app: callme-service
    spec:
      containers:
        - name: callme-service
          image: piomin/callme-service:1.2.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
            - containerPort: 9091
          env:
            - name: VERSION
              value: "v1"
            - name: BPL_JFR_ENABLED
              value: "true"
            - name: JAVA_TOOL_OPTIONS
              value: "-Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service
  labels:
    app: callme-service
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  - port: 9091
    name: jfr-jmx
  selector:
    app: callme-service

Let’s apply our deployment manifest to the demo-jfr namespace:

$ kubectl apply -f k8s/deployment-jfr.yaml -n demo-jfr

Here’s a list of pods of our callme-service app:

$ kubectl get po -n demo-jfr -l app=callme-service -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE
callme-service-6bc5745885-kvqfr   1/1     Running   0          31m   10.134.0.29   worker-cluster-lvsqq-1

Using Cryostat with JFR

View Default Dashboards

Cryostat automatically detects all the pods related to the Kubernetes Service that expose the JMX port. Once we switch to the Cryostat Dashboard, we will see the name of our pod in the “Target” dropdown. The default dashboard shows diagrams illustrating CPU load, heap memory usage, and a number of running Java threads.

Then, we can go to the “Recordings” section. It shows a list of active recordings made by Java Flight Recorder for our app running on Kubernetes. By default, Cryostat creates and starts a single recording per each detected target.

We can expand the selected record to see a detailed view. It provides a summarized panel divided into several different categories like heap, memory leak, or exceptions. It highlights warnings with a yellow color and problems with a red color.

We can display a detailed description of each case. We just need to click on the selected field with a problem name. The detailed description will appear in the context menu.

Create and Use a Custom Event Template

We can create a custom recording strategy by defining a new event template. Firstly, we need to go to the “Events” section, and then to the “Event Templates” tab. There are three built-in templates. We can use each of them as a base for our custom template. After deciding which of them to choose we can download it to our laptop. The default file extension is *.jfc.

In order to edit the *.jfc files we need a special tool called JDK Mission Control. Each vendor provides such a tool for their distribution of JDK. In our case, it is BellSoft Liberica. Once we download and install Liberica Mission Control on the laptop we should go to Window -> Flight Recording Template Manager.

With the Flight Recording Template Manager, we can import and edit an exported event template. I choose the higher monitoring for “Garbage Collection”, “Allocation Profiling”, “Compiler”, and “Thread Dump”.

Once a new template is ready, we should save it under the selected name. For me, it is the “Continuous Detailed” name. After that, we need to export the template to the file.

Then, we need to switch to the Cryostat Dashboard. We have to import the newly created template exported to the *.jfc file.

Once you import the template, you should see a new strategy in the “Event Templates” section.

We can create a recording based on our custom “Continuous_Detailed” template. After some time, Cryostat should gather data generated by the Java Flight Recorder for the app running on Kubernetes. However, this time we want to make some advanced analysys using Liberica Mission Control rather than just with the Cryostat Dashboard. Therefore we will export the recording to the *.jfr file. Such a file may be then imported to the JDK Mission Control tool.

Use the JDK Mission Control Tool

Let’s open the exported *.jfr file with Liberica Mission Control. Once we do it, we can analyze all the important aspects related to the performance of our Java app. We can display a table with memory allocation per the object type.

We can display a list of running Java threads.

Finally, we go to the “Event Browser” section. In the “Custom Events” category we should find our custom event under the name determined by the @Label annotation on the ProcessingEvent class. We can see the history of all generated JFR events together with the duration, start time, and the name of the processing thread.

Final Thoughts

Cryostat helps you to manage the Java Flight Recorder on Kubernetes at scale. It provides a graphical dashboard that allows to monitoring of all the Java workloads that expose JFR data over JMX. The important thing is that even after an app crash we can export the archived monitoring report and analyze it using advanced tools like JDK Mission Control.

The post Java Flight Recorder on Kubernetes appeared first on Piotr's TechBlog.

You can also run Cloud Native Buildpacks on “vanilla” Kubernetes together with e.g. Tekton. In the following article on my blog, you will find information on how to use Buildpacks in your Tekton CI pipeline for building Java app images.

Introduction

If you already have some experience with OpenShift, you probably know that it comes with its own mechanism for building container images. Such a build is fully executed on the OpenShift cluster. OpenShift provides the BuildConfig CRD responsible for a build configuration. There are three primary build strategies available: Docker, S2I, and custom build. By the way, you can build the image in any way you want and still run such a container on OpenShift. However, today we are discussing the OpenShift recommended approach for building container images called “OpenShift Builds”.

Recently, Red Hat has introduced a new supported way of using “OpenShift Builds” based on the Shipwright project. Therefore, you can currently find two chapters in the OpenShift documentation: “Builds with BuildConfig” and “Builds with Shipwright”. Maybe Shipwright will become a single, default strategy in the future, but for now, we can decide which of them to use.

In this article, I’ll show you how to install and manage Shipwright on OpenShift. By default, Red Hat supports Source-to-Image (S2I) and Buildah build strategies. We will create our own strategy for using the Cloud Native Buildpacks.

Install Builds for Red Hat OpenShift Operator

In order to enable Shiwright builds on OpenShift, we need to install the operator called “Builds for Red Hat OpenShift”. It depends on the OpenShift Pipelines operator. However, we don’t have to take care of it, since that operator is automatically installed together with the “Builds for Red Hat OpenShift” operator. In the OpenShift Console go to the Operators -> Operator Hub, find and install the required operator.

After that, we need to create the ShiwrightBuild object. It needs to contain the name of the target namespace for running Shipwright in the targetNamespace field. In my case, the target namespace is openshift-builds.

apiVersion: operator.shipwright.io/v1alpha1
kind: ShipwrightBuild
metadata:
  name: openshift-builds
spec:
  targetNamespace: openshift-builds

We can easily create the object on the operator page in OpenShift Console. If the status is “Ready” it means that Shipwright is running on our cluster.

Let’s display a list of pods inside the openshift-builds namespace. As you see, two pods are running there.

$ oc get po -n openshift-builds
NAME                                           READY   STATUS    RESTARTS   AGE
shipwright-build-controller-5d5b4f7d97-llj4f   1/1     Running   0          2m31s
shipwright-build-webhook-76775cf988-pjk6k      1/1     Running   0          2m31s

This step is optional. We can install the Shipwright CLI on our laptop to interact with the cluster. The release binary is available in the following repository under the Releases section. Once you install it, you can execute the following command for verification:

$ shp version

OpenShift Console supports Shipwright Builds. Once you install and configure the operator, you will have a dedicated space inside the “Builds” section. You can create a new build, run a build, and display a history of previous runs.

Configure Build Strategy for Buildpacks

Before we will define any build, we need to configure the build strategy. From a high-level perspective, build strategy defines how to build an application with an image-building tool. We can create the BuildStrategy object for the namespace-scoped strategy, and the ClusterBuildStrategy object for the cluster-wide strategy. The “Builds for Red Hat OpenShift” operator comes with two predefined global strategies: buildah and source-to-image.

However, our goal is to create a strategy for Cloud Native Buildpacks. Fortunately, we can just copy such a strategy from the samples provided in the Shiwright repository on GitHub. We won’t get into the details of that strategy implementation. The YAML manifest is visible below. It uses the paketobuildpacks/builder-jammy-full:latest image during the build. We should apply that manifest to our OpenShift cluster.

apiVersion: shipwright.io/v1beta1
kind: ClusterBuildStrategy
metadata:
  name: buildpacks-v3
spec:
  volumes:
    - name: platform-env
      emptyDir: {}
  parameters:
    - name: platform-api-version
      description: The referenced version is the minimum version that all relevant buildpack implementations support.
      default: "0.7"
  steps:
    - name: build-and-push
      image: docker.io/paketobuildpacks/builder-jammy-full:latest
      env: 
        - name: CNB_PLATFORM_API
          value: $(params.platform-api-version)
        - name: PARAM_SOURCE_CONTEXT
          value: $(params.shp-source-context)
        - name: PARAM_OUTPUT_IMAGE
          value: $(params.shp-output-image)
      command:
        - /bin/bash
      args:
        - -c
        - |
          set -euo pipefail

          echo "> Processing environment variables..."
          ENV_DIR="/platform/env"

          envs=($(env))

          # Denying the creation of non required files from system environments.
          # The creation of a file named PATH (corresponding to PATH system environment)
          # caused failure for python source during pip install (https://github.com/Azure-Samples/python-docs-hello-world)
          block_list=("PATH" "HOSTNAME" "PWD" "_" "SHLVL" "HOME" "")

          for env in "${envs[@]}"; do
            blocked=false

            IFS='=' read -r key value string <<< "$env"

            for str in "${block_list[@]}"; do
              if [[ "$key" == "$str" ]]; then
                blocked=true
                break
              fi
            done

            if [ "$blocked" == "false" ]; then
              path="${ENV_DIR}/${key}"
              echo -n "$value" > "$path"
            fi
          done

          LAYERS_DIR=/tmp/.shp/layers
          CACHE_DIR=/tmp/.shp/cache

          mkdir -p "$CACHE_DIR" "$LAYERS_DIR"

          function announce_phase {
            printf "===> %s\n" "$1" 
          }

          announce_phase "ANALYZING"
          /cnb/lifecycle/analyzer -layers="$LAYERS_DIR" "${PARAM_OUTPUT_IMAGE}"

          announce_phase "DETECTING"
          /cnb/lifecycle/detector -app="${PARAM_SOURCE_CONTEXT}" -layers="$LAYERS_DIR"

          announce_phase "RESTORING"
          /cnb/lifecycle/restorer -cache-dir="$CACHE_DIR" -layers="$LAYERS_DIR"

          announce_phase "BUILDING"
          /cnb/lifecycle/builder -app="${PARAM_SOURCE_CONTEXT}" -layers="$LAYERS_DIR"

          exporter_args=( -layers="$LAYERS_DIR" -report=/tmp/report.toml -cache-dir="$CACHE_DIR" -app="${PARAM_SOURCE_CONTEXT}")
          grep -q "buildpack-default-process-type" "$LAYERS_DIR/config/metadata.toml" || exporter_args+=( -process-type web ) 

          announce_phase "EXPORTING"
          /cnb/lifecycle/exporter "${exporter_args[@]}" "${PARAM_OUTPUT_IMAGE}"

          # Store the image digest
          grep digest /tmp/report.toml | tail -n 1 | tr -d ' \"\n' | sed s/digest=// > "$(results.shp-image-digest.path)"
      volumeMounts:
        - mountPath: /platform/env
          name: platform-env
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 250m
          memory: 65Mi
  securityContext:
    runAsUser: 1001
    runAsGroup: 1000

As you see, a new build strategy is available under the buildpacks-v3 name.

Create Build for the Sample App

Once we create a strategy, we can proceed to the build creation. We will push the image to the registry on quay.io. It requires to have an existing account there. The quay.io registry allows us to create a robot account on a private account. Then we may use it to authenticate against the repository during the build. Firstly, go to your account settings and find the “Robot Accounts” section there. If you don’t have any robot account you need to create it. On the existing account choose the settings icon, and then the “View Credentials” item in the context menu.

In the “Kubernetes Secret” section download the YAML manifest containing the authentication credentials.

Our robot account must have the Write access to the target repository. In my case, it is the pminkows/sample-kotlin-spring registry.

Once you download the manifest you have to apply it to the OpenShift cluster. Let’s also create a new project with the demo-builds name.

$ oc new-project demo-builds
$ oc apply -f pminkows-piomin-secret.yml

The manifest contains a single Secret with our Quay robot account authentication credentials.

Finally, we can create the Build object. It contains three sections. In the first of them, we are defining the address of the container image repository to push our output image (1). We need to authenticate against the repository with the previously created pminkows-piomin-pull-secret Secret. In the source section, we should provide the address of the repository with the app source code (2). The repository with a sample Spring Boot app is located on my GitHub account. In the last section, we need to set the build strategy (3). We choose the previously created buildpacks-v3 strategy for Cloud Native Buildpacks.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: sample-spring-kotlin-build
  namespace: demo-builds
spec:
  # (1)
  output:
    image: quay.io/pminkows/sample-kotlin-spring:1.0-shipwright
    pushSecret: pminkows-piomin-pull-secret
  # (2)
  source:
    git:
      url: https://github.com/piomin/sample-spring-kotlin-microservice.git
  # (3)
  strategy:
    name: buildpacks-v3
    kind: ClusterBuildStrategy

Once you apply the sample-spring-kotlin-build Build object you can switch to the “Builds” section in the OpenShift Console. In the “Shipwright Builds” tab, we can display a list of builds in the demo-builds namespace. In order to run the build, we just need to choose the “Start” option in the context menu.

Running the Shipwright Build on OpenShift

Before we run the build, we have to set some permissions for the ServiceAccount. By default, Shipwright uses the pipeline ServiceAccount to run builds. You should already have the pipeline account created in the demo-builds namespace automatically with the project creation.

Shipwright requires a privileged SCC assigned to the pipeline ServiceAccount to run the build pods. Let’s add the required permission with the following commands:

$ oc adm policy add-scc-to-user privileged -z pipeline -n demo-builds
$ oc adm policy add-role-to-user edit -z pipeline -n demo-builds

We can start the build in the OpenShift Console or with the Shipwright CLI. Firstly, let’s display a list of builds using the following shp command:

$ shp build list
NAME				OUTPUT							STATUS
sample-spring-kotlin-build	quay.io/pminkows/sample-kotlin-spring:1.0-shipwright	all validations succeeded

Let’s run the build with the following command:

$ shp build run sample-spring-kotlin-build --follow

We can switch to the OpenShift Console. As you see, our build is running.

Shipwright runs the pod with Paketo Buildpacks Builder according to the strategy.

Here’s the output of the shp command. Once the image is ready, it is pushed to the Quay registry.

The build has been finished successfully. We can see it in the Shipwright “Builds” section in the OpenShift Console (status Succeeded).

Let’s switch to the quai.io dashboard. As you see, the image tag 1.0-shipwright is already there.

Of course, we can build many times. The history of executions is available in the BuildRuns tab for each Build.

Final Thoughts

With Shipwright you can easily switch between different build image strategies on OpenShift. You are not limited only to the Source-to-image (S2I) or Docker strategy, but you can use different tools for that including e.g. Cloud Native Buildpacks. For now, the strategy based on Buildpacks is not supported by Red Hat as the Shipwright builds feature. However, I hope it will change soon

The post OpenShift Builds with Shipwright and Cloud Native Buildpacks appeared first on Piotr's TechBlog.

Unfortunately, those results were wrong. Or maybe I should say, were not averaged well enough. After this publication, I also received interesting materials presented on London Java Community. It compares the performance of the Payara application server running on various JDKs. Here’s the link to that presentation (~1h). The results showed there seem to confirm my results. Or at least they confirm the general rule – there are some performance differences between Open JDK implementations. Let’s check it out.

This time I’ll do a very accurate comparison with several repeats to get reproducible results. I’ll test the following JVM implementations:

For all the tests I’ll use Paketo Java buildpack. We can easily switch between several JVM implementations with Paketo. I’ll test a simple Spring Boot 3 app that uses Spring Data to interact with the Mongo database. Let’s proceed to the details!

If you have already built images with Dockerfile it is possible that you were using the official OpenJDK base image from the Docker Hub. However, currently, the announcement on the image site says that it is officially deprecated and all users should find suitable replacements. In this article, we will compare all the most popular replacements, so I hope it may help you to make a good choice

Testing Environment

Before we run tests it is important to have a provisioned environment. I’ll run all the tests locally. In order to build images, I’m going to use Paketo Buildpacks. Here are some details of my environment:

1. Machine: MacBook Pro 32G RAM Intel 
2. OS: macOS Ventura 13.1
3. Kubernetes (v1.25.2) on Docker Desktop: 14G RAM + 4vCPU

We will use Java 17 for app compilation. In order to run load tests, I’m going to leverage the k6 tool. Our app is written in Spring Boot. It connects to the Mongo database running on the same instance of Kubernetes. Each time I’m testing a new JVM provider I’m removing the previous version of the app and database. Then I’m deploying the new, full configuration once again. We will measure the following parameters:

1. App startup time (the best  
2. result and average) – we will read it directly from the Spring Boot logs 
3. Throughput – with k6 we will simulate 5 and 10 virtual users. It will measure the number of processing requests 
4. The size of the image
5. The RAM memory consumed by the pod during the load tests. Basically, we will execute the kubectl top pod command

We will also set the memory limit for the container to 1G. In our load tests, the app will insert data into the Mongo database. It is exposing the REST endpoint invoked during the tests. To measure startup time as accurately as possible I’ll restart the app several times.

Let’s take a look at the Deployment YAML manifest. It injects credentials to the Mongo database and set the memory limit to 1G (as I already mentioned):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-boot-on-kubernetes-deployment
spec:
  selector:
    matchLabels:
      app: sample-spring-boot-on-kubernetes
  template:
    metadata:
      labels:
        app: sample-spring-boot-on-kubernetes
    spec:
      containers:
      - name: sample-spring-boot-on-kubernetes
        image: piomin/sample-spring-boot-on-kubernetes
        ports:
        - containerPort: 8080
        env:
          - name: MONGO_DATABASE
            valueFrom:
              configMapKeyRef:
                name: mongodb
                key: database-name
          - name: MONGO_USERNAME
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-user
          - name: MONGO_PASSWORD
            valueFrom:
              secretKeyRef:
                name: mongodb
                key: database-password
          - name: MONGO_URL
            value: mongodb
        readinessProbe:
          httpGet:
            port: 8080
            path: /readiness
            scheme: HTTP
          timeoutSeconds: 1
          periodSeconds: 10
          successThreshold: 1
          failureThreshold: 3
        resources:
          limits:
            memory: 1024Mi

Source Code and Images

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. You will also find all the images in my Docker Hub repository piomin/sample-spring-boot-on-kubernetes. Every single image is tagged with the vendor’s name.

Our Spring Boot app exposes several endpoints, but I’ll test the POST /persons endpoint for inserting data into Mongo. In the integration with Mongo, I’m using the Spring Data MongoDB project and its CRUD repository pattern.

// controller

@RestController
@RequestMapping("/persons")
public class PersonController {

   private PersonRepository repository;

   PersonController(PersonRepository repository) {
      this.repository = repository;
   }

   @PostMapping
   public Person add(@RequestBody Person person) {
      return repository.save(person);
   }

   // other endpoints implementation
}


// repository

public interface PersonRepository extends CrudRepository<Person, String> {

   Set<Person> findByFirstNameAndLastName(String firstName, 
                                          String lastName);
   Set<Person> findByAge(int age);
   Set<Person> findByAgeGreaterThan(int age);

}

The Size of the Image

The size of the image is the simplest option to measure. If you would like to check what is exactly inside the image you can use the dive tool. The difference in the size between vendors results from the number of java tools and binaries included inside. From my perspective, the smaller the size the better. I’d rather not use anything that is inside the image. Of course, except all the staff required to run my app successfully. But you may have a different case. Anyway, here’s the content of the app for the Oracle JDK after executing the dive piomin/sample-spring-boot-on-kubernetes:oracle command. As you see, JDK takes up most of the space.

On the other hand, we can analyze the smallest image. I think it explains the differences in image size since Zulu contains JRE, not the whole JDK.

Here are the result ordered from the smallest image to the biggest.

Let’s visualize our first results. I think it excellent shows which image contains JDK and which JRE.

Startup Time

Honestly, it is not very easy to measure a startup time, since the difference between the vendors is not large. Also, the subsequent results for the same provider may differ a lot. For example, on the first try the app starts in 5.8s and after the pod restart 8.4s. My methodology was pretty simple. I restarted the app several times for each JDK provider to measure the average startup time and the fastest startup in the series. Then I repeated the same exercise again to verify if the results are repeatable. The proportions between the first and second series of startup time between corresponding vendors were similar. In fact, the difference between the fastest and the slowest average startup time is not large. I get the best result for Eclipse Temurin (7.2s) and the worst for IBM Semeru OpenJ9 (9.05s).

Let’s see the full list of results. It shows the average startup time of the application from the fastest one.

Once again, here’s the graphical representation of our results. The differences between vendors are sometimes rather cosmetic. Maybe, if the same exercise once again from the beginning the results would be quite different.

As I mentioned before, I also measured the fastest attempt. This time the best top 3 are Eclipse Temurin, Amazon Corretto, and BellSoft Liberica.

Memory

I’m measuring the memory usage of the app under the heavy load with a test simulating 10 users continuously sending requests. It gives me a really large throughput at the level of the app – around 500 requests per second. The results are in line with the expectations. Almost all the vendors have very similar memory usage except IBM Semeru, which uses OpenJ9 JVM. In theory, OpenJ9 should also give us a better startup time. However, in my case, the significant difference is just in the memory footprint. For IBM Semeru the memory usage is around 135MB, while for other vendors it varies in the range of 210-230MB.

Here’s the graphical visualization of our results:

Throughput

In order to generate high incoming traffic to the app I used the k6 tool. It allows us to create tests in JavaScript. Here’s the implementation of our test. It is calling the HTTP POST /persons endpoint with input data in JSON. Then it verifies if the request has been successfully processed on the server side.

import http from 'k6/http';
import { check } from 'k6';

export default function () {

  const payload = JSON.stringify({
      firstName: 'aaa',
      lastName: 'bbb',
      age: 50,
      gender: 'MALE'
  });

  const params = {
    headers: {
      'Content-Type': 'application/json',
    },
  };

  const res = http.post(`http://localhost:8080/persons`, payload, params);

  check(res, {
    'is status 200': (res) => res.status === 200,
    'body size is > 0': (r) => r.body.length > 0,
  });
}

Here’s the k6 command for running our test. It is possible to define the duration and number of simultaneous virtual users. In the first step, I’m simulating 5 virtual users:

$ k6 run -d 90s -u 5 load-tests.js

Then, I’m running the tests for 10 virtual users twice per vendor.

$ k6 run -d 90s -u 10 load-tests.js

Here are the sample results printed after executing the k6 test:

I repeated the exercise per the JDK vendor. Here are the throughput results for 5 virtual users:

Here are the throughput results for 10 virtual users:

Final Thoughts

After repeating the load tests several times I need to admit that there are no significant differences in performance between all JDK vendors. We were using the same JVM settings for testing (set by the Paketo Buildpack). Probably, the more tests I will run, the results between different vendors would be even more similar. So, in summary, the results from my tweet have not been confirmed. Ok, so let’s back to the question – which JDK to choose on Kubernetes?

Probably it somehow depends on where you are running your cluster. If for example, it’s Kubernetes EKS on AWS it’s worth using Amazon Corretto. However, if you are looking for the smallest image size you should choose between Azul Zulu, IBM Semeru, BellSoft Liberica, and Adoptium Eclipse Temurin. Additionally, IBM Semeru will consume significantly less memory than other distributions, since it is built on top of OpenJ9.

Don’t forget about best practices when deploying Java apps on Kubernetes. Here’s my article about it.

The post Which JDK to Choose on Kubernetes appeared first on Piotr's TechBlog.

I’m writing a lot about topics related to both Kubernetes and Java. You can find many practical examples on my blog. Some time ago I published a similar article to that one – but mostly focused on best practices for microservices-based apps. You can find it here.

Don’t Set Limits Too Low

Should we set limits for Java apps on Kubernetes or not? The answer seems to be obvious. There are many tools that validate your Kubernetes YAML manifests, and for sure they will print a warning if you don’t set CPU or memory limits. However, there are some “hot discussions” in the community about that. Here’s an interesting article that does not recommend setting any CPU limits. Here’s another article written as a counterpoint to the previous one. They consider CPU limits, but we could as well begin a similar discussion for memory limits. Especially in the context of Java apps

However, for memory management, the proposition seems to be quite different. Let’s read the other article – this time about memory limits and requests. In short, it recommends always setting the memory limit. Moreover, the limit should be the same as the request. In the context of Java apps, it is also important that we may limit the memory with JVM parameters like -Xmx, -XX:MaxMetaspaceSize or -XX:ReservedCodeCacheSize. Anyway, from the Kubernetes perspective, the pod receives the resources it requests. The limit has nothing to do with it.

It all leads me to the first recommendation today – don’t set your limits too low. Even if you set a CPU limit, it shouldn’t impact your app. For example, as you probably know, even if your Java app doesn’t consume much CPU in normal work, it requires a lot of CPU to start fast. For my simple Spring Boot app that connects MongoDB on Kubernetes, the difference between no limit and even 0.5 core is significant. Normally it starts below 10 seconds:

With the CPU limit set to 500 millicores, it starts ~30 seconds:

Of course, we could find some examples. But we will discuss them also in the next sections.

Consider Memory Usage First

Let’s focus just on the memory limit. If you run a Java app on Kubernetes, you have two levels of limiting maximum usage: container and JVM. However, there are also some defaults if you don’t specify any settings for JVM. JVM sets its maximum heap size to approximately 25% of the available RAM in case you don’t set the -Xmx parameter. This value is counted based on the memory visible inside the container. Once you won’t set a limit at the container level, JVM will see the whole memory of the node.

Before running the app on Kubernetes, you should at least measure how much memory it consumes at the expected load. Fortunately, there are tools that may optimize memory configuration for Java apps running in containers. For example, Paketo Buildpacks comes with a built-in memory calculator. It calculates the -Xmx JVM flag using the formula Heap = Total Container Memory - Non-Heap - Headroom. On the other hand, the non-heap value is calculated using the following formula: Non-Heap = Direct Memory + Metaspace + Reserved Code Cache + (Thread Stack * Thread Count).

Paketo Buildpacks is currently a default option for building Spring Boot apps (with the mvn spring-boot:build-image command). Let’s try it for our sample app. Assuming we will set the memory limit to 512M it will calculate -Xmx at a level of 130M.

Is it fine for my app? I should at least perform some load tests to verify how my app performs under heavy traffic. But once again – don’t set the limits too low. For example, with the 1024M limit, the -Xmx equals 650M.

As you see we take care of memory usage with JVM parameters. It prevents us from OOM kills described in the article mentioned in the first section. Therefore, setting the request at the same level as the limit does not make much sense. I would recommend setting it a little higher than normal usage – let’s say 20% more.

Proper Liveness and Readiness Probes

Introduction

It is essential to understand the difference between liveness and readiness probes in Kubernetes. If both these probes are not implemented carefully, they can degrade the overall operation of a service, for example by causing unnecessary restarts. The third type of probe, the startup probe, is a relatively new feature in Kubernetes. It allows us to avoid setting initialDelaySeconds on liveness or readiness probes and therefore is especially useful if your app startup takes a lot of time. For more details about Kubernetes probes in general and best practices, I can recommend that very interesting article.

A liveness probe is used to decide whether to restart the container or not. If an application is unavailable for any reason, restarting the container sometimes can make sense. On the other hand, a readiness probe is used to decide if a container can handle incoming traffic. If a pod has been recognized as not ready, it is removed from load balancing. Failure of the readiness probe does not result in pod restart. The most typical liveness or readiness probe for web applications is realized via an HTTP endpoint.

Since subsequent failures of the liveness probe result in pod restart, it should not check the availability of your app integrations. Such things should be verified by the readiness probe.

Configuration Details

The good news is that the most popular Java frameworks like Spring Boot or Quarkus provide an auto-configured implementation of both Kubernetes probes. They follow best practices, so we usually don’t have to take about basics. However, in Spring Boot besides including the Actuator module you need to enable them with the following property:

management:
  endpoint: 
    health:
      probes:
        enabled: true

Since Spring Boot Actuator provides several endpoints (e.g. metrics, traces) it is a good idea to expose it on a different port than a default (usually 8080). Of course, the same rule applies to other popular Java frameworks. On the other hand, a good practice is to check your main app port – especially in the readiness probe. Since it defines if our app is ready to process incoming requests, it should listen also on the main port. It looks just the opposite with the liveness probe. If let’s say the whole working thread pool is busy, I don’t want to restart my app. I just don’t want to receive incoming traffic for some time.

We can also customize other aspects of Kubernetes probes. Let’s say that our app connects to the external system, but we don’t verify that integration in our readiness probe. It is not critical and doesn’t have a direct impact on our operational status. Here’s a configuration that allows us to include in the probe only the selected set of integrations (1) and also exposes readiness on the main server port (2).

spring:
  application:
    name: sample-spring-boot-on-kubernetes
  data:
    mongodb:
      host: ${MONGO_URL}
      port: 27017
      username: ${MONGO_USERNAME}
      password: ${MONGO_PASSWORD}
      database: ${MONGO_DATABASE}
      authentication-database: admin

management:
  endpoint.health:
    show-details: always
    group:
      readiness:
        include: mongo # (1)
        additional-path: server:/readiness # (2)
    probes:
      enabled: true
  server:
    port: 8081

Hardly ever our application is able to exist without any external solutions like databases, message brokers, or just other applications. When configuring the readiness probe we should consider connection settings to that system carefully. Firstly you should consider the situation when external service is not available. How you will handle it? I suggest decreasing these timeouts to lower values as shown below.

spring:
  application:
    name: sample-spring-kotlin-microservice
  datasource:
    url: jdbc:postgresql://postgres:5432/postgres
    username: postgres
    password: postgres123
    hikari:
      connection-timeout: 2000
      initialization-fail-timeout: 0
  jpa:
    database-platform: org.hibernate.dialect.PostgreSQLDialect
  rabbitmq:
    host: rabbitmq
    port: 5672
    connection-timeout: 2000

Choose The Right JDK

If you have already built images with Dockerfile it is possible that you were using the official OpenJDK base image from the Docker Hub. However, currently, the announcement on the image site says that it is officially deprecated and all users should find suitable replacements. I guess it may be quite confusing, so you will find a detailed explanation of the reasons here.

All right, so let’s consider which alternative we should choose. Different vendors provide several replacements. If you are looking for a detailed comparison between them you should go to the following site. It recommends using Eclipse Temurin in the 21 version.

On the other hand, the most popular image build tools like Jib or Cloud Native Buildpacks automatically choose a vendor for you. By default, Jib uses Eclipse Temurin, while Paketo Buildpacks uses Bellsoft Liberica implementation. Of course, you can easily override these settings. I think it might make sense if you, for example, run your app in the environment matched to the JDK provider, like AWS and Amazon Corretto.

Let’s say we use Paketo Buildpacks and Skaffold for deploying Java apps on Kubernetes. In order to replace a default Bellsoft Liberica buildpack with another one we just need to set it literally in the buildpacks section. Here’s an example that leverages the Amazon Corretto buildpack.

apiVersion: skaffold/v2beta22
kind: Config
metadata:
  name: sample-spring-boot-on-kubernetes
build:
  artifacts:
    - image: piomin/sample-spring-boot-on-kubernetes
      buildpacks:
        builder: paketobuildpacks/builder:base
        buildpacks:
          - paketo-buildpacks/amazon-corretto
          - paketo-buildpacks/java
        env:
          - BP_JVM_VERSION=21

We can also easily test the performance of our apps with different JDK vendors. If you are looking for an example of such a comparison you can read my article describing such tests and results. I measured the different JDK performance for the Spring Boot 3 app that interacts with the Mongo database using several available Paketo Java Buildpacks.

Consider Migration To Native Compilation 

Native compilation is a real “game changer” in the Java world. But I can bet that not many of you use it – especially on production. Of course, there were (and still are) numerous challenges in the migration of existing apps into the native compilation. The static code analysis performed by the GraalVM during build time can result in errors like ClassNotFound, as or MethodNotFound. To overcome these challenges, we need to provide several hints to let GraalVM know about the dynamic elements of the code. The number of those hints usually depends on the number of libraries and the general number of language features used in the app.

The Java frameworks like Quarkus or Micronaut try to address challenges related to a native compilation by design. For example, they are avoiding using reflection wherever possible. Spring Boot has also improved native compilation support a lot through the Spring Native project. So, my advice in this area is that if you are creating a new application, prepare it the way it will be ready for native compilation. For example, with Quarkus you can simply generate a Maven configuration that contains a dedicated profile for building a native executable.

<profiles>
  <profile>
    <id>native</id>
    <activation>
      <property>
        <name>native</name>
      </property>
    </activation>
    <properties>
      <skipITs>false</skipITs>
      <quarkus.package.type>native</quarkus.package.type>
    </properties>
  </profile>
</profiles>

Once you add it you can just a native build with the following command:

$ mvn clean package -Pnative

Then you can analyze if there are any issues during the build. Even if you do not run native apps on production now (for example your organization doesn’t approve it), you should place GraalVM compilation as a step in your acceptance pipeline. You can easily build the Java native image for your app with the most popular frameworks. For example, with Spring Boot you just need to provide the following configuration in your Maven pom.xml as shown below:

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>build-info</goal>
        <goal>build-image</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <image>
      <builder>paketobuildpacks/builder:tiny</builder>
      <env>
        <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
        <BP_NATIVE_IMAGE_BUILD_ARGUMENTS>
          --allow-incomplete-classpath
        </BP_NATIVE_IMAGE_BUILD_ARGUMENTS>
      </env>
    </image>
  </configuration>
</plugin>

Configure Logging Properly

Logging is probably not the first thing you are thinking about when writing your Java apps. However, at the global scope, it becomes very important since we need to be able to collect, store data, and finally search and present the particular entry quickly. The best practice is to write your application logs to the standard output (stdout) and standard error (stderr) streams.

Fluentd is a popular open-source log aggregator that allows you to collect logs from the Kubernetes cluster, process them, and then ship them to a data storage backend of your choice. It integrates seamlessly with Kubernetes deployments. Fluentd tries to structure data as JSON to unify logging across different sources and destinations. Assuming that probably the best way is to prepare logs in this format. With JSON format we can also easily include additional fields for tagging logs and then easily search them in the visual tool with various criteria.

In order to format our logs to JSON readable by Fluentd we may include the Logstash Logback Encoder library in Maven dependencies.

<dependency>
   <groupId>net.logstash.logback</groupId>
   <artifactId>logstash-logback-encoder</artifactId>
   <version>7.2</version>
</dependency>

Then we just need to set a default console log appender for our Spring Boot application in the file logback-spring.xml.

<configuration>
    <appender name="consoleAppender" class="ch.qos.logback.core.ConsoleAppender">
        <encoder class="net.logstash.logback.encoder.LogstashEncoder"/>
    </appender>
    <logger name="jsonLogger" additivity="false" level="DEBUG">
        <appender-ref ref="consoleAppender"/>
    </logger>
    <root level="INFO">
        <appender-ref ref="consoleAppender"/>
    </root>
</configuration>

Should we avoid using additional log appenders and just print logs just to the standard output? From my experience, the answer is – no. You can still alternative mechanisms for sending the logs. Especially if you use more than one tool for collecting logs in your organization – for example internal stack on Kubernetes and global stack outside. Personally, I’m using a tool that helps me to resolve performance problems e.g. message broker as a proxy. In Spring Boot we easily use RabbitMQ for that. Just include the following starter:

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>

Then you need to provide a similar appender configuration in the logback-spring.xml:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>

  <springProperty name="destination" source="app.amqp.url" />

  <appender name="AMQP"
		class="org.springframework.amqp.rabbit.logback.AmqpAppender">
    <layout>
      <pattern>
{
  "time": "%date{ISO8601}",
  "thread": "%thread",
  "level": "%level",
  "class": "%logger{36}",
  "message": "%message"
}
      </pattern>
    </layout>

    <addresses>${destination}</addresses>	
    <applicationId>api-service</applicationId>
    <routingKeyPattern>logs</routingKeyPattern>
    <declareExchange>true</declareExchange>
    <exchangeName>ex_logstash</exchangeName>

  </appender>

  <root level="INFO">
    <appender-ref ref="AMQP" />
  </root>

</configuration>

Create Integration Tests

Ok, I know – it’s not directly related to Kubernetes. However, since we use Kubernetes to manage and orchestrate containers, we should also run integration tests on the containers. Fortunately, with Java frameworks, we can simplify that process a lot. For example, Quarkus allows us to annotate the test with @QuarkusIntegrationTest. It is a really powerful solution in conjunction with the Quarkus containers build feature. We can run the tests against an already-built image containing the app. First, let’s include the Quarkus Jib module:

<dependency>
   <groupId>io.quarkus</groupId>
   <artifactId>quarkus-container-image-jib</artifactId>
</dependency>

Then we have to enable container build by setting the quarkus.container-image.build property to true in the application.properties file. In the test class, we can use @TestHTTPResource and @TestHTTPEndpoint annotations to inject the test server URL. Then we are creating a client with the RestClientBuilder and call the service started on the container. The name of the test class is not accidental. In order to be automatically detected as the integration test, it has the IT suffix.

@QuarkusIntegrationTest
public class EmployeeControllerIT {

    @TestHTTPEndpoint(EmployeeController.class)
    @TestHTTPResource
    URL url;

    @Test
    void add() {
        EmployeeService service = RestClientBuilder.newBuilder()
                .baseUrl(url)
                .build(EmployeeService.class);
        Employee employee = new Employee(1L, 1L, "Josh Stevens", 
                                         23, "Developer");
        employee = service.add(employee);
        assertNotNull(employee.getId());
    }

    @Test
    public void findAll() {
        EmployeeService service = RestClientBuilder.newBuilder()
                .baseUrl(url)
                .build(EmployeeService.class);
        Set<Employee> employees = service.findAll();
        assertTrue(employees.size() >= 3);
    }

    @Test
    public void findById() {
        EmployeeService service = RestClientBuilder.newBuilder()
                .baseUrl(url)
                .build(EmployeeService.class);
        Employee employee = service.findById(1L);
        assertNotNull(employee.getId());
    }
}

You can find more details about that process in my previous article about advanced testing with Quarkus. The final effect is visible in the picture below. When we run the tests during the build with the mvn clean verify command, our test is executed after building the container image.

That Quarkus feature is based on the Testcontainers framework. We can also use Testcontainers with Spring Boot. Here’s the sample test of the Spring REST app and its integration with the PostgreSQL database.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class PersonControllerTests {

    @Autowired
    TestRestTemplate restTemplate;

    @Container
    static PostgreSQLContainer<?> postgres = 
       new PostgreSQLContainer<>("postgres:15.1")
            .withExposedPorts(5432);

    @DynamicPropertySource
    static void registerMySQLProperties(DynamicPropertyRegistry registry) {
        registry.add("spring.datasource.url", postgres::getJdbcUrl);
        registry.add("spring.datasource.username", postgres::getUsername);
        registry.add("spring.datasource.password", postgres::getPassword);
    }

    @Test
    @Order(1)
    void add() {
        Person person = Instancio.of(Person.class)
                .ignore(Select.field("id"))
                .create();
        person = restTemplate.postForObject("/persons", person, Person.class);
        Assertions.assertNotNull(person);
        Assertions.assertNotNull(person.getId());
    }

    @Test
    @Order(2)
    void updateAndGet() {
        final Integer id = 1;
        Person person = Instancio.of(Person.class)
                .set(Select.field("id"), id)
                .create();
        restTemplate.put("/persons", person);
        Person updated = restTemplate.getForObject("/persons/{id}", Person.class, id);
        Assertions.assertNotNull(updated);
        Assertions.assertNotNull(updated.getId());
        Assertions.assertEquals(id, updated.getId());
    }

}

Final Thoughts

I hope that this article will help you avoid some common pitfalls when running Java apps on Kubernetes. Treat it as a summary of other people’s recommendations I found in similar articles and my private experience in that area. Maybe you will find some of those rules are quite controversial. Feel free to share your opinions and feedback in the comments. It will also be valuable for me. If you like this article, once again, I recommend reading another one from my blog – more focused on running microservices-based apps on Kubernetes – Best Practices For Microservices on Kubernetes. But it also contains several useful (I hope) recommendations.

The post Best Practices for Java Apps on Kubernetes appeared first on Piotr's TechBlog.

If you are interested in more details about Spring Boot native compilation please refer to my article Microservices on Knative with Spring Boot and GraalVM.

Prerequisites

Native compilation for Java is a memory-intensive process. Therefore, we need to reserve at least 8GB for our Kubernetes cluster. We also have to install Tekton and Knative there, so it is worth having even more memory.

1. Install Knative Serving – we will use the latest version of Knative (1.3). Go to the following site for the installation manual. Once you did that, you can just verify if it works with the following command:

$ kubectl get pods -n knative-serving

2. Install Tekton – you can go to that site for more details. However, there is just a single command to install it:

$ kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then go to the callme-service directory. After that, you should just follow my instructions.

Spring Boot for Native GraalVM

In order to expose the REST endpoints, we need to include Spring Boot Starter Web. Our service also stored data in the H2 database, so we include Spring Boot Starter Data JPA. The last dependency is for native compilation support. The current version of Spring Native is 0.11.3.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.experimental</groupId>
  <artifactId>spring-native</artifactId>
  <version>0.11.3</version>
</dependency>

Let’s switch to the code. It is our model class exposed by the REST endpoint. It contains the current date, the name of the Kubernetes pod, and the version number.

@Entity
public class Callme {

   @Id
   @GeneratedValue
   private Integer id;
   @Temporal(TemporalType.TIMESTAMP)
   private Date addDate;
   private String podName;
   private String version;

   // getters, setters, constructor ...

}

There is a single endpoint that creates an event, stores it in the database and returns it as a result. The name of the pod and the name of the namespace are taken directly from Kubernetes Deployment. We use the version number from Maven pom.xml as the application version.

@RestController
@RequestMapping("/callme")
public class CallmeController {

   @Value("${spring.application.name}")
   private String appName;
   @Value("${POD_NAME}")
   private String podName;
   @Value("${POD_NAMESPACE}")
   private String podNamespace;
   @Autowired
   private CallmeRepository repository;
   @Autowired(required = false)
   BuildProperties buildProperties;

   @GetMapping("/ping")
   public String ping() {
      Callme c = repository.save(new Callme(new Date(), podName,
            buildProperties != null ? buildProperties.getVersion() : null));
      return appName +
            " v" + c.getVersion() +
            " (id=" + c.getId() + "): " +
            podName +
            " in " + podNamespace;
   }

}

In order to use the Maven version, we need to generate the build-info.properties file during the build. Therefore we should add the build-info goal in the spring-boot-maven-plugin execution properties. If you would like to build a native image locally just set the configuration environment property BP_NATIVE_IMAGE to true. Then you can just run the command mvn spring-boot:build-image.

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>build-info</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <image>
      <builder>paketobuildpacks/builder:tiny</builder>
      <env>
        <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
      </env>
    </image>
  </configuration>
</plugin>

Create Tekton Pipelines

Our pipeline consists of three tasks. In the first of them, we are cloning the source code repository with the Spring Boot application. In the second step, we are building the image natively on Kubernetes using the Cloud Native Builpacks task. After building the image we are pushing to the remote registry. Finally, we are running the image on Kubernetes as the Knative service.

Firstly, we need to install the Tekton git-clone task:

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/git-clone/0.4/git-clone.yaml

We also need the buildpacks task that allows us to run Cloud Native Buildpacks on Kubernetes:

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/buildpacks/0.3/buildpacks.yaml

Finally, we are installing the kubernetes-actions task to deploy Knative Service using the YAML manifest.

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kubernetes-actions/0.2/kubernetes-actions.yaml

All our tasks are ready. Let’s just verify it:

$ kubectl get task                     
NAME                 AGE
buildpacks           1m
git-clone            1m
kubernetes-actions   33s

Finally, we are going to create a Tekton pipeline. In the buildpacks task reference, we need to set several parameters. Since we have two Maven modules in the repository, we first need to set the working directory to the callme-service (1). Also, we use Paketo Buildpacks, so we change the default builder image to the paketobuildpacks/builder:base (2). Finally, we need to enable native build for Cloud Native Buildpacks. In order to do that, we should set the environment variable BP_NATIVE_IMAGE to true (3). After building and pushing the image we can deploy it on Kubernetes (4).

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: build-spring-boot-pipeline
spec:
  params:
    - description: image URL to push
      name: image
      type: string
  tasks:
    - name: fetch-repository
      params:
        - name: url
          value: 'https://github.com/piomin/sample-spring-boot-graalvm.git'
        - name: subdirectory
          value: ''
        - name: deleteExisting
          value: 'true'
      taskRef:
        kind: Task
        name: git-clone
      workspaces:
        - name: output
          workspace: source-workspace
    - name: buildpacks
      params:
        - name: APP_IMAGE
          value: $(params.image)
        - name: SOURCE_SUBPATH # (1)
          value: callme-service
        - name: BUILDER_IMAGE # (2)
          value: 'paketobuildpacks/builder:base'
        - name: ENV_VARS # (3)
          value:
            - BP_NATIVE_IMAGE=true
      runAfter:
        - fetch-repository
      taskRef:
        kind: Task
        name: buildpacks
      workspaces:
        - name: source
          workspace: source-workspace
        - name: cache
          workspace: cache-workspace
    - name: deploy
      params:
        - name: args # (4)
          value: 
            - apply -f callme-service/k8s/
      runAfter:
        - buildpacks
      taskRef:
        kind: Task
        name: kubernetes-actions
      workspaces:
        - name: manifest-dir
          workspace: source-workspace
  workspaces:
    - name: source-workspace
    - name: cache-workspace

In order to push the image into the remote secure registry, you need to create a Secret containing your username and password.

$ kubectl create secret docker-registry docker-user-pass \
    --docker-username=<USERNAME> \
    --docker-password=<PASSWORD> \
    --docker-server=https://index.docker.io/v1/

After that, you should create ServiceAccount that uses a newly created Secret. As you probably figured out our pipeline uses that ServiceAccount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: buildpacks-service-account
secrets:
  - name: docker-user-pass

Deploy Knative Service with gradual rollouts

Here’s the YAML manifest with our Knative Service for the 1.0 version. It is a very simple definition. The only additional thing we need to do is to inject the name of the pod and namespace into the container.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: callme-service
spec:
  template:
    spec:
      containers:
      - name: callme
        image: piomin/callme-service:1.0
        ports:
          - containerPort: 8080
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

In order to inject the name of the pod and namespace into the container, we use Downward API. This Kubernetes feature is by default disabled on Knative. To enable it we need to add the property kubernetes.podspec-fieldref with value enabled in the config-features ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-features
  namespace: knative-serving
data:
  kubernetes.podspec-fieldref: enabled

Now, let’s run our pipeline for the 1.0 version of our sample application. To run it, you should also create a PersistentVolumeClaim with the name tekton-workspace-pvc.

apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
spec:
  params:
    - name: image
      value: 'piomin/callme-service:1.0'
  pipelineRef:
    name: build-spring-boot-pipeline
  serviceAccountName: buildpacks-service-account
  workspaces:
    - name: source-workspace
      persistentVolumeClaim:
        claimName: tekton-workspace-pvc
      subPath: source
    - name: cache-workspace
      persistentVolumeClaim:
        claimName: tekton-workspace-pvc
      subPath: cache

Finally, it is time to release a new version of our application – 1.1. Firstly, you should change the version number in Maven pom.xml.

We should also change the version number in the k8s/ksvc.yaml manifest. However, the most important thing is related to the annotation serving.knative.dev/rollout-duration. It enables gradual rollouts for Knative Service. The value 300s of this parameter means that our rollout to the latest revision will take exactly 300 seconds. Knative is going to roll out to 1% of traffic first, and then in equal incremental steps for the rest of the assigned traffic. In that case, it will increase the traffic to the latest service by 1% every 3 seconds.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: callme-service
  annotations:
    serving.knative.dev/rollout-duration: "300s"
spec:
  template:
    spec:
      containers:
      - name: callme
        image: piomin/callme-service:1.1
        ports:
          - containerPort: 8080
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

Verify Canary Release with Knative

Let’s verify our canary release process built with Knative and Tekton. Once you run the pipeline for the 1.0 and 1.1 version of our sample application, you can display a list of Knative Service. As you the latest revision is callme-service-00002, but the rollout is still in progress:

We can send some test requests to our Knative Service. For me, Knative is available on localhost:80, since I’m using Kubernetes on the Docker Desktop. The only thing I need to do is to set the name URL of the service in the Host header.

$ curl http://localhost:80/callme/ping \
  -H "Host:callme-service.default.example.com"

Here are some responses. As you see, the first two requests have been processed by the 1.0 version of our application. While the last request by the 1.1 version.

Let’s verify the current percentage traffic distribution between the two revisions. Currently, it is 52% to 1.1 and 48% to 1.0.

Finally, after the rollout procedure is finished you should get the same response as me.

Final Thoughts

As you see the process of canary release with Knative is very simple. You only need to set a single annotation on the Knative Service. You can compare it for example with Argo Rollouts which allows us to perform progressive traffic shifting for a standard Kubernetes Deployment.

The post Canary Release on Kubernetes with Knative and Tekton appeared first on Piotr's TechBlog.