If you want to build and run tests against Kubernetes, you can read my article about integration tests with JUnit. On the other hand, if you are looking for other testing tools for testing in a Kubernetes-native environment you can refer to that article about Testkube.

Introduction

Before we start, let’s do a brief introduction. There are three simple Spring Boot apps that communicate with each other. The first-service app calls the endpoint exposed by the caller-service app, and then the caller-service app calls the endpoint exposed by the callme-service app. The diagram visible below illustrates that architecture.

So in short, our goal is to deploy all the sample apps on Kind during the CircleCI build and then test the communication by calling the endpoint exposed by the first-service through the Kubernetes Service.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. It contains three apps: first-service, caller-service, and callme-service. The main Skaffold config manifest is available in the project root directory. Required Kubernetes YAML manifests are always placed inside the k8s directory. Once you take a look at the source code, you should just follow my instructions. Let’s begin.

Our sample Spring Boot apps are very simple. They are exposing a single “ping” endpoint over HTTP and call “ping” endpoints exposed by other apps. Here’s the @RestController in the first-service app:

@RestController
@RequestMapping("/first")
public class FirstController {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(FirstController.class);

   @Autowired
   Optional<BuildProperties> buildProperties;
   @Autowired
   RestTemplate restTemplate;
   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping")
   public String ping() {
      LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() 
         ? buildProperties.get().getName() : "first-service", version);
      String response = restTemplate.getForObject(
         "http://caller-service:8080/caller/ping", String.class);
      LOGGER.info("Calling: response={}", response);
      return "I'm first-service " + version + ". Calling... " + response;
   }

}

Here’s the @RestController inside the caller-service app. The endpoint is called by the first-service app through the RestTemplate bean.

@RestController
@RequestMapping("/caller")
public class CallerController {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(CallerController.class);

   @Autowired
   Optional<BuildProperties> buildProperties;
   @Autowired
   RestTemplate restTemplate;
   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping")
   public String ping() {
      LOGGER.info("Ping: name={}, version={}", 
         buildProperties.or(Optional::empty), version);
      String response = restTemplate.getForObject(
         "http://callme-service:8080/callme/ping", String.class);
      LOGGER.info("Calling: response={}", response);
      return "I'm caller-service " + version + ". Calling... " + response;
   }

}

Finally, here’s the @RestController inside the callme-service app. It also exposes a single GET /callme/ping endpoint called by the caller-service app:

@RestController
@RequestMapping("/callme")
public class CallmeController {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(CallmeController.class);
   private static final String INSTANCE_ID = UUID.randomUUID().toString();
   private Random random = new Random();

   @Autowired
   Optional<BuildProperties> buildProperties;
   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping")
   public String ping() {
      LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() 
         ? buildProperties.get().getName() : "callme-service", version);
      return "I'm callme-service " + version;
   }

}

Build and Deploy Images with Skaffold and Jib

Firstly, let’s take a look at the main Maven pom.xml in the project root directory. We use the latest version of Spring Boot and the latest LTS version of Java for compilation. All three app modules inherit settings from the parent pom.xml. In order to build the image with Maven we are including jib-maven-plugin. Since it is still using Java 17 in the default base image, we need to override this behavior with the <from>.<image> tag. We will declare eclipse-temurin:21-jdk-ubi9-minimal as the base image. Note that jib-maven-plugin is activated only if we enable the jib Maven profile during the build.

<modelVersion>4.0.0</modelVersion>

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>3.2.0</version>
  <relativePath />
</parent>

<groupId>pl.piomin.services</groupId>
<artifactId>sample-istio-services</artifactId>
<version>1.1.0</version>
<packaging>pom</packaging>

<properties>
  <java.version>21</java.version>
</properties>

<modules>
  <module>caller-service</module>
  <module>callme-service</module>
  <module>first-service</module>
</modules>

<profiles>
  <profile>
    <id>jib</id>
    <build>
      <plugins>
        <plugin>
          <groupId>com.google.cloud.tools</groupId>
          <artifactId>jib-maven-plugin</artifactId>
          <version>3.4.0</version>
          <configuration>
            <from>
              <image>eclipse-temurin:21-jdk-ubi9-minimal</image>
            </from>
          </configuration>
        </plugin>
      </plugins>
    </build>
  </profile>
</profiles>

Now, let’s take a look at the main skaffold.yaml file. Skaffold builds the image using Jib support and deploys all three apps on Kubernetes using manifests available in the k8s/deployment.yaml file inside each app module. Skaffold disables JUnit tests for Maven and activates the jib profile. It is also able to deploy Istio objects after activating the istio Skaffold profile. However, we won’t use it today.

apiVersion: skaffold/v4beta5
kind: Config
metadata:
  name: simple-istio-services
build:
  artifacts:
    - image: piomin/first-service
      jib:
        project: first-service
        args:
          - -Pjib
          - -DskipTests
    - image: piomin/caller-service
      jib:
        project: caller-service
        args:
          - -Pjib
          - -DskipTests
    - image: piomin/callme-service
      jib:
        project: callme-service
        args:
          - -Pjib
          - -DskipTests
  tagPolicy:
    gitCommit: {}
manifests:
  rawYaml:
    - '*/k8s/deployment.yaml'
deploy:
  kubectl: {}
profiles:
  - name: istio
    manifests:
      rawYaml:
        - k8s/istio-*.yaml
        - '*/k8s/deployment-versions.yaml'
        - '*/k8s/istio-*.yaml'

Here’s the typical Deployment for our apps. The app is running on the 8080 port.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: first-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: first-service
  template:
    metadata:
      labels:
        app: first-service
    spec:
      containers:
        - name: first-service
          image: piomin/first-service
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"

For testing purposes, we need to expose the first-service outside of the Kind cluster. In order to do that, we will use the Kubernetes NodePort Service. Our app will be available under the 30000 port.

apiVersion: v1
kind: Service
metadata:
  name: first-service
  labels:
    app: first-service
spec:
  type: NodePort
  ports:
  - port: 8080
    name: http
    nodePort: 30000
  selector:
    app: first-service

How It Works

In this section, we will discuss how we would run the whole process locally. Of course, our goal is to configure it as the CircleCI pipeline. In order to expose the Kubernetes Service outside Kind we need to define the externalPortMappings section in the configuration manifest. As you probably remember, we are exposing our app under the 30000 port. The following file is available in the repository under the k8s/kind-cluster-test.yaml path:

apiVersion: kind.x-k8s.io/v1alpha4
kind: Cluster
nodes:
  - role: control-plane
    extraPortMappings:
      - containerPort: 30000
        hostPort: 30000
        listenAddress: "0.0.0.0"
        protocol: tcp

Assuming we already installed kind CLI on our machine, we need to execute the following command to create a new cluster:

$ kind create cluster --name c1 --config k8s/kind-cluster-test.yaml

You should have the same result as visible on my screen:

We have a single-node Kind cluster ready. There is a single c1-control-plane container running on Docker. As you see, it exposes 30000 port outside of the cluster:

The Kubernetes context is automatically switched to kind-c1. So now, we just need to run the following command from the repository root directory to build and deploy the apps:

$ skaffold run

If you see a similar output in the skaffold run logs, it means that everything works fine.

We can verify a list of Kubernetes services. The first-service is exposed under the 30000 port as expected.

$ kubectl get svc
NAME             TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
caller-service   ClusterIP   10.96.47.193   <none>        8080/TCP         2m24s
callme-service   ClusterIP   10.96.98.53    <none>        8080/TCP         2m24s
first-service    NodePort    10.96.241.11   <none>        8080:30000/TCP   2m24s

Assuming you have already installed the Grafana k6 tool locally, you may run load tests using the following command:

$ k6 run first-service/src/test/resources/k6/load-test.js

That’s all. Now, let’s define the same actions with the CircleCI workflow.

Test Kubernetes Deployment with the CircleCI Workflow

The CircleCI config.yml file should be placed in the .circle directory. We are doing two things in our pipeline. In the first step, we are executing Maven unit tests without the Kubernetes cluster. That’s why we need a standard executor with OpenJDK 21 and the maven ORB. In order to run Kind during the CircleCI build, we need to have access to the Docker daemon. Therefore, we use the latest version of the ubuntu-2204 machine.

version: 2.1

orbs:
  maven: circleci/maven@1.4.1

executors:
  jdk:
    docker:
      - image: 'cimg/openjdk:21.0'
  machine_executor_amd64:
    machine:
      image: ubuntu-2204:2023.10.1
    environment:
      architecture: "amd64"
      platform: "linux/amd64"

After that, we can proceed to the job declaration. The name of our job is deploy-k8s. It uses the already-defined machine executor. Let’s discuss the required steps after running a standard checkout command:

1. We need to install the kubectl CLI and copy it to the /usr/local/bin directory. Skaffold uses kubectl to interact with the Kubernetes cluster.
2. After that, we have to install the skaffold CLI
3. Our job also requires the kind CLI to be able to create or delete Kind clusters on Docker…
4. … and the Grafana k6 CLI to run load tests against the app deployed on the cluster
5. There is a good chance that this step won’t required once CircleCI releases a new version of ubuntu-2204 machine (probably 2024.1.1 according to the release strategy). For now, ubuntu-2204 provides OpenJDK 17, so we need to install OpenJDK 17 to successfully build the app from the source code
6. After installing all the required tools we can create a new Kubernetes with the kind create cluster command.
7. Once a cluster is ready, we can deploy our apps using the skaffold run command.
8. Once the apps are running on the cluster, we can proceed to the tests phase. We are running the test defined inside the first-service/src/test/resources/k6/load-test.js file.
9. After doing all the required steps, it is important to remove the Kind cluster

 jobs:
  deploy-k8s:
    executor: machine_executor_amd64
    steps:
      - checkout
      - run: # (1)
          name: Install Kubectl
          command: |
            curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"
            chmod +x kubectl
            sudo mv ./kubectl /usr/local/bin/kubectl
      - run: # (2)
          name: Install Skaffold
          command: |
            curl -Lo skaffold https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64
            chmod +x skaffold
            sudo mv skaffold /usr/local/bin
      - run: # (3)
          name: Install Kind
          command: |
            [ $(uname -m) = x86_64 ] && curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.20.0/kind-linux-amd64
            chmod +x ./kind
            sudo mv ./kind /usr/local/bin/kind
      - run: # (4)
          name: Install Grafana K6
          command: |
            sudo gpg -k
            sudo gpg --no-default-keyring --keyring /usr/share/keyrings/k6-archive-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys C5AD17C747E3415A3642D57D77C6C491D6AC1D69
            echo "deb [signed-by=/usr/share/keyrings/k6-archive-keyring.gpg] https://dl.k6.io/deb stable main" | sudo tee /etc/apt/sources.list.d/k6.list
            sudo apt-get update
            sudo apt-get install k6
      - run: # (5)
          name: Install OpenJDK 21
          command: |
            java -version
            sudo apt-get update && sudo apt-get install openjdk-21-jdk
            sudo update-alternatives --set java /usr/lib/jvm/java-21-openjdk-amd64/bin/java
            sudo update-alternatives --set javac /usr/lib/jvm/java-21-openjdk-amd64/bin/javac
            java -version
            export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
      - run: # (6)
          name: Create Kind Cluster
          command: |
            kind create cluster --name c1 --config k8s/kind-cluster-test.yaml
      - run: # (7)
          name: Deploy to K8s
          command: |
            export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
            skaffold run
      - run: # (8)
          name: Run K6 Test
          command: |
            kubectl get svc
            k6 run first-service/src/test/resources/k6/load-test.js
      - run: # (9)
          name: Delete Kind Cluster
          command: |
            kind delete cluster --name c1

Here’s the definition of our load test. It has to be written in JavaScript. It defines some thresholds like a % of maximum failed requests or maximum response time for 95% of requests. As you see, we are testing the http://localhost:30000/first/ping endpoint:

import { sleep } from 'k6';
import http from 'k6/http';

export const options = {
  duration: '60s',
  vus: 10,
  thresholds: {
    http_req_failed: ['rate<0.25'],
    http_req_duration: ['p(95)<1000'],
  },
};

export default function () {
  http.get('http://localhost:30000/first/ping');
  sleep(2);
}

Finally, the last part of the CircleCI config file. It defines pipeline workflow. In the first step, we are running tests with Maven. After that, we proceeded to the deploy-k8s job.

workflows:
  build-and-deploy:
    jobs:
      - maven/test:
          name: test
          executor: jdk
      - deploy-k8s:
          requires:
            - test

Once we push a change to the sample Git repository we trigger a new CircleCI build. You can verify it by yourself here in my CircleCI project page.

As you see all the pipeline steps have been finished successfully.

We can display logs for every single step. Here are the logs from the k6 load test step.

There were some errors during the warm-up. However, the test shows that our scenario works on the Kubernetes cluster.

Final Thoughts

CircleCI is one of the most popular CI/CD platforms. Personally, I’m using it for running builds and tests for all my demo repositories on GitHub. For the sample projects dedicated to the Kubernetes cluster, I want to verify such steps as building images with Jib, Kubernetes deployment scripts, or Skaffold configuration. This article shows how to easily perform such tests with CircleCI and Kubernetes cluster running on Kind. Hope it helps

The post Kubernetes Testing with CircleCI, Kind, and Skaffold appeared first on Piotr's TechBlog.

If you are interested in testing on Kubernetes you can also read my article about integration tests with JUnit. There is also a post about contract testing on Kubernetes with Microcks available here.

Introduction

Testkube is a Kubernetes native test orchestration and execution framework. It allows us to run automated tests inside the Kubernetes cluster. It supports several popular testing or build tools like JMeter, Grafana k6, and Maven. We can easily integrate with the CI/CD pipelines or GitOps workflows. We can manage Kubetest by using the CRD objects directly, with the CLI, or through the UI dashboard. Let’s check how it works.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. It contains only a single app. Once you clone it you can go to the src/test directory. You will find there both the JUnit tests written in Java and the k6 tests written in JavaScript. After that, you should just follow my instructions. Let’s begin.

Run Kubetest on Kubernetes

In the first step, we are going to install Testkube on Kubernetes using its Helm chart. Let’s add the kubeshop Helm repository and fetch latest charts info:

$ helm repo add kubeshop https://kubeshop.github.io/helm-charts
$ helm repo update

Then, we can install Testkube in the testkube namespace by executing the following helm command:

$ helm install testkube kubeshop/testkube \
    --create-namespace --namespace testkube

Once the installation is finished, we can verify a list of running in the testkube namespace. The testkube-api-server and testkube-dashboard are the most important components. However, there are also some additional tools installed like Mongo database or Minio.

$ oc get po -n testkube
NAME                                                    READY   STATUS    RESTARTS        AGE
testkube-api-server-d4d7f9f8b-xpxc9                     1/1     Running   1 (6h17m ago)   6h18m
testkube-dashboard-64578877c7-xghsz                     1/1     Running   0               6h18m
testkube-minio-testkube-586877d8dd-8pmmj                1/1     Running   0               6h18m
testkube-mongodb-dfd8c7878-wzkbp                        1/1     Running   0               6h18m
testkube-nats-0                                         3/3     Running   0               6h18m
testkube-nats-box-567d94459d-6gc4d                      1/1     Running   0               6h18m
testkube-operator-controller-manager-679b998f58-2sv2x   2/2     Running   0               6h18m

We can also install testkube CLI on our laptop. It is not required, but we will use it during the exercise just try the full spectrum of options. You can find CLI installation instructions here. I’m installing it on macOS:

$ brew install testkube

Once the installation is finished, you can run the testkube version command to see that warm “Hello” screen

Run Maven Tests with Testkube

Firstly, let’s take a look at the JUnit tests inside our sample Spring Boot app. We are using the TestRestTemplate bean to call all the exposed REST endpoints exposed. There are three JUnit tests for testing adding, getting, and removing the Person objects.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestMethodOrder(MethodOrderer.OrderAnnotation::class)
class PersonControllerTests {

   @Autowired
   lateinit var template: TestRestTemplate

   @Test
   @Order(1)
   fun shouldAddPerson() {
      var person = Instancio.of(Person::class.java)
         .ignore(Select.field("id"))
         .create()
      person = template
         .postForObject("/persons", person, Person::class.java)
      Assertions.assertNotNull(person)
      Assertions.assertNotNull(person.id)
      Assertions.assertEquals(1001, person.id)
   }

   @Test
   @Order(2)
   fun shouldUpdatePerson() {
      var person = Instancio.of(Person::class.java)
         .set(Select.field("id"), 1)
         .create()
      template.put("/persons", person)
      var personRemote = template
         .getForObject("/persons/{id}", Person::class.java, 1)
      Assertions.assertNotNull(personRemote)
      Assertions.assertEquals(person.age, personRemote.age)
   }

   @Test
   @Order(3)
   fun shouldDeletePerson() {
      template.delete("/persons/{id}", 1)
      val person = template
         .getForObject("/persons/{id}", Person::class.java, 1)
      Assertions.assertNull(person)
   }

}

We are using Maven as a build tool. The current version of Spring Boot is 3.2.0. The version of JDK used for the compilation is 17. Here’s the fragment of our pom.xml in the repository root directory:

<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.2.0</version>
  </parent>
  <groupId>pl.piomin.services</groupId>
  <artifactId>sample-spring-kotlin-microservice</artifactId>
  <version>1.5.3</version>

  <properties>
    <java.version>17</java.version>
    <kotlin.version>1.9.21</kotlin.version>
  </properties>

  <dependencies>
    ...   
    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-test</artifactId>
      <scope>test</scope>
    </dependency>
    <dependency>
      <groupId>org.instancio</groupId>
      <artifactId>instancio-junit</artifactId>
      <version>3.6.0</version>
      <scope>test</scope>
    </dependency>
  </dependencies>
</project>

Testkube provides the Executor CRD for defining a way of running each test. There are several default executors per each type of supported build or test tool. We can display a list of provided executors by running the testkube get executor command. You will see the list of all tools supported by Testkube. Of course, the most interesting executors for us are k6-executor and maven-executor.

$ testkube get executor

Context:  (1.16.8)   Namespace: testkube
----------------------------------------

  NAME                 | URI | LABELS
-----------------------+-----+-----------------------------------
  artillery-executor   |     |
  curl-executor        |     |
  cypress-executor     |     |
  ginkgo-executor      |     |
  gradle-executor      |     |
  jmeter-executor      |     |
  jmeterd-executor     |     |
  k6-executor          |     |
  kubepug-executor     |     |
  maven-executor       |     |
  playwright-executor  |     |
  postman-executor     |     |
  soapui-executor      |     |
  tracetest-executor   |     |
  zap-executor         |     |

By default, maven-executor uses JDK 11 for running Maven tests. Moreover, it still doesn’t provide images for running tests against JDK19+. For me, this is quite a big drawback since the latest LTS version of Java is 21. The maven-executor-jdk17 Executor contains the name of the running image (1) and a list of supported test types (2).

apiVersion: executor.testkube.io/v1
kind: Executor
metadata:
  name: maven-executor-jdk17
  namespace: testkube
spec:
  args:
    - '--settings'
    - <settingsFile>
    - <goalName>
    - '-Duser.home'
    - <mavenHome>
  command:
    - mvn
  content_types:
    - git-dir
    - git
  executor_type: job
  features:
    - artifacts
  # (1)
  image: kubeshop/testkube-maven-executor:jdk17 
  meta:
    docsURI: https://kubeshop.github.io/testkube/test-types/executor-maven
    iconURI: maven
  # (2)
  types:
    - maven:jdk17/project
    - maven:jdk17/test
    - maven:jdk17/integration-test

Finally, we just need to define the Test object that references to maven-executor-jdk17 by the type parameter. Of course, we also need to set the address of the Git repository and the name of the branch.

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  name: sample-spring-kotlin
  namespace: testkube
spec:
  content:
    repository:
      branch: master
      type: git
      uri: https://github.com/piomin/sample-spring-kotlin-microservice.git
    type: git
  type: maven:jdk17/test

Finally, we can run the sample-spring-kotlin test using the following command:

$ testkube run test sample-spring-kotlin

Using UI Dashboard

First of all, let’s expose the Testkube UI dashboard on the local port. The dashboard also requires a connection to the testkube-api-server from the web browser. After exposing the dashboard with the following port-forward command we can access it under the http://localhost:8080 address:

$ kubectl port-forward svc/testkube-dashboard 8080 -n testkube
$ kubectl port-forward svc/testkube-api-server 8088 -n testkube

Once we access the Testkube dashboard we will see a list of all defined tests:

Then, we can click the selected tile with the test to see the details. You will be redirected to the history of previous executions available in the “Recent executions” tab. There are six previous executions of our sample-spring-kotlin test. Two of them were finished successfully, the four others were failed.

Let’s take a look at the logs of the last one execution. As you see, all three JUnit tests were successful.

Run Load Tests with Testkube and Grafana k6

In this section, we will create the tests for the instance of our sample app running on Kubernetes. So, in the first step, we need to deploy the app. Here’s the Deployment manifest. We can apply it to the default namespace. The manifest uses the latest image of the sample app available in the registry under the quay.io/pminkows/sample-kotlin-string:1.5.3 address.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  labels:
    app: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring
        image: quay.io/pminkows/sample-kotlin-spring:1.5.3
        ports:
        - containerPort: 8080

Let’s also create the Kubernetes Service that exposes app pods internally:

apiVersion: v1
kind: Service
metadata:
  name: sample-kotlin-spring
spec:
  selector:
    app: sample-kotlin-spring
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080

After that, we can proceed to the Test manifest. This time, we don’t have to override the default executor, since the k6 version is not important. The test source is located inside the sample Git repository in the src/test/resources/k6/load-tests-get.js (1) file in the master branch. In that case, the repository type is git (2). The k6 test should run for 5 seconds and should use 5 concurrent threads (3). We also need to set the address of a target service as the PERSONS_URI environment variable (4). Of course, we are testing through the Kubernetes Service visible internally under the sample-kotlin-spring.default.svc host and port 8080. The type of the test is k6/script (5).

apiVersion: tests.testkube.io/v3
kind: Test
metadata:
  labels:
    executor: k6-executor
    test-type: k6-script
  name: load-tests-gets
  namespace: testkube
spec:
  content:
    repository:
      branch: master
      # (1)
      path: src/test/resources/k6/load-tests-get.js
      # (2) 
      type: git
      uri: https://github.com/piomin/sample-spring-kotlin-microservice.git
    type: git
  executionRequest:
    # (3)
    args:
      - '-u'
      - '5'
      - '-d'
      - 10s
    # (4)
    variables:
      PERSONS_URI:
        name: PERSONS_URI
        type: basic
        value: http://sample-kotlin-spring.default.svc:8080
        valueFrom: {}
  # (5)
  type: k6/script

Let’s take a look at the k6 test file written in JavaScript. As I mentioned before, you can find it in the src/test/resources/k6/load-tests-get.js file. The test calls the GET /persons/{id} endpoint. It sets the random number between 1 and 1000 as the id path parameter and reads a target service URL from the PERSONS_URI environment variable.

import http from 'k6/http';
import { check } from 'k6';
import { randomIntBetween } from 'https://jslib.k6.io/k6-utils/1.2.0/index.js';

export default function () {
  const id = randomIntBetween(1, 1000);
  const res = http.get(`${__ENV.PERSONS_URI}/persons/${id}`);
  check(res, {
    'is status 200': (res) => res.status === 200,
    'body size is > 0': (r) => r.body.length > 0,
  });
}

Finally, we can run the load-tests-gets test with the following command:

$ testkube run test load-tests-gets

The same as for the Maven test we can verify the execution history in the Testkube dashboard:

We can also display all the logs from the test:

Final Thoughts

Testkube provides a unified way to run Kubernetes tests for the several most popular testing tools. It may be a part of your CI/CD pipeline or a GitOps process. Honestly, I’m still not very convinced if I need a dedicated Kubernetes-native solution for automated tests, instead e.g. a stage in my pipeline that runs test commands. However, you can also use Testkube to execute load or integration tests against the app running on Kubernetes. It is possible to schedule them periodically. Thanks to that you can verify your apps continuously using a single, central tool.

The post Testing Java Apps on Kubernetes with Testkube appeared first on Piotr's TechBlog.

There are a lot of articles about Quarkus on my blog. If you don’t have any experience with that framework and you are looking for something to begin you can read, for example, the following article. Also, there are a lot of simple guides to the Quarkus features available on its website here.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then you need to go to the person-virtual-service directory. After that, just follow my instructions.

Prerequisites

Before we start, we need to install several tools on the local machine. First of all, you have to install JDK 19. There are also some other tools I’m using in that exercise:

• Docker – our sample Quarkus app connects to the Postgresql database. We can easily run Postgres on Docker automatically using the Quarkus Dev Services feature
• JProfiler – in order to visualize and observe the thread pool
• k6 – a javascript-based tool for load testing

Use Case

We will build a simple REST application using the Quarkus framework. It will connect to the Postgres database and expose some endpoints for adding data and the basic search operations. Apart from enabling virtual threads support in Quarkus, our main goal is not to block the thread. In the virtual threads nomenclature, this thread is called “carrier thread”. It is the platform thread responsible for executing a virtual thread. It might be blocked by e.g. JDBC client driver. In order to avoid it, we should use non-blocking clients.

Dependencies

We need to include two Quarkus modules into the Maven dependencies. The first of them is Quarkus Resteasy Reactive which provides an implementation of JAX-RS specification and allows us to create reactive REST services. The quarkus-reactive-pg-client module provides an implementation of a reactive driver for the Postgres database.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-resteasy-reactive-jackson</artifactId>
</dependency>
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-reactive-pg-client</artifactId>
</dependency>

We should also set JDK 19 as a default compiler. Since virtual threads are available as a preview feature we need to set --enable-preview as a JVM argument.

<plugin>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-maven-plugin</artifactId>
  <version>${quarkus.version}</version>
  <extensions>true</extensions>
  <executions>
    <execution>
      <goals>
        <goal>build</goal>
        <goal>generate-code</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <source>19</source>
    <target>19</target>
    <compilerArgs>
      <arg>--enable-preview</arg>
    </compilerArgs>
    <jvmArgs>--enable-preview --add-opens java.base/java.lang=ALL-UNNAMED</jvmArgs>
  </configuration>
</plugin>

I’m also generating some test data for load tests. There is a really useful library for that – Datafaker.

<dependency>
  <groupId>net.datafaker</groupId>
  <artifactId>datafaker</artifactId>
  <version>1.6.0</version>
</dependency>

Enable Virtual Threads for Reactive Services

The Quarkus Resteasy Reactive module works in a non-blocking, event-loop style way. Instead of using the “event loop”, we need to force the execution of an endpoint handler on a new virtual thread. To do that we need to annotate the REST endpoint with the @RunOnVirtualThread. The rest of the code may be pretty similar to the blocking-style way of building services. Of course, instead of blocking the database driver, we will use its reactive alternative. The implementation is provided inside the PersonRepositoryAsyncAwait class.

@Path("/persons")
public class PersonResource {

   @Inject
   PersonRepositoryAsyncAwait personRepository;
   @Inject
   Logger log;

   @POST
   @RunOnVirtualThread
   public Person addPerson(Person person) {
      person = personRepository.save(person);
      return person;
   }

   @GET
   @RunOnVirtualThread
   public List<Person> getPersons() {
      return personRepository.findAll();
   }

   @GET
   @Path("/name/{name}")
   @RunOnVirtualThread
   public List<Person> getPersonsByName(@PathParam("name") String name) {
      return personRepository.findByName(name);
   }

   @GET
   @Path("/age-greater-than/{age}")
   @RunOnVirtualThread
   public List<Person> getPersonsByName(@PathParam("age") int age) {
      return personRepository.findByAgeGreaterThan(age);
   }

   @GET
   @Path("/{id}")
   @RunOnVirtualThread
   public Person getPersonById(@PathParam("id") Long id) {
      log.infof("(%s) getPersonById(%d)", Thread.currentThread(), id);
      return personRepository.findById(id);
   }

}

Let’s switch to the repository implementation. We use an implementation of the SmallRye Mutiny client for Postgres. The PgPool client allows us to create and execute SQL queries in a non-blocking way. In order to create a query, we should call the preparedQuery method. Finally, we need to invoke the executeAndAwait method to perform the operation asynchronously.

@ApplicationScoped
public class PersonRepositoryAsyncAwait {

   @Inject
   PgPool pgPool;
   @Inject
   Logger log;

   public Person save(Person person) {
      Long id = pgPool
         .preparedQuery("INSERT INTO person(name, age, gender) VALUES ($1, $2, $3) RETURNING id")
         .executeAndAwait()
         .iterator().next().getLong("id");
      person.setId(id);
      return person;
   }

   public List<Person> findAll() {
      log.info("FindAll()" + Thread.currentThread());
      RowSet<Row> rowSet = pgPool
         .preparedQuery("SELECT id, name, age, gender FROM person")
         .executeAndAwait();
      return iterateAndCreate(rowSet);
   }

   public Person findById(Long id) {
      RowSet<Row> rowSet = pgPool
         .preparedQuery("SELECT id, name, age, gender FROM person WHERE id = $1")
         .executeAndAwait(Tuple.of(id));
      List<Person> persons = iterateAndCreate(rowSet);
      return persons.size() == 0 ? null : persons.get(0);
   }

   public List<Person> findByName(String name) {
      RowSet<Row> rowSet = pgPool
         .preparedQuery("SELECT id, name, age, gender FROM person WHERE id = $1")
         .executeAndAwait(Tuple.of(name));
      return iterateAndCreate(rowSet);
   }

   public List<Person> findByAgeGreaterThan(int age) {
      RowSet<Row> rowSet = pgPool
         .preparedQuery("SELECT id, name, age, gender FROM person WHERE age > $1")
         .executeAndAwait(Tuple.of(age));
      return iterateAndCreate(rowSet);
   }

   private List<Person> iterateAndCreate(RowSet<Row> rowSet) {
      List<Person> persons = new ArrayList<>();
      for (Row row : rowSet) {
         persons.add(Person.from(row));
      }
      return persons;
   }

}

Prepare Test Data

Before we run load tests let’s add some test data to the Postgres database. We will use the Datafaker library for generating persons’ names. We will use the same reactive, non-blocking PgPool client as before. The following part of the code generates and inserts 1000 persons into the database on the Quarkus app startup. It is a part of our repository implementation.

@ApplicationScoped
public class PersonRepositoryAsyncAwait {

   // ... methods

   @Inject
   @ConfigProperty(name = "myapp.schema.create", defaultValue = "true")
   boolean schemaCreate;

   void config(@Observes StartupEvent ev) {
      if (schemaCreate) {
         initDb();
      }
   }

   private void initDb() {
      List<Tuple> persons = new ArrayList<>(1000);
      Faker faker = new Faker();
      for (int i = 0; i < 1000; i++) {
         String name = faker.name().fullName();
         String gender = faker.gender().binaryTypes().toUpperCase();
         int age = faker.number().numberBetween(18, 65);
         int externalId = faker.number().numberBetween(100000, 999999);
         persons.add(Tuple.of(name, age, gender, externalId));
      }

      pgPool.query("DROP TABLE IF EXISTS person").execute()
         .flatMap(r -> pgPool.query("""
                  create table person (
                    id serial primary key,
                    name varchar(255),
                    gender varchar(255),
                    age int,
                    external_id int
                  )
                  """).execute())
         .flatMap(r -> pgPool
            .preparedQuery("insert into person(name, age, gender, external_id) values($1, $2, $3, $4)")
            .executeBatch(persons))
         .await().indefinitely();
    }
}

Load Testing with k6 and JProfiler

I’m using the k6 tool for load testing, but you can use any other popular tool as well. In order to install it on macOS just run the following homebrew command:

$ brew install k6

The k6 tool uses Javascript for creating tests. We need to prepare an input with the test definition. My test generates a number between 1 and 1000 and then places it as a path parameter for the GET /persons/{id} endpoint. Finally, it checks if the response status is 200 and the body is not empty.

import http from 'k6/http';
import { sleep, check } from 'k6';

export default function () {
  let r = Math.floor(Math.random() * 1000) + 1;
  const res = http.get(`http://localhost:8080/persons/${r}`);
  check(res, {
    'is status 200': (res) => res.status === 200,
    'body size is > 0': (r) => r.body.length > 0,
  });
  sleep(1);
}

To simplify running the Quarkus app from your IDE you can annotate the main class with @QuarkusMain.

@QuarkusMain
public class PersonVirtualApp {

   public static void main(String... args) {
      Quarkus.run(args);
   }

}

Don’t forget to enable Java preview mode in your IDE. Here’s how it looks in IntelliJ.

Once you run the app and then attach JProfiler to the running JVM process you may execute tests with k6. To do that pass the location of the file with the test definition, and then set the number of concurrent threads (--vus parameter). Finally set the test duration with the --duration parameter. I’m running the test four times using a different number of concurrent threads (20, 50, 100, 200).

$ k6 run --vus 20 --duration 90s k6-test.js

Analyze Test Results

Let’s take a look at the fragment of application logs. You can see that there are many virtual threads running by the same “carrier” thread, for example, on the ForkJoinPool-1-worker-9.

Let’s switch to the JProfiler UI for a moment. Here’s the fragment of a thread pool visualization.

We run the same test four times with a different number of concurrent users (20, 50, 100, 200). Here’s a visualization of total Java threads (non-virtual) running. As you can see there is no significant difference between tests for 20 and 200 users.

During the first test (20 virtual users), we generated ~1.8k requests in 90 seconds.

During the last test (200 virtual users), we generated ~17.8k requests in 90 seconds.

Final Thoughts

One of the most important things I really like in Quarkus is that it is always up-to-date with the latest features. Once, the virtual threads have been released I can refactor my current app and use them instead of the standard platform threads. In this article, I presented how to use virtual threads with the app that connects to the database. If you are interested in other articles about Quarkus you can see the full list here. Enjoy

The post Quarkus with Java Virtual Threads appeared first on Piotr's TechBlog.