You can read more about Istio, Spring Boot, and other Java and Kubernetes-related technologies in my latest book, Hands-On Java with Kubernetes.

You can also find many other articles on my blog about Istio. For example, this article is about Quarkus and tracing with Istio.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Two sample applications for this exercise are available in the spring-boot-istio directory.

Prerequisites

We will start with the most straightforward Istio installation on a local Kubernetes cluster. This could be Minikube, which you can run with the following command. You can set slightly lower resource limits than I did.

minikube start --memory='8gb' --cpus='6'

minikube start --memory='8gb' --cpus='6'

To complete the exercise below, you need to install istioctl in addition to kubectl. Here you will find the available distributions for the latest versions of kubectl and istioctl. I install them on my laptop using Homebrew.

$ brew install kubectl
$ brew install istioctl

$ brew install kubectl
$ brew install istioctl

To install Istio with default parameters, run the following command:

istioctl install

istioctl install

After a moment, Istio should be running in the istio-system namespace.

It is also worth installing Kiali to verify the Istio resources we have created. Kiali is an observability and management tool for Istio that provides a web-based dashboard for service mesh monitoring. It visualizes service-to-service traffic, validates Istio configuration, and integrates with tools like Prometheus, Grafana, and Jaeger.

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.28/samples/addons/kiali.yaml

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.28/samples/addons/kiali.yaml

Once Kiali is successfully installed on Kubernetes, we can expose its web dashboard locally with the following istioctl command:

istioctl dashboard kiali

istioctl dashboard kiali

To test access to the Istio mesh from outside the cluster, you need to expose the ingress gateway. To do this, run the minikube tunnel command.

Use Spring Boot Istio Library

To test our library’s functionality, we will create a simple Spring Boot application that exposes a single REST endpoint.

@RestController
@RequestMapping("/callme")
public class CallmeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(CallmeController.class);

    @Autowired
    Optional<BuildProperties> buildProperties;
    @Value("${VERSION}")
    private String version;

    @GetMapping("/ping")
    public String ping() {
        LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() ?
            buildProperties.get().getName() : "callme-service", version);
        return "I'm callme-service " + version;
    }
    
}

@RestController
@RequestMapping("/callme")
public class CallmeController {

    private static final Logger LOGGER = LoggerFactory
        .getLogger(CallmeController.class);

    @Autowired
    Optional<BuildProperties> buildProperties;
    @Value("${VERSION}")
    private String version;

    @GetMapping("/ping")
    public String ping() {
        LOGGER.info("Ping: name={}, version={}", buildProperties.isPresent() ?
            buildProperties.get().getName() : "callme-service", version);
        return "I'm callme-service " + version;
    }
    
}

Then, in addition to the standard Spring Web starter, add the istio-spring-boot-starter dependency.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>com.github.piomin</groupId>
  <artifactId>istio-spring-boot-starter</artifactId>
  <version>1.2.1</version>
</dependency>

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>com.github.piomin</groupId>
  <artifactId>istio-spring-boot-starter</artifactId>
  <version>1.2.1</version>
</dependency>

Finally, we must add the @EnableIstio annotation to our application’s main class. We can also enable Istio Gateway to expose the REST endpoint outside the cluster. An Istio Gateway is a component that controls how external traffic enters or leaves a service mesh.

@SpringBootApplication
@EnableIstio(enableGateway = true)
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true)
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Let’s deploy our application on the Kubernetes cluster. To do this, we must first create a role with the necessary permissions to manage Istio resources in the cluster. The role must be assigned to the ServiceAccount used by the application.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: callme-service-with-starter
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules", "gateways"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: callme-service-with-starter
subjects:
  - kind: ServiceAccount
    name: callme-service-with-starter
    namespace: spring
roleRef:
  kind: Role
  name: callme-service-with-starter
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: callme-service-with-starter

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: callme-service-with-starter
rules:
  - apiGroups: ["networking.istio.io"]
    resources: ["virtualservices", "destinationrules", "gateways"]
    verbs: ["create", "get", "list", "watch", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: callme-service-with-starter
subjects:
  - kind: ServiceAccount
    name: callme-service-with-starter
    namespace: spring
roleRef:
  kind: Role
  name: callme-service-with-starter
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: callme-service-with-starter

Here are the Deployment and Service resources.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
  template:
    metadata:
      labels:
        app: callme-service-with-starter
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
  template:
    metadata:
      labels:
        app: callme-service-with-starter
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

The application repository is configured to run it with Skaffold. Of course, you can apply YAML manifests to the cluster with kubectl apply. To do this, simply navigate to the callme-service-with-starter/k8s directory and apply the deployment.yaml file. As part of the exercise, we will run our applications in the spring namespace.

skaffold dev -n spring

skaffold dev -n spring

The sample Spring Boot application creates two Istio objects at startup: a VirtualService and a Gateway. We can verify them in the Kiali dashboard.

The default host name generated for the gateway includes the deployment name and the .ext suffix. We can change the suffix name using the domain field in the @EnableIstio annotation. Assuming you run the minikube tunnel command, you can call the service using the Host header with the hostname in the following way:

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext"
I'm callme-service v1

Additional Capabilities with Spring Boot Istio

The library’s behavior can be customized by modifying the @EnableIstio annotation. For example, you can enable the fault injection mechanism using the fault field. Both abort and delay are possible. You can make this change without redeploying the app. The library updates the existing VirtualService.

@SpringBootApplication
@EnableIstio(enableGateway = true, fault = @Fault(percentage = 50))
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, fault = @Fault(percentage = 50))
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Now, you can call the same endpoint through the gateway. There is a 50% chance that you will receive this response.

Now let’s analyze a slightly more complex scenario. Let’s assume that we are running two different versions of the same application on Kubernetes. We use Istio to manage its versioning. Traffic is forwarded to the particular version based on the X-Version header in the incoming request. If the header value is v1 the request is sent to the application Pod with the label version=v1, and similarly, for the version v2. Here’s the annotation for the v1 application main class.

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v1",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v1") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v1",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v1") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

The Deployment manifest for the callme-service-with-starter-v1 should define two labels: app and version.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v1
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v1
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v1
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v1
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v1"
      serviceAccountName: callme-service-with-starter

Unlike the skaffold dev command, the skaffold run command simply launches the application on the cluster and terminates. Let’s first release version v1, and then move on to version v2.

skaffold run -n spring

skaffold run -n spring

Then, we can deploy the v2 version of our sample Spring Boot application. In this exercise, it is just an “artificial” version, since we deploy the same source code, but with different labels and environment variables injected in the Deployment manifest.

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v2",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v2") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

@SpringBootApplication
@EnableIstio(enableGateway = true, version = "v2",
	matches = { 
	  @Match(type = MatchType.HEADERS, key = "X-Version", value = "v2") })
public class CallmeApplication {

	public static void main(String[] args) {
		SpringApplication.run(CallmeApplication.class, args);
	}
	
}

Here’s the callme-service-with-starter-v2 Deployment manifest.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v2
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v2
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v2"
      serviceAccountName: callme-service-with-starter

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service-with-starter-v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service-with-starter
      version: v2
  template:
    metadata:
      labels:
        app: callme-service-with-starter
        version: v2
    spec:
      containers:
        - name: callme-service-with-starter
          image: piomin/callme-service-with-starter
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
          env:
            - name: VERSION
              value: "v2"
      serviceAccountName: callme-service-with-starter

Service, on the other hand, remains unchanged. It still refers to Pods labeled with app=callme-service-with-starter. However, this time it includes both application instances marked as v1 or v2.

apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

apiVersion: v1
kind: Service
metadata:
  name: callme-service-with-starter
  labels:
    app: callme-service-with-starter
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  selector:
    app: callme-service-with-starter

Version v2 should be run in the same way as before, using the skaffold run command. There are three Istio objects generated during apps startup: Gateway, VirtualService and DestinationRule.

A generated DestinationRule contains two subsets for both v1 and v2 versions.

The automatically generated VirtualService looks as follows.

kind: VirtualService
apiVersion: networking.istio.io/v1
metadata:
  name: callme-service-with-starter-route
spec:
  hosts:
  - callme-service-with-starter
  - callme-service-with-starter.ext
  gateways:
  - callme-service-with-starter
  http:
  - match:
    - headers:
        X-Version:
          prefix: v1
    route:
    - destination:
        host: callme-service-with-starter
        subset: v1
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx
  - match:
    - headers:
        X-Version:
          prefix: v2
    route:
    - destination:
        host: callme-service-with-starter
        subset: v2
      weight: 100
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx

kind: VirtualService
apiVersion: networking.istio.io/v1
metadata:
  name: callme-service-with-starter-route
spec:
  hosts:
  - callme-service-with-starter
  - callme-service-with-starter.ext
  gateways:
  - callme-service-with-starter
  http:
  - match:
    - headers:
        X-Version:
          prefix: v1
    route:
    - destination:
        host: callme-service-with-starter
        subset: v1
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx
  - match:
    - headers:
        X-Version:
          prefix: v2
    route:
    - destination:
        host: callme-service-with-starter
        subset: v2
      weight: 100
    timeout: 6s
    retries:
      attempts: 3
      perTryTimeout: 2s
      retryOn: 5xx

To test the versioning mechanism generated using the Spring Boot Istio library, set the X-Version header for each call.

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v1"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v2"
I'm callme-service v2

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v1"
I'm callme-service v1

$ curl http://localhost/callme/ping -H "Host:callme-service-with-starter.ext" -H "X-Version:v2"
I'm callme-service v2

Conclusion

I am still working on this library, and new features will be added in the near future. I hope it will be helpful for those of you who want to get started with Istio without getting into the details of its configuration.

The post Istio Spring Boot Library Released appeared first on Piotr's TechBlog.

The other potential solution in that context is the Vertical Pod Autoscaler. In the latest version, it supports in-place pod resize. Vertical Pod Autoscaler (VPA) in Kubernetes automatically adjusts CPU and memory requests/limits for pods based on their actual usage, ensuring containers receive the appropriate resources. Unlike the Horizontal Pod Autoscaler (HPA), which scales resources, not replicas, it may restart pods to apply changes. For now, VPA does not support this use case, but once this feature is implemented, the situation will change.

On the other hand, Kube Startup CPU Boost is a dedicated feature for scenarios with high CPU requirements during app startup. It is a controller that increases CPU resource requests and limits during Kubernetes workload startup. Once the workload is up and running, the resources are set back to their original values. Let’s see how this solution works in practice!

If you’re looking to go deeper into the practical side of running Java applications on Kubernetes, I explore these topics in much more detail in my book, Hands-On Java with Kubernetes.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. The sample application is based on Spring Boot and exposes several REST endpoints. However, in this exercise, we will use a ready-made image published on my Quay: quay.io/pminkows/sample-kotlin-spring:1.5.1.1.

Install Kube Startup CPU Boost

The Kubernetes cluster you are using must enable the In-Place Pod Resize feature. The activation method may vary by Kubernetes distribution. For the Minikube I am using in today’s example, it looks like this:

minikube start --memory='8gb' --cpus='6' --feature-gates=InPlacePodVerticalScaling=true

minikube start --memory='8gb' --cpus='6' --feature-gates=InPlacePodVerticalScaling=true

After that, we can proceed with installing the Kube Startup CPU Boost controller. There are several ways to achieve it. The easiest way to do this is with a Helm chart. Let’s add the following Helm repository:

helm repo add kube-startup-cpu-boost https://google.github.io/kube-startup-cpu-boost

helm repo add kube-startup-cpu-boost https://google.github.io/kube-startup-cpu-boost

Then, we can install the kube-startup-cpu-boost chart in the dedicated kube-startup-cpu-boost-system namespace using the following command:

helm install -n kube-startup-cpu-boost-system kube-startup-cpu-boost \
  kube-startup-cpu-boost/kube-startup-cpu-boost --create-namespace

helm install -n kube-startup-cpu-boost-system kube-startup-cpu-boost \
  kube-startup-cpu-boost/kube-startup-cpu-boost --create-namespace

If the installation was successful, you should see the following pod running in the kube-startup-cpu-boost-system namespace as below.

$ kubectl get pod -n kube-startup-cpu-boost-system
NAME                                                         READY   STATUS    RESTARTS   AGE
kube-startup-cpu-boost-controller-manager-75f95d5fb6-692s6   1/1     Running   0          36s

$ kubectl get pod -n kube-startup-cpu-boost-system
NAME                                                         READY   STATUS    RESTARTS   AGE
kube-startup-cpu-boost-controller-manager-75f95d5fb6-692s6   1/1     Running   0          36s

Install Monitoring Stack (optional)

Then, we can install Prometheus monitoring. It is an optional step to verify pod resource usage in the graphical form. Firstly, let’s install the following Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

After that, we can install the latest version of the kube-prometheus-stack chart in the monitoring namespace.

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Let’s verify the installation succeeded by listing the pods running in the monitoring namespace.

$ kubectl get pods -n monitoring
NAME                                                         READY   STATUS    RESTARTS   AGE
alertmanager-my-kube-prometheus-stack-alertmanager-0         2/2     Running   0          38s
my-kube-prometheus-stack-grafana-f8bb6b8b8-mzt4l             3/3     Running   0          48s
my-kube-prometheus-stack-kube-state-metrics-99f4574c-bf5ln   1/1     Running   0          48s
my-kube-prometheus-stack-operator-6d58dd9d6c-6srtg           1/1     Running   0          48s
my-kube-prometheus-stack-prometheus-node-exporter-tdwmr      1/1     Running   0          48s
prometheus-my-kube-prometheus-stack-prometheus-0             2/2     Running   0          38s

$ kubectl get pods -n monitoring
NAME                                                         READY   STATUS    RESTARTS   AGE
alertmanager-my-kube-prometheus-stack-alertmanager-0         2/2     Running   0          38s
my-kube-prometheus-stack-grafana-f8bb6b8b8-mzt4l             3/3     Running   0          48s
my-kube-prometheus-stack-kube-state-metrics-99f4574c-bf5ln   1/1     Running   0          48s
my-kube-prometheus-stack-operator-6d58dd9d6c-6srtg           1/1     Running   0          48s
my-kube-prometheus-stack-prometheus-node-exporter-tdwmr      1/1     Running   0          48s
prometheus-my-kube-prometheus-stack-prometheus-0             2/2     Running   0          38s

Finally, we can expose the Prometheus console over localhost using the port forwarding feature:

kubectl port-forward svc/my-kube-prometheus-stack-prometheus 9090:9090 -n monitoring

kubectl port-forward svc/my-kube-prometheus-stack-prometheus 9090:9090 -n monitoring

Configure Kube Startup CPU Boost

The Kube Startup CPU Boost configuration is pretty intuitive. We need to create a StartupCPUBoost resource. It can manage multiple applications based on a given selector. In our case, it is a single sample-kotlin-spring Deployment determined by the app.kubernetes.io/name label (1). The next step is to define the resource management policy (2). The Kube Startup CPU Boost increases both request and limit by 50%. Resources should only be increased for the duration of the startup (3). Therefore, once the readiness probe succeeds, the resource level will return to its initial state. Of course, everything happens in-place without restarting the container.

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions: # (1)
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy: # (2)
    containerPolicies:
    - containerName: sample-kotlin-spring
      percentageIncrease:
        value: 50
  durationPolicy: # (3)
    podCondition:
      type: Ready
      status: "True"

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions: # (1)
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy: # (2)
    containerPolicies:
    - containerName: sample-kotlin-spring
      percentageIncrease:
        value: 50
  durationPolicy: # (3)
    podCondition:
      type: Ready
      status: "True"

Next, we will deploy our sample application. Here’s the Deployment manifest of our Spring Boot app. The name of the app container is sample-kotlin-spring, which matches the target Deployment name defined inside the StartupCPUBoost object (1). Then, we set the CPU limit to 500 millicores (2). There’s also a new field resizePolicy. It tells Kubernetes whether a change to CPU or memory can be applied in-place or requires a Pod restart. (3). The NotRequired value means that changing the resource limit or request will not trigger a pod restart. The Deployment object also contains a readiness probe that calls the GET/actuator/health/readiness exposed with the Spring Boot Actuator (4).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  namespace: demo
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/name: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
        app.kubernetes.io/name: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring # (1)
        image: quay.io/pminkows/sample-kotlin-spring:1.5.1.1
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m # (2)
            memory: "1Gi"
          requests:
            cpu: 200m
            memory: "256Mi"
        resizePolicy: # (3)
        - resourceName: "cpu"
          restartPolicy: "NotRequired"
        readinessProbe: # (4)
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  namespace: demo
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/name: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
        app.kubernetes.io/name: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring # (1)
        image: quay.io/pminkows/sample-kotlin-spring:1.5.1.1
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m # (2)
            memory: "1Gi"
          requests:
            cpu: 200m
            memory: "256Mi"
        resizePolicy: # (3)
        - resourceName: "cpu"
          restartPolicy: "NotRequired"
        readinessProbe: # (4)
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3

Here are the pod requests and limits configured by Kube Startup CPU Boost. As you can see, the request is set to 300m, while the limit is completely removed.

Once the application startup process completes, Kube Startup CPU Boost restores the initial request and limit.

Now we can switch to the Prometheus console to see the history of CPU request values for our pod. As you can see, the request was temporarily increased during the pod startup.

The chart below illustrates CPU usage when the application is launched and then during normal operation.

We can also define the fixed resources for a target container. The CPU requests and limits of the selected container will be set to the given values (1). If you do not want the operator to remove the CPU limit during boost time, set the REMOVE_LIMITS environment variable to false in the kube-startup-cpu-boost-controller-manager Deployment.

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions:
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy:
    containerPolicies: # (1)
    - containerName: sample-kotlin-spring
      fixedResources:
        requests: "500m"
        limits: "2"
  durationPolicy:
    podCondition:
      type: Ready
      status: "True"

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions:
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy:
    containerPolicies: # (1)
    - containerName: sample-kotlin-spring
      fixedResources:
        requests: "500m"
        limits: "2"
  durationPolicy:
    podCondition:
      type: Ready
      status: "True"

Conclusion

There are many ways to address application CPU demand during startup. First, you don’t need to set a CPU limit for Deployment. What’s more, many people believe that setting a CPU limit doesn’t make sense, but for different reasons. In this situation, the request issue remains, but given the short timeframe and the significantly higher usage than in the declaration, it isn’t material.

Other solutions are related to strictly Java features. If we compile the application natively with GraalVM or use the CRaC feature, we will significantly speed up startup and reduce CPU requirements.

Finally, several solutions rely on in-place resizing. If you use Kyverno, consider its mutate policy, which can modify resources in response to an application startup event. The Kube Startup CPU Boost tool described in this article operates similarly but is designed exclusively for this use case. In the near future, Vertical Pod Autoscaler will also offer a CPU boost via in-place resize.

The post Startup CPU Boost in Kubernetes with In-Place Pod Resize appeared first on Piotr's TechBlog.

Here is a brief overview of all my published books.

Motivation

I won’t hide that this post is mainly directed at my blog subscribers and people who enjoy reading it and value my writing style. As you know, all posts and content on my blog, along with sample application repositories on GitHub, are always accessible to you for free. Over the past eight years, I have worked to publish high-quality content on my blog, and I plan to keep doing so. It is a part of my life, a significant time commitment, but also a lot of fun and a hobby.

I want to explain why I decided to write this book, why now, and why in this way. But first, a bit of background. I wrote my last book first, then my first book, over seven years ago. It focused on topics I was mainly involved with at the time, specifically Spring Boot and Spring Cloud. Since then, a lot of time has passed, and much has changed – not only in the technology itself but also a little in my personal life. Today, I am more involved in Kubernetes and container topics than, for example, Spring Cloud. For years, I have been helping various organizations transition from traditional application architectures to cloud-native models based on Kubernetes. Of course, Java remains my main area of expertise. Besides Spring Boot, I also really like the Quarkus framework. You can read a lot about both in my book on Kubernetes.

Based on my experience over the past few years, involving development teams is a key factor in the success of the Kubernetes platform within an organization. Ultimately, it is the applications developed by these teams that are deployed there. For developers to be willing to use Kubernetes, it must be easy for them to do so. That is why I persuade organizations to remove barriers to using Kubernetes and to design it in a way that makes it easier for development teams. On my blog and in this book, I aim to demonstrate how to quickly and simply launch applications on Kubernetes using frameworks such as Spring Boot and Quarkus.

It’s an unusual time to publish a book. AI agents are producing more and more technical content online. More often than not, instead of grabbing a book, people turn to an AI chatbot for a quick answer, though not always the best one. Still, a book that thoroughly introduces a topic and offers a step-by-step guide remains highly valuable.

Content of the Book

This book demonstrates that Java is an excellent choice for building applications that run on Kubernetes. In the first chapter, I’ll show you how to quickly build your application, create its image, and run it on Kubernetes without writing a single line of YAML or Dockerfile. This chapter also covers the minimum Kubernetes architecture you must understand to manage applications effectively in this environment. The second chapter, on the other hand, demonstrates how to effectively organize your local development environment to work with a Kubernetes cluster. You’ll see several options for running a distribution of your cluster locally and learn about the essential set of tools you should have. The third chapter outlines best practices for building applications on the Kubernetes platform. Most of the presented requirements are supported by simple examples and explanations of the benefits of meeting them. The fourth chapter presents the most valuable tools for the inner development loop with Kubernetes. After reading the first four chapters, you will understand the main Kubernetes components related to application management, enabling you to navigate the platform efficiently. You’ll also learn to leverage Spring Boot and Quarkus features to adapt your application to Kubernetes requirements.

In the following chapters, I will focus on the benefits of migrating applications to Kubernetes. The first area to cover is security. Chapter five discusses mechanisms and tools for securing applications running in a cluster. Chapter six describes Spring and Quarkus projects that enable native integration with the Kubernetes API from within applications. In chapter seven, you’ll learn about the service mesh tool and the benefits of using it to manage HTTP traffic between microservices. Chapter eight addresses the performance and scalability of Java applications in a Kubernetes environment. Chapter Eight demonstrates how to design a CI/CD process that runs entirely within the cluster, leveraging Kubernetes-native tools for pipeline building and the GitOps approach. This book also covers AI. In the final, ninth chapter, you’ll learn how to run a simple Java application that integrates with an AI model deployed on Kubernetes.

Publication

I decided to publish my book on Leanpub. Leanpub is a platform for writing, publishing, and selling books, especially popular among technical content authors. I previously published a book with Packt, but honestly, I was alone during the writing process. Leanpub is similar but offers several key advantages over publishers like Packt. First, it allows you to update content collaboratively with readers and keep it current. Even though my book is finished, I don’t rule out adding more chapters, such as on AI on Kubernetes. I also look forward to your feedback and plan to improve the content and examples in the repository continuously. Overall, this has been another exciting experience related to publishing technical content.

And when you buy such a book, you can be sure that most of the royalties go to me as the author, unlike with other publishers, where most of the royalties go to them as promoters. So, I’m looking forward to improving my book with you!

Conclusion

My book aims to bring together all the most interesting elements surrounding Java application development on Kubernetes. It is intended not only for developers but also for architects and DevOps teams who want to move to the Kubernetes platform.

The post A Book: Hands-On Java with Kubernetes appeared first on Piotr's TechBlog.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Quarkus Buildpacks Extension

Recently, support for Cloud Native Buildpacks in Quarkus has been significantly enhanced. Here you can access the repository containing the source code for the Paketo Quarkus buildpack. To implement this solution, add one dependency to your application.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-container-image-buildpack</artifactId>
</dependency>

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-container-image-buildpack</artifactId>
</dependency>

Next, run the build command with Maven and activate the quarkus.container-image.build parameter. Also, set the appropriate Java version needed for your application. For the sample Quarkus application in this article, the Java version is 21.

mvn clean package \
  -Dquarkus.container-image.build=true \
  -Dquarkus.buildpack.builder-env.BP_JVM_VERSION=21

mvn clean package \
  -Dquarkus.container-image.build=true \
  -Dquarkus.buildpack.builder-env.BP_JVM_VERSION=21

To build, you need Docker or Podman running. Here’s the output from the command run earlier.

As you can see, Quarkus uses, among other buildpacks, the buildpack as mentioned earlier.

The new image is now available for use.

$ docker images sample-quarkus/person-service:1.0.0-SNAPSHOT
REPOSITORY                      TAG              IMAGE ID       CREATED        SIZE
sample-quarkus/person-service   1.0.0-SNAPSHOT   e0b58781e040   45 years ago   160MB

$ docker images sample-quarkus/person-service:1.0.0-SNAPSHOT
REPOSITORY                      TAG              IMAGE ID       CREATED        SIZE
sample-quarkus/person-service   1.0.0-SNAPSHOT   e0b58781e040   45 years ago   160MB

Quarkus with OpenShift Builds Shipwright

Install the Openshift Build Operator

Now, we will move the image building process to the OpenShift cluster. OpenShift offers built-in support for creating container images directly within the cluster through OpenShift Builds, using the BuildConfig solution. For more details, please refer to my previous article. However, in this article, we explore a new technology for building container images called OpenShift Builds with Shipwright. To enable this solution on OpenShift, you need to install the following operator.

After installing this operator, you will see a new item in the “Build” menu called “Shiwright”. Switch to it, then select the “ClusterBuildStrategies” tab. There are two strategies on the list designed for Cloud Native Buildpacks. We are interested in the buildpacks strategy.

Create and Run Build with Shipwright

Finally, we can create the Shiwright Build object. It contains three sections. In the first step, we define the address of the container image repository where we will push our output image. For simplicity, we will use the internal registry provided by the OpenShift cluster itself. In the source section, we specify the repository address where the application source code is located. In the last section, we need to set the build strategy. We chose the previously mentioned buildpacks strategy for Cloud Native Buildpacks. Some parameters need to be set for the buildpacks strategy: run-image and cnb-builder-image. The cnb-builder-image indicates the name of the builder image containing the buildpacks. The run-image refers to a base image used to run the application. We will also activate the buildpacks Maven profile during the build to set the Quarkus property that switches from fast-jar to uber-jar packaging.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: buildpack-quarkus-build
spec:
  env:
    - name: BP_JVM_VERSION
      value: '21'
  output:
    image: 'image-registry.openshift-image-registry.svc:5000/builds/sample-quarkus-microservice:1.0'
  paramValues:
    - name: run-image
      value: 'paketobuildpacks/run-java-21-ubi9-base:latest'
    - name: cnb-builder-image
      value: 'paketobuildpacks/builder-jammy-java-tiny:latest'
    - name: env-vars
      values:
        - value: BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS=-Pbuildpacks
  retention:
    atBuildDeletion: true
  source:
    git:
      url: 'https://github.com/piomin/sample-quarkus-microservice.git'
    type: Git
  strategy:
    kind: ClusterBuildStrategy
    name: buildpacks

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: buildpack-quarkus-build
spec:
  env:
    - name: BP_JVM_VERSION
      value: '21'
  output:
    image: 'image-registry.openshift-image-registry.svc:5000/builds/sample-quarkus-microservice:1.0'
  paramValues:
    - name: run-image
      value: 'paketobuildpacks/run-java-21-ubi9-base:latest'
    - name: cnb-builder-image
      value: 'paketobuildpacks/builder-jammy-java-tiny:latest'
    - name: env-vars
      values:
        - value: BP_MAVEN_ADDITIONAL_BUILD_ARGUMENTS=-Pbuildpacks
  retention:
    atBuildDeletion: true
  source:
    git:
      url: 'https://github.com/piomin/sample-quarkus-microservice.git'
    type: Git
  strategy:
    kind: ClusterBuildStrategy
    name: buildpacks

Here’s the Maven buildpacks profile that sets a single Quarkus property quarkus.package.jar.type. We must change it to uber-jar, because the paketobuildpacks/builder-jammy-java-tiny builder expects a single jar instead of the multi-folder layout used by the default fast-jar format. Of course, I would prefer to use the paketocommunity/builder-ubi-base builder, which can recognize the fast-jar format. However, at this time, it does not function correctly with OpenShift Builds.

<profiles>
  <profile>
    <id>buildpacks</id>
    <activation>
      <property>
        <name>buildpacks</name>
      </property>
    </activation>
    <properties>
      <quarkus.package.jar.type>uber-jar</quarkus.package.jar.type>
    </properties>
  </profile>
</profiles>

<profiles>
  <profile>
    <id>buildpacks</id>
    <activation>
      <property>
        <name>buildpacks</name>
      </property>
    </activation>
    <properties>
      <quarkus.package.jar.type>uber-jar</quarkus.package.jar.type>
    </properties>
  </profile>
</profiles>

To start the build, you can use the OpenShift console or execute the following command:

shp build run buildpack-quarkus-build --follow

shp build run buildpack-quarkus-build --follow

We can switch to the OpenShift Console. As you can see, our build is running.

The history of such builds is available on OpenShift. You can also review the build logs.

Finally, you should see your image in the list of OpenShift internal image streams.

$ oc get imagestream
NAME                          IMAGE REPOSITORY                                                                                                    TAGS                UPDATED
sample-quarkus-microservice   default-route-openshift-image-registry.apps.pminkows.95az.p1.openshiftapps.com/builds/sample-quarkus-microservice   1.2,0.0.1,1.1,1.0   13 hours ago

$ oc get imagestream
NAME                          IMAGE REPOSITORY                                                                                                    TAGS                UPDATED
sample-quarkus-microservice   default-route-openshift-image-registry.apps.pminkows.95az.p1.openshiftapps.com/builds/sample-quarkus-microservice   1.2,0.0.1,1.1,1.0   13 hours ago

Conclusion

OpenShift Build Shipwright lets you perform the entire application image build process on the OpenShift cluster in a standardized manner. Cloud Native Buildpacks is a popular mechanism for building images without writing a Dockerfile yourself. In this case, support for Buildpacks on the OpenShift side is an interesting alternative to the Source-to-Image approach.

The post Quarkus with Buildpacks and OpenShift Builds appeared first on Piotr's TechBlog.

For comparison, please refer to the article where I demonstrate how to prepare a Backstage instance for running on Kubernetes step-by-step. Today, for the sake of clarity, we will be working in a Developer Hub instance running on OpenShift using an operator. However, we can also easily install Developer Hub on vanilla Kubernetes using a Helm chart.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. We will also use another repository with a sample Backstage plugin. This time, I won’t create a plugin myself, but I will use an existing one. The plugin provides a collection of scaffolder actions for interacting with Kubernetes on Backstage, including apply and delete. Once you clone both of those repositories, you should only follow my instructions.

Prerequisites

You must have an OpenShift cluster with the Red Hat Developer Hub installed and configured with a Kubernetes plugin. I will briefly explain it in the next section without getting more into the details. You can find more information in the already mentioned article about Red Hat Developer Hub.

You must also have Node.js, NPM, and Yarn installed and configured on your laptop. It is used for plugin compilation and building. The npm package @janus-idp/cli used for developing and exporting Backstage plugins as dynamic plugins, requires Podman when working in image mode.

Motivation

Red Hat Developer Hub comes with a curated set of plugins preinstalled on its container image. It is easy to enable such a plugin just by changing the configuration available in Kubernetes ConfigMap. The situation becomes complicated when we attempt to install and configure a third-party plugin. In this case, I would like to extend Backstage with a set of actions that enable the creation and management of Kubernetes resources directly, rather than applying them via Argo CD. To do that, we must install the Backstage Scaffolder Actions for Kubernetes plugin. To use this plugin in Red Hat Developer Hub without rebuilding its image, you must export plugins as derived dynamic plugin packages. This is our goal.

Convert to a dynamic Backstage plugin

The backstage-k8s-scaffolder-actions is a backend plugin. It meets all the requirements to be converted to a dynamic plugin form. It has a valid package.json file in its root directory, containing all required metadata and dependencies. That plugin is compatible with the new Backstage backend system, which means that it was created using createBackendPlugin() or createBackendModule(). Let’s clone its repository first:

$ git clone https://github.com/kirederik/backstage-k8s-scaffolder-actions.git
$ cd backstage-k8s-scaffolder-actions

Then we must install the @janus-idp/cli npm package with the following command:

yarn add @janus-idp/cli

After that, you should run both those commands inside the plugin directory:

$ yarn install
$ yarn build

If the commands were successful, you can proceed with the plugin conversion procedure. The plugin defines some shared dependencies that must be explicitly specified with the --shared-package flag.

Here’s the command used to convert our plugin to a dynamic form supported by Red Hat Developer Hub:

npx @janus-idp/cli@latest package export-dynamic-plugin \
  --shared-package '!@backstage/cli-common' \
  --shared-package '!@backstage/cli-node' \
  --shared-package '!@backstage/config-loader' \
  --shared-package '!@backstage/config' \
  --shared-package '!@backstage/errors' \
  --shared-package '!@backstage/types'

Package and publish Backstage dynamic plugins

After exporting a third-party plugin, you can package the derived package into one of the following supported formats:

• Open Container Initiative (OCI) image (recommended)
• TGZ file
• JavaScript package

Since the OCI image option is recommended, we will proceed accordingly. However, first you must ensure that podman is running on your laptop and is logged in to your container registry.

podman login quay.io

Then you can run the following @janus-idp/cli command with npx. It must specify the target image repository address, including image name and tag. The target image address is quay.io/pminkows/backstage-k8s-scaffolder-actions:v0.5. It is tagged with the latest version of the plugin.

npx @janus-idp/cli@latest package package-dynamic-plugins \
  --tag quay.io/pminkows/backstage-k8s-scaffolder-actions:v0.5

Here’s the command output. Ultimately, it provides instructions on how to install and enable the plugin within the Red Hat Developer configuration. Copy that statement for future use.

The previous command packages the plugin and builds its image.

Finally, let’s push the image with our plugin to the target registry:

podman push quay.io/pminkows/backstage-k8s-scaffolder-actions:v0.5

Install and enable the Backstage dynamic plugins in Developer Hub

My instance of Developer Hub is running in the backstage namespace. The operator manages it.

Here’s the Backstage CR object responsible for creating a Developer Hub instance. The dynamicPluginsConfigMapName property specifies the name of the ConfigMap that stores the plugins’ configuration.

apiVersion: rhdh.redhat.com/v1alpha3
kind: Backstage
metadata:
  name: developer-hub
  namespace: backstage
spec:
  application:
    appConfig:
      configMaps:
        - name: app-config-rhdh
      mountPath: /opt/app-root/src
    dynamicPluginsConfigMapName: dynamic-plugins-rhdh
    extraEnvs:
      secrets:
        - name: app-secrets-rhdh
    extraFiles:
      mountPath: /opt/app-root/src
    replicas: 1
    route:
      enabled: true
  database:
    enableLocalDb: true

Then, we must modify the dynamic-plugins-rhdh ConfigMap to register our plugin in Red Hat Developer Hub. You must paste the previously copied two lines of code generated by the npx @janus-idp/cli@latest package package-dynamic-plugins command.

kind: ConfigMap
apiVersion: v1
metadata:
  name: dynamic-plugins-rhdh
  namespace: backstage
data:
  dynamic-plugins.yaml: |-
    plugins:
      # ... other plugins

      - package: oci://quay.io/pminkows/backstage-k8s-scaffolder-actions:v0.5!devangelista-backstage-scaffolder-kubernetes
        disabled: false

That’s all! After the change is applied to ConfigMap, the operator should restart the pod with Developer Hub. It can take some time, as the pod will be restarted, since all plugins must be enabled during pod startup.

$ oc get pod
NAME                                      READY   STATUS    RESTARTS   AGE
backstage-developer-hub-896c5f9d9-vvddb   1/1     Running   0          4m21s
backstage-psql-developer-hub-0            1/1     Running   0          8d

You can verify the logs with the oc logs command. Developer Hub prints a list of available actions provided by the installed plugins. You should see three actions delivered by the Backstage Scaffolder Actions for Kubernetes plugin, starting with the kube: prefix.

Prepare the Backstage template

Finally, we will test a new plugin by calling the kube:apply action from our Backstage template. It uses the kube:apply to create a Secret in the specified namespace under a given name. This template is available in the backstage-templates repository.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  description: Create a Secret in Kubernetes
  name: create-secret
  title: Create a Secret
spec:
  lifecycle: experimental
  owner: user
  type: example
  parameters:
    - properties:
        name:
          description: The namespace name
          title: Name
          type: string
          ui:autofocus: true
      required:
        - name
      title: Namespace Name
    - properties:
        secretName:
          description: The secret name
          title: Secret Name
          type: string
          ui:autofocus: true
      required:
        - secretName
      title: Secret Name
    - title: Cluster Name
      properties:
        cluster:
          type: string
          enum:
            - ocp
          ui:autocomplete:
            options:
              - ocp
  steps:
    - action: kube:apply
      id: k-apply
      name: Create a Resouce
      input:
        namespaced: true
        clusterName: ${{ parameters.cluster }}
        manifest: |
          kind: Secret
          apiVersion: v1
          metadata:
            name: ${{ parameters.secretName }}
            namespace: ${{ parameters.name }}
          data:
            username: YWRtaW4=

You should import the repository with templates into your Developer Hub instance. The app-config-rhdh ConfigMap should contain the full address templates list file in the repository, and the Kubernetes cluster address and connection credentials.

catalog:
  rules:
    - allow: [Component, System, API, Resource, Location, Template]
  locations:
    - type: url
      target: https://github.com/piomin/backstage-templates/blob/master/templates.yaml
      rules:
        - allow: [Template, Location]

kubernetes:
  clusterLocatorMethods:
    - clusters:
      - authProvider: serviceAccount
        name: ocp
        serviceAccountToken: ${OPENSHIFT_TOKEN}
        skipTLSVerify: true
        url: https://api.${DOMAIN}:6443
      type: config
  customResources:
    - apiVersion: v1beta1
      group: tekton.dev
      plural: pipelineruns
    - apiVersion: v1beta1
      group: tekton.dev
      plural: taskruns
    - apiVersion: v1
      group: route.openshift.io
      plural: routes
  serviceLocatorMethod:
    type: multiTenant

You can access the Developer Hub instance through the OpenShift Route:

$ oc get route
NAME                      HOST/PORT                                                                 PATH   SERVICES                  PORT           TERMINATION     WILDCARD
backstage-developer-hub   backstage-developer-hub-backstage.apps.piomin.ewyw.p1.openshiftapps.com   /      backstage-developer-hub   http-backend   edge/Redirect   None

Then find and use the following template available in the Developer Hub.

You will have to insert the Secret name, namespace, and choose a target OpenShift cluster. Then accept the action.

You should see a similar screen. The Secret has been successfully created.

Let’s verify if it exists on our OpenShift cluster:

Final Thoughts

Red Hat is steadily developing its Backstage-based product, adding new functionality in future versions. The ability to easily create and install custom plug-ins in the Developer Hub appears to be a key element in building an attractive platform for developers. This article focuses on demonstrating how to convert a standard Backstage plugin into a dynamic form supported by Red Hat Developer Hub.

The post Backstage Dynamic Plugins with Red Hat Developer Hub appeared first on Piotr's TechBlog.

Argo CD ApplicationSet supports several different generators. In this article, we will focus on the Git generator type. It generates Argo CD Applications based on directory structure or branch changes in a Git repository. It contains two subtypes: the Git directory generator and the Git file generator. If you are interested in other Argo CD ApplicationSet generators you can find some articles on my blog. For example, the following post shows how to use List Generator to promote images between environments. You can also find a post about the Cluster Decision Resource generator, which shows how to spread applications dynamically between multiple Kubernetes clusters.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. You must go to the appset-helm-demo directory, which contains the whole configuration required for that exercise. Then you should only follow my instructions.

Argo CD Installation

Argo CD is the only tool we need to install on our Kubernetes cluster for that exercise. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

helm install my-argo-cd argo/argo-cd -n argocd

I use OpenShift in that exercise. With the OpenShift Console, I can easily install ArgoCD on the cluster using the OpenShift GitOps operator.

Once we installed it we can easily access the Argo CD dashboard.

We can sign in there using OpenShift credentials.

Motivation

Our goal in this exercise is to deploy and run some applications (a simple Java app and Postgres database) on Kubernetes with minimal source code effort. Those two applications only show how to create a standard that can be easily applied to any application type deployed on our cluster. In this standard, a directory structure determines how and where our applications are deployed on Kubernetes. My example configuration is stored in a single Git repository. However, we can easily extend it with multiple repositories, where Argoc CD switches between the central repository and other Git repositories containing a configuration for concrete applications.

Here’s a directory structure and files for deploying our two applications. Both the custom app and Postgres database are deployed in three environments: dev, test, and prod. We use Helm charts for deploying them. Each environment directory contains a Helm values file with installation parameters. The configuration distinguishes two different types of installation: apps and components. Each app is installed using the same Helm chart dedicated to a standard deployment. Each component is installed using a custom Helm chart provided by that component. For example, for Postgres, we will use the following Bitnami chart.

.
├── apps
│   ├── aaa-1
│   │   └── basic
│   │       ├── prod
│   │       │   └── values.yaml
│   │       ├── test
│   │       │   └── values.yaml
│   │       ├── uat
│   │       │   └── values.yaml
│   │       └── values.yaml
│   ├── aaa-2
│   └── aaa-3
└── components
    └── aaa-1
        └── postgresql
            ├── prod
            │   ├── config.yaml
            │   └── values.yaml
            ├── test
            │   ├── config.yaml
            │   └── values.yaml
            └── uat
                ├── config.yaml
                └── values.yaml

Before deploying the application, we should prepare namespaces with quotas, Argo CD projects, and ApplicationSet generators for managing application deployments. Here’s the structure of a global configuration repository. It also uses Helm chart to apply that part of manifests to the Kubernetes cluster. Each directory inside the projects directory determines our project name. On the other hand, a project contains several Kubernetes namespaces. Each project may contain several different Kubernetes Deployments.

.
└── projects
    ├── aaa-1
    │   └── values.yaml
    ├── aaa-2
    │   └── values.yaml
    └── aaa-3
        └── values.yaml

Prepare Global Cluster Configuration

Helm Template for Namespaces and Quotas

Here’s the Helm template for creating namespaces and quotas for each namespace. We will create the project namespace per each environment (stage).

{{- range .Values.stages }}
---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ $.Values.projectName }}-{{ .name }}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: {{ $.Values.projectName }}-{{ .name }}
spec:
  hard:
    {{- if .config }}
    {{- with .config.quotas }}
    pods: {{ .pods | default "10" }}
    requests.cpu: {{ .cpuRequest | default "2" }}
    requests.memory: {{ .memoryRequest | default "2Gi" }}
    limits.cpu: {{ .cpuLimit | default "8" }}
    limits.memory: {{ .memoryLimit | default "8Gi" }}
    {{- end }}
    {{- else }}
    pods: "10"
    requests.cpu: "2"
    requests.memory: "2Gi"
    limits.cpu: "8"
    limits.memory: "8Gi"
    {{- end }}
{{- end }}

Helm Template for the Argo CD AppProject

Helm chart will also create a dedicated Argo CD AppProject object per our project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: {{ .Values.projectName }}
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'

Helm Template for Argo CD ApplicationSet

After that, we can proceed to the most tricky part of our exercise. Helm chart also defines a template for creating the Argo CD ApplicationSet. This ApplicationSet must analyze the repository structure, which contains the configuration of apps and components. We define two ApplicationSets per each project. The first uses the Git Directory generator to determine the structure of the apps catalog and deploy the apps in all environments using my custom spring-boot-api-app chart. The chart parameters can be overridden with Helm values placed in each app directory.

The second ApplicationSet uses the Git Files generator to determine the structure of the components catalog. It reads the contents of the config.yaml file in each directory. The config.yaml file sets the repository, name, and version of the Helm chart that must be used to install the component on Kubernetes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-apps-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          {{- range .Values.stages }}
          - path: appset-helm-demo/apps/{{ $.Values.projectName }}/*/{{ .name }}
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: spring-boot-api-app
          repoURL: 'https://piomin.github.io/helm-charts/'
          targetRevision: 0.3.8
          helm:
            valueFiles:
              - $values/appset-helm-demo/apps/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-components-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        files:
          {{- range .Values.stages }}
          - path: appset-helm-demo/components/{{ $.Values.projectName }}/*/{{ .name }}/config.yaml
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: '{{`{{ .chart.name }}`}}'
          repoURL: '{{`{{ .chart.repository }}`}}'
          targetRevision: '{{`{{ .chart.version }}`}}'
          helm:
            valueFiles:
              - $values/appset-helm-demo/components/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

There are several essential elements in this configuration, which we should pay attention to. Both Helm and ApplicationSet use templating engines based on {{ ... }} placeholders. So to avoid conflicts we should escape Argo CD ApplicationSet templating elements from the Helm templating elements. The following part of the template responsible for generating the Argo CD Application name is a good example of that approach: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'. First, we use the AppliocationSet Git generator parameter index .path.segments 3 that returns the name of the third part of the directory path. Those elements are escaped with the ` char so Helm doesn’t try to analyze it.

Helm Chart Structure

Our ApplicationSets use the “Multiple Sources for Application” feature to read parameters from Helm values files and inject them into the Helm chart from a remote repository. Thanks to that, our configuration repositories for apps and components contain only values.yaml files in the standardized directory structure. The only chart we store in the sample repository has been described above and is responsible for creating the configuration required to run app Deployments on the cluster.

.
└── chart
    ├── Chart.yaml
    ├── templates
    │   ├── additional.yaml
    │   ├── applicationsets.yaml
    │   ├── appproject.yaml
    │   └── namespaces.yaml
    └── values.yaml

By default, each project defines three environments (stages): test, uat, prod.

stages:
  - name: test
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

We can override a default behavior for the specific project in Helm values. Each project directory contains the values.yaml file. Here are Helm parameters for the aaa-3 project that override CPU request quota from 2 CPUs to 4 CPUs only for the test environment.

stages:
  - name: test
    config:
      quotas:
        cpuRequest: 4
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

Run the Synchronization Process

Generate Global Structure on the Cluster

To start a process we must create the ApplicationSet that reads the structure of the projects directory. Each subdirectory in the projects directory indicates the name of our project. Our ApplicationSet uses a Git directory generator to create an Argo CD Application per each project. Its name contains the name of the subdirectory and the config suffix. Each generated Application uses the previously described Helm chart to create all namespaces, quotas, and other resources requested by the project. It also leverages the “Multiple Sources for Application” feature to allow us to override default Helm chart settings. It reads a project name from the directory name and passes it as a parameter to the generated Argo CD Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-config
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          - path: appset-helm-demo/projects/*
  template:
    metadata:
      name: '{{.path.basename}}-config'
    spec:
      destination:
        namespace: '{{.path.basename}}'
        server: 'https://kubernetes.default.svc'
      project: default
      sources:
        - path: appset-helm-demo/chart
          repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          helm:
            valueFiles:
              - $values/appset-helm-demo/projects/{{.path.basename}}/values.yaml
            parameters:
              - name: projectName
                value: '{{.path.basename}}'
              - name: argoNamespace
                value: openshift-gitops
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Once we create the global-config ApplicationSet object magic happens. Here’s the list of Argo CD Applications generated from our directories in Git configuration repositories.

First, there are three Argo CD Applications with the projects’ configuration. That’s happening because we defined 3 subdirectories in the projects directory with names aaa-1, aaa-2 and aaa-3.

The configuration applied by those Argo CD Applications is pretty similar since they are using the same Helm chart. We can look at the list of resources managed by the aaa-3-config Application. There are three namespaces (aaa-3-test, aaa-3-uat, aaa-3-prod) with resource quotas, a single Argo CD AppProject, and two ApplicationSet objects responsible for generating Argo CD Applications for apps and components directories.

In this configuration, we can verify if the value of the request.cpu ResourceQuota object has been overridden from 2 CPUs to 4 CPUs.

Let’s analyze what happened. Here’s a list of Argo CD ApplicationSets. The global-config ApplicationSet generated Argo CD Application per each detected project inside the projects directory. Then, each of these Applications applied two ApplicationSet objects to cluster using the Helm template.

$ kubectl get applicationset
NAME                      AGE
aaa-1-components-config   29m
aaa-1-apps-config         29m
aaa-2-components-config   29m
aaa-2-apps-config         29m
aaa-3-components-config   29m
aaa-3-apps-config         29m
global-config             29m

There’s also a list of created namespaces:

$ kubectl get ns
NAME                                               STATUS   AGE
aaa-1-prod                                         Active   34m
aaa-1-test                                         Active   34m
aaa-1-uat                                          Active   34m
aaa-2-prod                                         Active   34m
aaa-2-test                                         Active   34m
aaa-2-uat                                          Active   34m
aaa-3-prod                                         Active   34m
aaa-3-test                                         Active   34m
aaa-3-uat                                          Active   34m

Generate and Apply Deployments

Our sample configuration contains only two Deployments. We defined the basic subdirectory in the apps directory and the postgres subdirectory in the components directory inside the aaa-1 project. The aaa-2 and aaa-3 projects don’t contain any Deployments for simplification. However, the more subdirectories with the values.yaml file we create there, the more applications will be deployed on the cluster. Here’s a typical values.yaml file for a simple app deployed with a standard Helm chart. It defines the image repository, name, and tag. It also set the Deployment name and environment.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: prod

For the postgres component we must set more parameters in Helm values. Here’s the final list:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

The following Argo CD Application has been generated by the aaa-1-apps-config ApplicationSet. It detected the basic subdirectory in the apps directory. The basic subdirectory contained 3 subdirectories: test, uat and prod with values.yaml file. As a result, we have Argo CD per environment responsible for deploying the basic app in the target namespaces.

Here’s a list of resources managed by the basic-prod Application. It uses my custom Helm chart and applies Deployment and Service objects to the cluster.

The following Argo CD Application has been generated by the aaa-1-components-config ApplicationSet. It detected the basic subdirectory in the components directory. The postgres subdirectory contained 3 subdirectories: test, uat and prod with values.yaml and config.yaml files. The ApplicationSet Files generator reads the repository, name, and version from the configuration in the config.yaml file.

Here’s the config.yaml file with the Bitnami Postgres chart settings. We could place here any other chart we want to install something else on the cluster.

chart:
  repository: https://charts.bitnami.com/bitnami
  name: postgresql
  version: 15.5.38

Here’s the list of resources installed by the Bitnami Helm chart used by the generated Argo CD Applications.

Final Thoughts

This article proves that Argo CD ApplicationSet and Helm templates can be used together to create advanced configuration structures. It shows how to use ApplicationSet Git Directory and Files generators to analyze the structure of directories and files in the Git config repository. With that approach, we can propose a standardization in the configuration structure across the whole organization and propagate it similarly for all the applications deployed in the Kubernetes clusters. Everything can be easily managed at the cluster admin level with the single global Argo CD ApplicationSet that accesses many different repositories with configuration.

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

The solution to these challenges is Kargo, an open-source tool implementing continuous promotion within CI/CD pipelines. It provides a structured mechanism for promoting changes in complex environments involving Kubernetes and GitOps. I’ve been following Kargo for several months. It’s an interesting tool that provides stage-to-stage promotion using GitOps principles with Argo CD. It reached the version in October 2024. Let’s take a closer look at it.

If you are interested in promotion between environments on Kubernetes using the GitOps approach, you can read about another tool that tackles that challenge – Devtron. Here’s the link to my article that explains its concept.

Understand the Concept

Before we start with Kargo, we need to understand the concept around that tool. Let’s analyze several basic terms defined and implemented by Kargo.

A project is a bunch of related Kargo resources that describe one or more delivery pipelines. It’s the basic unit of organization and multi-tenancy in Kargo. Every Kargo project has its own cluster-scoped Kubernetes resource of type Project. We should put all the resources related to a certain project into the same Kubernetes namespace.

A stage represents environments in Kargo. Stages are the most important concept in Kargo. We can link them together in a directed acyclic graph to describe a delivery pipeline. Typically, a delivery pipeline starts with a test or dev stage and ends with one or more prod stages.

A Freight object represents resources that Kargo promotes from one stage to another. It can reference one or more versioned artifacts, such as container images, Kubernetes manifests loaded from Git repositories, or Helm charts from chart repositories. 

A warehouse is a source of freight. It can refer to container image repositories, Git, or Helm chart repositories.

In that context, we should treat a promotion as a request to move a piece of freight into a specified stage.

Source Code

If you want to try out this exercise, go ahead and take a look at my source code. To do that, just clone my GitHub repository. It contains the sample Spring Boot application in the basic directory. The application exposes a single REST endpoint that returns the app Maven version number. Go to that directory. Then, you can follow my further instructions.

Kargo Installation

A few things must be ready before installing Kargo. We must have a Kubernetes cluster and the helm CLI installed on our laptop. I use Minikube. Kargo integrates with Cert-Manager, Argo CD, and Argo Rollouts. We can install all those tools using official Helm charts. Let’s begin with Cert-Manager. First, we must add the jetstack Helm repository:

helm repo add jetstack https://charts.jetstack.io

Here’s the helm command that installs it in the cert-manager namespace.

helm install cert-manager --namespace cert-manager jetstack/cert-manager \
  --set crds.enabled=true \
  --set crds.keep=true

To install Argo CD, we must first add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

Then, we can install Argo CD in the argocd namespace.

helm install argo-cd argo/argo-cd

The Argo Rollouts chart is located in the same Helm repository as the Argo CD chart. We will also install it in the argocd namespace:

helm install my-argo-rollouts argo/argo-rollouts

Finally, we can proceed to the Kargo installation. We will install it in the kargo namespace. The installation command sets two Helm parameters. We should set the Bcrypt password hash and a key used to sign JWT tokens for the admin account:

helm install kargo \
  oci://ghcr.io/akuity/kargo-charts/kargo \
  --namespace kargo \
  --create-namespace \
  --set api.adminAccount.passwordHash='$2y$10$xu2U.Ux5nV5wKmerGcrDlO261YeiTlRrcp2ngDGPxqXzDyiPQvDXC' \
  --set api.adminAccount.tokenSigningKey=piomin \
  --wait

We can generate and print the password hash using e.g. htpasswd. Here’s the sample command:

htpasswd -nbB admin 123456

After installation is finished, we can verify it by displaying a list of pods running in the kargo namespace.

$ kubectl get po -n kargo
NAME                                          READY   STATUS    RESTARTS   AGE
kargo-api-dbb4d5cb7-zvnc6                     1/1     Running   0          44s
kargo-controller-c4964bbb7-4ngnv              1/1     Running   0          44s
kargo-management-controller-dc5569759-596ch   1/1     Running   0          44s
kargo-webhooks-server-6df6dd58c-g5jlp         1/1     Running   0          44s

Kargo provides a UI dashboard that allows us to display and manage continuous promotion configuration. Let’s expose it locally on the 8443 port using a port-forward feature:

kubectl port-forward svc/kargo-api -n kargo 8443:443

Once we sign in to the dashboard using the admin password set during an installation we can create a new kargo-demo project:

Sample Application

Our sample application is simple. It exposes a single GET /basic/ping endpoint that returns the version number read from Maven pom.xml.

@RestController
@RequestMapping("/basic")
public class BasicController {

    @Autowired
    Optional<BuildProperties> buildProperties;

    @GetMapping("/ping")
    public String ping() {
        return "I'm basic:" + buildProperties.orElseThrow().getVersion();
    }
}

We will build an application image using the Jib Maven Plugin. It is already configured in Maven pom.xml. I set my Docker Hub as the target registry, but you can relate it to your account.

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.4</version>
  <configuration>
    <container>
      <user>1001</user>
    </container>
    <to>
      <image>piomin/basic:${project.version}</image>
    </to>
  </configuration>
</plugin>

The following Maven command builds the application and its image from a source code. Before the build, we should increase the version number in the project.version field in pom.xml. We begin with the 1.0.0 version, which should be pushed to your registry before proceeding.

mvn clean package -DskipTests jib:build

Here’s the result of my initial build.

Let’s switch to the Docker Registry dashboard after pushing the 1.0.0 version.

Configure Kargo for Promotion on Kubernetes

First, we will create the Kargo Warehouse object. It refers to the piomin/basic repository containing the image with our sample app. The Warehouse object is responsible for discovering new image tags pushed into the registry. We also use my Helm chart to deploy the image to Kubernetes. However, we will only use the latest version of that chart. Otherwise, we should also place that chart inside the basic Warehouse object to enable new chart version discovery.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
 name: basic
 namespace: demo
spec:
 subscriptions:
 - image:
     discoveryLimit: 5
     repoURL: piomin/basic

Then, we will create the Argo CD ApplicationSet to generate an application per environment. There are three environments: test, uat, prod. Each Argo CD application must be annotated by kargo.akuity.io/authorized-stage containing the project and stage name. Argo uses multiple sources. The argocd-showcase repository contains Helm values files with parameters per each stage. The piomin.github.io/helm-charts repository provides the spring-boot-api-app Helm chart that refers to those values.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: demo
 namespace: argocd
spec:
 generators:
 - list:
     elements:
     - stage: test
     - stage: uat
     - stage: prod
 template:
   metadata:
     name: demo-{{stage}}
     annotations:
       kargo.akuity.io/authorized-stage: demo:{{stage}}
   spec:
     project: default
     sources:
       - chart: spring-boot-api-app
         repoURL: 'https://piomin.github.io/helm-charts/'
         targetRevision: 0.3.8
         helm:
           valueFiles:
             - $values/values/values-{{stage}}.yaml
       - repoURL: 'https://github.com/piomin/argocd-showcase.git'
         targetRevision: HEAD
         ref: values
     destination:
       server: https://kubernetes.default.svc
       namespace: demo-{{stage}}
     syncPolicy:
       syncOptions:
       - CreateNamespace=true

Now, we can proceed to the most complex element of our exercise – stage creation. The Stage object refers to the previously created Warehouse object to request freight to promote. Then, it defines the steps to perform during the promotion process. Kargo’s promotion steps define the workflow of a promotion process. They do the things needed to promote a piece of freight into the next stage. We can use several built-in steps that cover the most common operations like cloning Git repo, updating Helm values, or pushing changes to the remote repository. Our stage definition contains five steps. After cloning the repository with Helm values, we must update the image.tag parameter in the values-test.yaml file with the tag value read from the basic Warehouse. Then, Kargo commits and pushes changes to the configuration repository and triggers Argo CD application synchronization.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
 name: test
 namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     direct: true
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Here’s the values-test.yaml file in the Argo CD configuration repository.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: test

Here’s the Stage definition of the uat environment. It is pretty similar to the definition of the test environment. It just defines the previous source stage to test.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: uat
  namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     stages:
       - test
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Perform Promotion Process

Once a new image tag is published to the registry, it becomes visible in the Kargo Dashboard. We must click the “Promote into Stage” button to promote a selected version to the target stage.

Then we should specify the source image tag. A choice is obvious since we only have the image tagged with 1.0.0. After approving the selection by clicking the “Yes” button Kargo starts a promotion process on Kubernetes.

After a while, we should have a new image promoted to the test stage.

Let’s repeat the promotion process of the 1.0.0 version for the other two stages. Each stage should be at a healthy status. That status is read directly from the corresponding Argo CD Application.

Let’s switch to the Argo CD dashboard. There are three applications.

We can make a test call of the sample application HTTP endpoint. Currently, all the environments run the same 1.0.0 version of the app. Let’s enable port forwarding for the basic service in the demo-prod namespace.

kubectl port-forward svc/basic -n demo-prod 8080:8080

The endpoint returns the application name and version as a response.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.0

Then, we will build another three versions of the basic application beginning from 1.0.1 to 1.0.5.

Once we push each version to the registry, we can refresh the list of images. With the default configuration, Kargo should add the latest version to the list. After pushing the 1.0.3 version I promoted it to the test stage. Then I refreshed a list after pushing the 1.0.4 tag. Now, the 1.0.3 tag can be promoted to a higher environment. In the illustration below, I’m promoting it to the uat stage.

After that, we can promote the 1.0.4 version to the test stage, and refresh a list of images once again to see the currently pushed 1.0.5 tag.

Here’s another promotion. This time, I moved the 1.0.3 version to the prod stage.

After clicking on the image tag tile, we will see its details. For example, the 1.0.3 tag has been verified on the test and uat stages. There is also an approval section. However, we still didn’t approve freight. To do that, we need to switch to the kargo CLI.

The kargo CLI binary for a particular OS on the project GitHub releases page. We must download and copy it to the directory under the PATH. Then, we can sign in to the Kargo server running on Kubernetes using admin credentials.

kargo login https://localhost:8443 --admin \
  --password 123456 \
  --insecure-skip-tls-verify

We can approve a specific freight. Let’s display a list of Freight objects.

$ kubectl get freight -n demo
NAME                                       ALIAS            ORIGIN (KIND)   ORIGIN (NAME)   AGE
0501f8d8018a953821ea437078c1ec34e6db5a6b   ideal-horse      Warehouse       basic           8m7s
683be41b1cef57ed755fc7a0f8e8d7776f90c63a   wiggly-tuatara   Warehouse       basic           11m
6a1219425ddeceabfe94f0605d7a5f6d9d20043e   ulterior-zebra   Warehouse       basic           13m
f737178af6492ea648f85fc7d082e34b7a085927   eager-snail      Warehouse       basic           7h49m

The kargo approve command takes Freight ID as the input parameter.

kargo approve --freight 6a1219425ddeceabfe94f0605d7a5f6d9d20043e \
  --stage uat \
  --project demo

Now, the image tag details window should display the approved stage name.

Let’s enable port forwarding for the basic service in the demo-uat namespace.

kubectl port-forward svc/basic -n demo-uat 8080:8080

Then we call the /basic/ping endpoint to check out the current version.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.3

Final Thoughts

This article explains the idea behind continuous app promotion between environments on Kubernetes with Kargo and Argo CD. Kargo is a relatively new project in the Kubernetes ecosystem, that smoothly addresses the challenges related to GitOps promotion. It seems promising. I will closely monitor the further development of this project.

The post Continuous Promotion on Kubernetes with GitOps appeared first on Piotr's TechBlog.

You can find other interesting articles on my blog if you are interested in the tools for generating Kubernetes manifests. Here’s the article that shows how to use the Dekorate library to generate Kubernetes manifests for the Spring Boot app.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. First, you need to clone the following GitHub repository. It contains several sample Java applications for a Kubernetes showcase. You must go to the “inner-dev-loop” directory, to proceed with exercise. Then you should follow my further instructions.

Prerequisites

Before we start the development, we must install some tools on our laptops. Of course, we should have Maven and at least Java 21 installed. We must also have access to the container engine (like Docker or Podman) and a Kubernetes cluster. I have everything configured on my local machine using Podman Desktop and Minikube. Finally, we need to install the Helm CLI. It can be used to deploy the Postgres database on Kubernetes using the popular Bitnami Helm chart. In summary, we need to have:

• Maven
• OpenJDK 21+
• Podman or Docker
• Kubernetes
• Helm CLI

Once we have everything in place, we can proceed to the next steps.

Create Spring Boot Application

In this exercise, we create a typical Spring Boot application that connects to the relational database and exposes REST endpoints for the basic CRUD operations. Both the application and database will run on Kubernetes. We install the Postgres database using the Bitnami Helm chart. To build and deploy the application in the Kubernetes cluster, we will use Maven and Eclipse JKube features. First, let’s take a look at the source code of our application. Here’s the list of included dependencies. It’s worth noting that Spring Boot Actuator is responsible for generating Kubernetes liveness and readiness health checks. JKube will be able to detect it and generate the required elements in the Deployment manifest.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
    <groupId>org.postgresql</groupId>
    <artifactId>postgresql</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
    <version>2.6.0</version>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-devtools</artifactId>
    <scope>runtime</scope>
    <optional>true</optional>
</dependency>

Here’s our Person entity class:

@Entity
public class Person {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String firstName;
    private String lastName;
    private int age;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    private Integer externalId;

    // getters and setters
}