Argo CD ApplicationSet supports several different generators. In this article, we will focus on the Git generator type. It generates Argo CD Applications based on directory structure or branch changes in a Git repository. It contains two subtypes: the Git directory generator and the Git file generator. If you are interested in other Argo CD ApplicationSet generators you can find some articles on my blog. For example, the following post shows how to use List Generator to promote images between environments. You can also find a post about the Cluster Decision Resource generator, which shows how to spread applications dynamically between multiple Kubernetes clusters.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. You must go to the appset-helm-demo directory, which contains the whole configuration required for that exercise. Then you should only follow my instructions.

Argo CD Installation

Argo CD is the only tool we need to install on our Kubernetes cluster for that exercise. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

helm install my-argo-cd argo/argo-cd -n argocd

I use OpenShift in that exercise. With the OpenShift Console, I can easily install ArgoCD on the cluster using the OpenShift GitOps operator.

Once we installed it we can easily access the Argo CD dashboard.

We can sign in there using OpenShift credentials.

Motivation

Our goal in this exercise is to deploy and run some applications (a simple Java app and Postgres database) on Kubernetes with minimal source code effort. Those two applications only show how to create a standard that can be easily applied to any application type deployed on our cluster. In this standard, a directory structure determines how and where our applications are deployed on Kubernetes. My example configuration is stored in a single Git repository. However, we can easily extend it with multiple repositories, where Argoc CD switches between the central repository and other Git repositories containing a configuration for concrete applications.

Here’s a directory structure and files for deploying our two applications. Both the custom app and Postgres database are deployed in three environments: dev, test, and prod. We use Helm charts for deploying them. Each environment directory contains a Helm values file with installation parameters. The configuration distinguishes two different types of installation: apps and components. Each app is installed using the same Helm chart dedicated to a standard deployment. Each component is installed using a custom Helm chart provided by that component. For example, for Postgres, we will use the following Bitnami chart.

.
├── apps
│   ├── aaa-1
│   │   └── basic
│   │       ├── prod
│   │       │   └── values.yaml
│   │       ├── test
│   │       │   └── values.yaml
│   │       ├── uat
│   │       │   └── values.yaml
│   │       └── values.yaml
│   ├── aaa-2
│   └── aaa-3
└── components
    └── aaa-1
        └── postgresql
            ├── prod
            │   ├── config.yaml
            │   └── values.yaml
            ├── test
            │   ├── config.yaml
            │   └── values.yaml
            └── uat
                ├── config.yaml
                └── values.yaml

Before deploying the application, we should prepare namespaces with quotas, Argo CD projects, and ApplicationSet generators for managing application deployments. Here’s the structure of a global configuration repository. It also uses Helm chart to apply that part of manifests to the Kubernetes cluster. Each directory inside the projects directory determines our project name. On the other hand, a project contains several Kubernetes namespaces. Each project may contain several different Kubernetes Deployments.

.
└── projects
    ├── aaa-1
    │   └── values.yaml
    ├── aaa-2
    │   └── values.yaml
    └── aaa-3
        └── values.yaml

Prepare Global Cluster Configuration

Helm Template for Namespaces and Quotas

Here’s the Helm template for creating namespaces and quotas for each namespace. We will create the project namespace per each environment (stage).

{{- range .Values.stages }}
---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ $.Values.projectName }}-{{ .name }}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: {{ $.Values.projectName }}-{{ .name }}
spec:
  hard:
    {{- if .config }}
    {{- with .config.quotas }}
    pods: {{ .pods | default "10" }}
    requests.cpu: {{ .cpuRequest | default "2" }}
    requests.memory: {{ .memoryRequest | default "2Gi" }}
    limits.cpu: {{ .cpuLimit | default "8" }}
    limits.memory: {{ .memoryLimit | default "8Gi" }}
    {{- end }}
    {{- else }}
    pods: "10"
    requests.cpu: "2"
    requests.memory: "2Gi"
    limits.cpu: "8"
    limits.memory: "8Gi"
    {{- end }}
{{- end }}

Helm Template for the Argo CD AppProject

Helm chart will also create a dedicated Argo CD AppProject object per our project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: {{ .Values.projectName }}
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'

Helm Template for Argo CD ApplicationSet

After that, we can proceed to the most tricky part of our exercise. Helm chart also defines a template for creating the Argo CD ApplicationSet. This ApplicationSet must analyze the repository structure, which contains the configuration of apps and components. We define two ApplicationSets per each project. The first uses the Git Directory generator to determine the structure of the apps catalog and deploy the apps in all environments using my custom spring-boot-api-app chart. The chart parameters can be overridden with Helm values placed in each app directory.

The second ApplicationSet uses the Git Files generator to determine the structure of the components catalog. It reads the contents of the config.yaml file in each directory. The config.yaml file sets the repository, name, and version of the Helm chart that must be used to install the component on Kubernetes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-apps-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          {{- range .Values.stages }}
          - path: appset-helm-demo/apps/{{ $.Values.projectName }}/*/{{ .name }}
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: spring-boot-api-app
          repoURL: 'https://piomin.github.io/helm-charts/'
          targetRevision: 0.3.8
          helm:
            valueFiles:
              - $values/appset-helm-demo/apps/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-components-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        files:
          {{- range .Values.stages }}
          - path: appset-helm-demo/components/{{ $.Values.projectName }}/*/{{ .name }}/config.yaml
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: '{{`{{ .chart.name }}`}}'
          repoURL: '{{`{{ .chart.repository }}`}}'
          targetRevision: '{{`{{ .chart.version }}`}}'
          helm:
            valueFiles:
              - $values/appset-helm-demo/components/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

There are several essential elements in this configuration, which we should pay attention to. Both Helm and ApplicationSet use templating engines based on {{ ... }} placeholders. So to avoid conflicts we should escape Argo CD ApplicationSet templating elements from the Helm templating elements. The following part of the template responsible for generating the Argo CD Application name is a good example of that approach: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'. First, we use the AppliocationSet Git generator parameter index .path.segments 3 that returns the name of the third part of the directory path. Those elements are escaped with the ` char so Helm doesn’t try to analyze it.

Helm Chart Structure

Our ApplicationSets use the “Multiple Sources for Application” feature to read parameters from Helm values files and inject them into the Helm chart from a remote repository. Thanks to that, our configuration repositories for apps and components contain only values.yaml files in the standardized directory structure. The only chart we store in the sample repository has been described above and is responsible for creating the configuration required to run app Deployments on the cluster.

.
└── chart
    ├── Chart.yaml
    ├── templates
    │   ├── additional.yaml
    │   ├── applicationsets.yaml
    │   ├── appproject.yaml
    │   └── namespaces.yaml
    └── values.yaml

By default, each project defines three environments (stages): test, uat, prod.

stages:
  - name: test
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

We can override a default behavior for the specific project in Helm values. Each project directory contains the values.yaml file. Here are Helm parameters for the aaa-3 project that override CPU request quota from 2 CPUs to 4 CPUs only for the test environment.

stages:
  - name: test
    config:
      quotas:
        cpuRequest: 4
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

Run the Synchronization Process

Generate Global Structure on the Cluster

To start a process we must create the ApplicationSet that reads the structure of the projects directory. Each subdirectory in the projects directory indicates the name of our project. Our ApplicationSet uses a Git directory generator to create an Argo CD Application per each project. Its name contains the name of the subdirectory and the config suffix. Each generated Application uses the previously described Helm chart to create all namespaces, quotas, and other resources requested by the project. It also leverages the “Multiple Sources for Application” feature to allow us to override default Helm chart settings. It reads a project name from the directory name and passes it as a parameter to the generated Argo CD Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-config
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          - path: appset-helm-demo/projects/*
  template:
    metadata:
      name: '{{.path.basename}}-config'
    spec:
      destination:
        namespace: '{{.path.basename}}'
        server: 'https://kubernetes.default.svc'
      project: default
      sources:
        - path: appset-helm-demo/chart
          repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          helm:
            valueFiles:
              - $values/appset-helm-demo/projects/{{.path.basename}}/values.yaml
            parameters:
              - name: projectName
                value: '{{.path.basename}}'
              - name: argoNamespace
                value: openshift-gitops
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Once we create the global-config ApplicationSet object magic happens. Here’s the list of Argo CD Applications generated from our directories in Git configuration repositories.

First, there are three Argo CD Applications with the projects’ configuration. That’s happening because we defined 3 subdirectories in the projects directory with names aaa-1, aaa-2 and aaa-3.

The configuration applied by those Argo CD Applications is pretty similar since they are using the same Helm chart. We can look at the list of resources managed by the aaa-3-config Application. There are three namespaces (aaa-3-test, aaa-3-uat, aaa-3-prod) with resource quotas, a single Argo CD AppProject, and two ApplicationSet objects responsible for generating Argo CD Applications for apps and components directories.

In this configuration, we can verify if the value of the request.cpu ResourceQuota object has been overridden from 2 CPUs to 4 CPUs.

Let’s analyze what happened. Here’s a list of Argo CD ApplicationSets. The global-config ApplicationSet generated Argo CD Application per each detected project inside the projects directory. Then, each of these Applications applied two ApplicationSet objects to cluster using the Helm template.

$ kubectl get applicationset
NAME                      AGE
aaa-1-components-config   29m
aaa-1-apps-config         29m
aaa-2-components-config   29m
aaa-2-apps-config         29m
aaa-3-components-config   29m
aaa-3-apps-config         29m
global-config             29m

There’s also a list of created namespaces:

$ kubectl get ns
NAME                                               STATUS   AGE
aaa-1-prod                                         Active   34m
aaa-1-test                                         Active   34m
aaa-1-uat                                          Active   34m
aaa-2-prod                                         Active   34m
aaa-2-test                                         Active   34m
aaa-2-uat                                          Active   34m
aaa-3-prod                                         Active   34m
aaa-3-test                                         Active   34m
aaa-3-uat                                          Active   34m

Generate and Apply Deployments

Our sample configuration contains only two Deployments. We defined the basic subdirectory in the apps directory and the postgres subdirectory in the components directory inside the aaa-1 project. The aaa-2 and aaa-3 projects don’t contain any Deployments for simplification. However, the more subdirectories with the values.yaml file we create there, the more applications will be deployed on the cluster. Here’s a typical values.yaml file for a simple app deployed with a standard Helm chart. It defines the image repository, name, and tag. It also set the Deployment name and environment.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: prod

For the postgres component we must set more parameters in Helm values. Here’s the final list:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

The following Argo CD Application has been generated by the aaa-1-apps-config ApplicationSet. It detected the basic subdirectory in the apps directory. The basic subdirectory contained 3 subdirectories: test, uat and prod with values.yaml file. As a result, we have Argo CD per environment responsible for deploying the basic app in the target namespaces.

Here’s a list of resources managed by the basic-prod Application. It uses my custom Helm chart and applies Deployment and Service objects to the cluster.

The following Argo CD Application has been generated by the aaa-1-components-config ApplicationSet. It detected the basic subdirectory in the components directory. The postgres subdirectory contained 3 subdirectories: test, uat and prod with values.yaml and config.yaml files. The ApplicationSet Files generator reads the repository, name, and version from the configuration in the config.yaml file.

Here’s the config.yaml file with the Bitnami Postgres chart settings. We could place here any other chart we want to install something else on the cluster.

chart:
  repository: https://charts.bitnami.com/bitnami
  name: postgresql
  version: 15.5.38

Here’s the list of resources installed by the Bitnami Helm chart used by the generated Argo CD Applications.

Final Thoughts

This article proves that Argo CD ApplicationSet and Helm templates can be used together to create advanced configuration structures. It shows how to use ApplicationSet Git Directory and Files generators to analyze the structure of directories and files in the Git config repository. With that approach, we can propose a standardization in the configuration structure across the whole organization and propagate it similarly for all the applications deployed in the Kubernetes clusters. Everything can be easily managed at the cluster admin level with the single global Argo CD ApplicationSet that accesses many different repositories with configuration.

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

The solution to these challenges is Kargo, an open-source tool implementing continuous promotion within CI/CD pipelines. It provides a structured mechanism for promoting changes in complex environments involving Kubernetes and GitOps. I’ve been following Kargo for several months. It’s an interesting tool that provides stage-to-stage promotion using GitOps principles with Argo CD. It reached the version in October 2024. Let’s take a closer look at it.

If you are interested in promotion between environments on Kubernetes using the GitOps approach, you can read about another tool that tackles that challenge – Devtron. Here’s the link to my article that explains its concept.

Understand the Concept

Before we start with Kargo, we need to understand the concept around that tool. Let’s analyze several basic terms defined and implemented by Kargo.

A project is a bunch of related Kargo resources that describe one or more delivery pipelines. It’s the basic unit of organization and multi-tenancy in Kargo. Every Kargo project has its own cluster-scoped Kubernetes resource of type Project. We should put all the resources related to a certain project into the same Kubernetes namespace.

A stage represents environments in Kargo. Stages are the most important concept in Kargo. We can link them together in a directed acyclic graph to describe a delivery pipeline. Typically, a delivery pipeline starts with a test or dev stage and ends with one or more prod stages.

A Freight object represents resources that Kargo promotes from one stage to another. It can reference one or more versioned artifacts, such as container images, Kubernetes manifests loaded from Git repositories, or Helm charts from chart repositories. 

A warehouse is a source of freight. It can refer to container image repositories, Git, or Helm chart repositories.

In that context, we should treat a promotion as a request to move a piece of freight into a specified stage.

Source Code

If you want to try out this exercise, go ahead and take a look at my source code. To do that, just clone my GitHub repository. It contains the sample Spring Boot application in the basic directory. The application exposes a single REST endpoint that returns the app Maven version number. Go to that directory. Then, you can follow my further instructions.

Kargo Installation

A few things must be ready before installing Kargo. We must have a Kubernetes cluster and the helm CLI installed on our laptop. I use Minikube. Kargo integrates with Cert-Manager, Argo CD, and Argo Rollouts. We can install all those tools using official Helm charts. Let’s begin with Cert-Manager. First, we must add the jetstack Helm repository:

helm repo add jetstack https://charts.jetstack.io

Here’s the helm command that installs it in the cert-manager namespace.

helm install cert-manager --namespace cert-manager jetstack/cert-manager \
  --set crds.enabled=true \
  --set crds.keep=true

To install Argo CD, we must first add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

Then, we can install Argo CD in the argocd namespace.

helm install argo-cd argo/argo-cd

The Argo Rollouts chart is located in the same Helm repository as the Argo CD chart. We will also install it in the argocd namespace:

helm install my-argo-rollouts argo/argo-rollouts

Finally, we can proceed to the Kargo installation. We will install it in the kargo namespace. The installation command sets two Helm parameters. We should set the Bcrypt password hash and a key used to sign JWT tokens for the admin account:

helm install kargo \
  oci://ghcr.io/akuity/kargo-charts/kargo \
  --namespace kargo \
  --create-namespace \
  --set api.adminAccount.passwordHash='$2y$10$xu2U.Ux5nV5wKmerGcrDlO261YeiTlRrcp2ngDGPxqXzDyiPQvDXC' \
  --set api.adminAccount.tokenSigningKey=piomin \
  --wait

We can generate and print the password hash using e.g. htpasswd. Here’s the sample command:

htpasswd -nbB admin 123456

After installation is finished, we can verify it by displaying a list of pods running in the kargo namespace.

$ kubectl get po -n kargo
NAME                                          READY   STATUS    RESTARTS   AGE
kargo-api-dbb4d5cb7-zvnc6                     1/1     Running   0          44s
kargo-controller-c4964bbb7-4ngnv              1/1     Running   0          44s
kargo-management-controller-dc5569759-596ch   1/1     Running   0          44s
kargo-webhooks-server-6df6dd58c-g5jlp         1/1     Running   0          44s

Kargo provides a UI dashboard that allows us to display and manage continuous promotion configuration. Let’s expose it locally on the 8443 port using a port-forward feature:

kubectl port-forward svc/kargo-api -n kargo 8443:443

Once we sign in to the dashboard using the admin password set during an installation we can create a new kargo-demo project:

Sample Application

Our sample application is simple. It exposes a single GET /basic/ping endpoint that returns the version number read from Maven pom.xml.

@RestController
@RequestMapping("/basic")
public class BasicController {

    @Autowired
    Optional<BuildProperties> buildProperties;

    @GetMapping("/ping")
    public String ping() {
        return "I'm basic:" + buildProperties.orElseThrow().getVersion();
    }
}

We will build an application image using the Jib Maven Plugin. It is already configured in Maven pom.xml. I set my Docker Hub as the target registry, but you can relate it to your account.

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.4.4</version>
  <configuration>
    <container>
      <user>1001</user>
    </container>
    <to>
      <image>piomin/basic:${project.version}</image>
    </to>
  </configuration>
</plugin>

The following Maven command builds the application and its image from a source code. Before the build, we should increase the version number in the project.version field in pom.xml. We begin with the 1.0.0 version, which should be pushed to your registry before proceeding.

mvn clean package -DskipTests jib:build

Here’s the result of my initial build.

Let’s switch to the Docker Registry dashboard after pushing the 1.0.0 version.

Configure Kargo for Promotion on Kubernetes

First, we will create the Kargo Warehouse object. It refers to the piomin/basic repository containing the image with our sample app. The Warehouse object is responsible for discovering new image tags pushed into the registry. We also use my Helm chart to deploy the image to Kubernetes. However, we will only use the latest version of that chart. Otherwise, we should also place that chart inside the basic Warehouse object to enable new chart version discovery.

apiVersion: kargo.akuity.io/v1alpha1
kind: Warehouse
metadata:
 name: basic
 namespace: demo
spec:
 subscriptions:
 - image:
     discoveryLimit: 5
     repoURL: piomin/basic

Then, we will create the Argo CD ApplicationSet to generate an application per environment. There are three environments: test, uat, prod. Each Argo CD application must be annotated by kargo.akuity.io/authorized-stage containing the project and stage name. Argo uses multiple sources. The argocd-showcase repository contains Helm values files with parameters per each stage. The piomin.github.io/helm-charts repository provides the spring-boot-api-app Helm chart that refers to those values.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
 name: demo
 namespace: argocd
spec:
 generators:
 - list:
     elements:
     - stage: test
     - stage: uat
     - stage: prod
 template:
   metadata:
     name: demo-{{stage}}
     annotations:
       kargo.akuity.io/authorized-stage: demo:{{stage}}
   spec:
     project: default
     sources:
       - chart: spring-boot-api-app
         repoURL: 'https://piomin.github.io/helm-charts/'
         targetRevision: 0.3.8
         helm:
           valueFiles:
             - $values/values/values-{{stage}}.yaml
       - repoURL: 'https://github.com/piomin/argocd-showcase.git'
         targetRevision: HEAD
         ref: values
     destination:
       server: https://kubernetes.default.svc
       namespace: demo-{{stage}}
     syncPolicy:
       syncOptions:
       - CreateNamespace=true

Now, we can proceed to the most complex element of our exercise – stage creation. The Stage object refers to the previously created Warehouse object to request freight to promote. Then, it defines the steps to perform during the promotion process. Kargo’s promotion steps define the workflow of a promotion process. They do the things needed to promote a piece of freight into the next stage. We can use several built-in steps that cover the most common operations like cloning Git repo, updating Helm values, or pushing changes to the remote repository. Our stage definition contains five steps. After cloning the repository with Helm values, we must update the image.tag parameter in the values-test.yaml file with the tag value read from the basic Warehouse. Then, Kargo commits and pushes changes to the configuration repository and triggers Argo CD application synchronization.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
 name: test
 namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     direct: true
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Here’s the values-test.yaml file in the Argo CD configuration repository.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: test

Here’s the Stage definition of the uat environment. It is pretty similar to the definition of the test environment. It just defines the previous source stage to test.

apiVersion: kargo.akuity.io/v1alpha1
kind: Stage
metadata:
  name: uat
  namespace: demo
spec:
 requestedFreight:
 - origin:
     kind: Warehouse
     name: basic
   sources:
     stages:
       - test
 promotionTemplate:
   spec:
     vars:
     - name: gitRepo
       value: https://github.com/piomin/argocd-showcase.git
     - name: imageRepo
       value: piomin/basic
     steps:
       - uses: git-clone
         config:
           repoURL: ${{ vars.gitRepo }}
           checkout:
           - branch: master
             path: ./out
       - uses: helm-update-image
         as: update-image
         config:
           path: ./out/values/values-${{ ctx.stage }}.yaml
           images:
           - image: ${{ vars.imageRepo }}
             key: image.tag
             value: Tag
       - uses: git-commit
         as: commit
         config:
           path: ./out
           messageFromSteps:
           - update-image
       - uses: git-push
         config:
           path: ./out
       - uses: argocd-update
         config:
           apps:
           - name: demo-${{ ctx.stage }}
             sources:
             - repoURL: ${{ vars.gitRepo }}
               desiredRevision: ${{ outputs.commit.commit }}

Perform Promotion Process

Once a new image tag is published to the registry, it becomes visible in the Kargo Dashboard. We must click the “Promote into Stage” button to promote a selected version to the target stage.

Then we should specify the source image tag. A choice is obvious since we only have the image tagged with 1.0.0. After approving the selection by clicking the “Yes” button Kargo starts a promotion process on Kubernetes.

After a while, we should have a new image promoted to the test stage.

Let’s repeat the promotion process of the 1.0.0 version for the other two stages. Each stage should be at a healthy status. That status is read directly from the corresponding Argo CD Application.

Let’s switch to the Argo CD dashboard. There are three applications.

We can make a test call of the sample application HTTP endpoint. Currently, all the environments run the same 1.0.0 version of the app. Let’s enable port forwarding for the basic service in the demo-prod namespace.

kubectl port-forward svc/basic -n demo-prod 8080:8080

The endpoint returns the application name and version as a response.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.0

Then, we will build another three versions of the basic application beginning from 1.0.1 to 1.0.5.

Once we push each version to the registry, we can refresh the list of images. With the default configuration, Kargo should add the latest version to the list. After pushing the 1.0.3 version I promoted it to the test stage. Then I refreshed a list after pushing the 1.0.4 tag. Now, the 1.0.3 tag can be promoted to a higher environment. In the illustration below, I’m promoting it to the uat stage.

After that, we can promote the 1.0.4 version to the test stage, and refresh a list of images once again to see the currently pushed 1.0.5 tag.

Here’s another promotion. This time, I moved the 1.0.3 version to the prod stage.

After clicking on the image tag tile, we will see its details. For example, the 1.0.3 tag has been verified on the test and uat stages. There is also an approval section. However, we still didn’t approve freight. To do that, we need to switch to the kargo CLI.

The kargo CLI binary for a particular OS on the project GitHub releases page. We must download and copy it to the directory under the PATH. Then, we can sign in to the Kargo server running on Kubernetes using admin credentials.

kargo login https://localhost:8443 --admin \
  --password 123456 \
  --insecure-skip-tls-verify

We can approve a specific freight. Let’s display a list of Freight objects.

$ kubectl get freight -n demo
NAME                                       ALIAS            ORIGIN (KIND)   ORIGIN (NAME)   AGE
0501f8d8018a953821ea437078c1ec34e6db5a6b   ideal-horse      Warehouse       basic           8m7s
683be41b1cef57ed755fc7a0f8e8d7776f90c63a   wiggly-tuatara   Warehouse       basic           11m
6a1219425ddeceabfe94f0605d7a5f6d9d20043e   ulterior-zebra   Warehouse       basic           13m
f737178af6492ea648f85fc7d082e34b7a085927   eager-snail      Warehouse       basic           7h49m

The kargo approve command takes Freight ID as the input parameter.

kargo approve --freight 6a1219425ddeceabfe94f0605d7a5f6d9d20043e \
  --stage uat \
  --project demo

Now, the image tag details window should display the approved stage name.

Let’s enable port forwarding for the basic service in the demo-uat namespace.

kubectl port-forward svc/basic -n demo-uat 8080:8080

Then we call the /basic/ping endpoint to check out the current version.

$ curl http://localhost:8080/basic/ping
I'm basic:1.0.3

Final Thoughts

This article explains the idea behind continuous app promotion between environments on Kubernetes with Kargo and Argo CD. Kargo is a relatively new project in the Kubernetes ecosystem, that smoothly addresses the challenges related to GitOps promotion. It seems promising. I will closely monitor the further development of this project.

The post Continuous Promotion on Kubernetes with GitOps appeared first on Piotr's TechBlog.

Before we begin, let me clarify some things and explain my decisions. If you were searching for something about Azure DevOps and OpenShift integration, you came across several articles about the Red Hat Azure DevOps extension for OpenShift. I won’t use that extension. In my opinion, it is not actively developed right now, and therefore it provides some limitations that may complicate our integration process. On the other hand, it won’t offer many useful features, so we can do just as well without it.

You will also find articles that show how to prepare a self-hosted agent image based on the Red Hat dotnet-runtime as a base image (e.g. this one). I also won’t use it. Instead, I’m going to leverage the image built on top of UBI9 provided by the tool called Blue Agent (formerly Azure Pipelines Agent). It is a self-hosted Azure Pipelines agent for Kubernetes, easy to run, secure, and auto-scaled. We will have to modify that image slightly, but more details later.

Prerequisites

In order to proceed with the exercise, we need an active subscription to the Azure cloud and an instance of Azure DevOps. We also have to run the OpenShift cluster, which is accessible for our pipelines running on Azure DevOps. I’m running that cluster also on the Azure cloud using the Azure Red Hat OpenShift managed service (ARO). The details about Azure DevOps creation or OpenShift installation are out of the scope of that article.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. Today you will have to clone two sample Git repositories. The first one contains a sample Spring Boot app used in our exercise. We will try to build that app with Azure Pipelines and deploy it on OpenShift. The second repository is a fork of the official Blue Agent repository. It contains a new version of the Dockerfile for our sample self-agent image based on Red Hat UBI9. Once you clone both of these repositories, you just need to follow my instructions.

Azure DevOps Self-Hosted Agent on OpenShift with Blue Agent

Build the Agent Image

In this section, we will build the image of the self-agent based on UBI9. Then, we will run it on the OpenShift cluster. We need to open the Dockerfile in the repository located in the src/docker/Dockerfile-ubi9 path. We don’t need to change much inside that file. It contains several clients, e.g. for AWS or Azure interaction. We will include the line for installing the oc client, which allows us to interact with OpenShift.

RUN curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz -o - | tar zxvf - -C /usr/bin/

After that, we need to build a new image. You can completely omit this step and pull the final image published on my Docker account and available under the piomin/blue-agent:ubi9 tag.

$ docker build -t piomin/blue-agent:ubi9 -f Dockerfile-ubi9 \
  --build-arg JQ_VERSION=1.6 \
  --build-arg AZURE_CLI_VERSION=2.63.0 \
  --build-arg AWS_CLI_VERSION=2.17.42 \
  --build-arg GCLOUD_CLI_VERSION=490.0.0 \
  --build-arg POWERSHELL_VERSION=7.2.23 \
  --build-arg TINI_VERSION=0.19.0 \
  --build-arg BUILDKIT_VERSION=0.15.2  \
  --build-arg AZP_AGENT_VERSION=3.243.1 \
  --build-arg ROOTLESSKIT_VERSION=2.3.1 \
  --build-arg GO_VERSION=1.22.7 \
  --build-arg YQ_VERSION=4.44.3 .

We can use a Helm chart to install Blue Agent on Kubernetes. This project is still under active development. I could not customize it with parameters to prepare a chart for installing Blue Agent on OpenShift. So, I just set some of them inside the values.yaml file:

pipelines:
  organizationURL: https://dev.azure.com/pminkows
  personalAccessToken: <AZURE_DEVOPS_PERSONAL_ACCESS_TOKEN>
  poolName: Default

image:
  repository: piomin/blue-agent
  version: 2
  flavor: ubi9

Deploy Agent on OpenShift

We can generate the YAML manifests for the defined parameters without installing it with the following command:

$ helm template -f blue-agent-values.yaml --dry-run .

In order to run it on OpenShift, we need to customize the Deployment object. First of all, I had to grant the privileged SCC (Security Context Constraint) to the container and remove some fields from the securityContext section. Here’s our Deployment object. It refers to the objects previously generated by the Helm chart: agent-blue-agent ServiceAccount and the Secret with the same name.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: agent-blue-agent    
  labels:
    app.kubernetes.io/component: agent
    app.kubernetes.io/instance: agent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: blue-agent
    app.kubernetes.io/part-of: blue-agent
    app.kubernetes.io/version: 3.243.1
    helm.sh/chart: blue-agent-7.0.3
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: agent
      app.kubernetes.io/name: blue-agent
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: agent
        app.kubernetes.io/name: blue-agent
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      serviceAccountName: agent-blue-agent
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 3600
      securityContext: {}
      containers:
        - resources:
            limits:
              cpu: '2'
              ephemeral-storage: 8Gi
              memory: 4Gi
            requests:
              cpu: '1'
              ephemeral-storage: 2Gi
              memory: 2Gi
          terminationMessagePath: /dev/termination-log
          lifecycle:
            preStop:
              exec:
                command:
                  - bash
                  - '-c'
                  - ''
                  - 'rm -rf ${AZP_WORK};'
                  - 'rm -rf ${TMPDIR};'
          name: azp-agent
          env:
            - name: AGENT_DIAGLOGPATH
              value: /app-root/azp-logs
            - name: VSO_AGENT_IGNORE
              value: AZP_TOKEN
            - name: AGENT_ALLOW_RUNASROOT
              value: '1'
            - name: AZP_AGENT_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: AZP_URL
              valueFrom:
                secretKeyRef:
                  name: agent-blue-agent
                  key: organizationURL
            - name: AZP_POOL
              value: Default
            - name: AZP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: agent-blue-agent
                  key: personalAccessToken
            - name: flavor_ubi9
            - name: version_7.0.3
          securityContext:
            privileged: true
          imagePullPolicy: Always
          volumeMounts:
            - name: azp-logs
              mountPath: /app-root/azp-logs
            - name: azp-work
              mountPath: /app-root/azp-work
            - name: local-tmp
              mountPath: /app-root/.local/tmp
          terminationMessagePolicy: File
          image: 'piomin/blue-agent:ubi9'
      serviceAccount: agent-blue-agent
      volumes:
        - name: azp-logs
          emptyDir:
            sizeLimit: 1Gi
        - name: azp-work
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 10Gi
                storageClassName: managed-csi
                volumeMode: Filesystem
        - name: local-tmp
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 1Gi
                storageClassName: managed-csi
                volumeMode: Filesystem
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 50%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600

After removing the --dry-run option from the helm template command we can install the solution on OpenShift. It’s worth noting that there were some problems with the Deployment object in the Helm chart, so we will apply it directly later.

Before applying Deployment we have to add privileged SCC to the ServiceAccount used by the agent. By the way, adding privileged SCC is not the best approach to managing containers on OpenShift. However, the oc client installed in the image creates a directory during the login procedure

$ oc adm policy add-scc-to-user privileged -z agent-blue-agent

Once we deploy an agent on OpenShift we should see three running pods. They are trying to connect to the Default pool defined in the Azure DevOps for our self-hosted agent.

$ oc get po
NAME                                READY   STATUS    RESTARTS   AGE
agent-blue-agent-5458b9b76d-2gk2z   1/1     Running   0          13h
agent-blue-agent-5458b9b76d-nwcxh   1/1     Running   0          13h
agent-blue-agent-5458b9b76d-tcpnp   1/1     Running   0          13h

Connect Self-hosted Agent to Azure DevOps

Let’s switch to the Azure DevOps instance. Then, we should go to our project. After that, we need to go to the Organization Settings -> Agent pools. We need to create a new agent pool. The pool name depends on the name configured for the Blue Agent deployed on OpenShift. In our case, this name is Default.

Once we create a pool we should see all three instances of agents in the following list.

We can switch to the OpenShift once again. Then, let’s take a look at the logs printed out by one of the agents. As you see, it has been started successfully and listens for the incoming jobs.

If agents are running on OpenShift and they successfully connect with Azure DevOps, we finished the first part of our exercise. Now, we can create a pipeline for our sample Spring Boot app.

Create Azure DevOps Pipeline for OpenShift

Azure Pipeline Definition

Firstly, we need to go to the sample-spring-boot-web repository. The pipeline is configured in the azure-pipelines.yml file in the repository root directory. Let’s take a look at it. It’s very simple simple. I’m just using the Azure DevOps Command-Line task to interact with OpenShift through the oc client. Of course, it has to define the target agent pool used for running its jobs (1). In the first step, we need to log in to the OpenShift cluster (2). We can use the internal address of the Kubernetes Service, since the agent is running inside the cluster. All the actions should be performed inside our sample myapp project (3). We need to create BuildConfig and Deployment objects using the templates from the repository (4). The pipeline uses its BuildId parameter to the output image. Finally, it starts the build configured by the previously applied BuildConfig object (5).

trigger:
- master

# (1)
pool:
  name: Default

steps:
# (2)
- task: CmdLine@2
  inputs:
    script: 'oc login https://172.30.0.1:443 -u kubeadmin -p $(OCP_PASSWORD) --insecure-skip-tls-verify=true'
  displayName: Login to OpenShift
# (3)
- task: CmdLine@2
  inputs:
    script: 'oc project myapp'
  displayName: Switch to project
# (4)
- task: CmdLine@2
  inputs:
    script: 'oc process -f ocp/openshift.yaml -o yaml -p IMAGE_TAG=v1.0-$(Build.BuildId) -p NAMESPACE=myapp | oc apply -f -'
  displayName: Create build
# (5)
- task: CmdLine@2
  inputs:
    script: 'oc start-build sample-spring-boot-web-bc -w'
    failOnStderr: true
  displayName: Start build
  timeoutInMinutes: 5
- task: CmdLine@2
  inputs:
    script: 'oc status'
  displayName: Check status

Integrate Azure Pipelines with OpenShift

We use the OpenShift Template object for defining the YAML manifest with Deployment and BuildConfig. The BuildConfig object typically manages the image build process on OpenShift. In order to build the image directly from the source code, it uses the source-2-image (S2I) tool. We need to set at least three parameters to configure the process properly. The first of them is the address of the output image (1). We can use the internal OpenShift repository available at image-registry.openshift-image-registry.svc:5000 or any other external registry like Quay or Docker Hub. We should also set the name of the builder image (2). Our app requires at least Java 21. Of course, the process requires the source code repository as the input (3). At the same time, we define the Deployment object. It uses the image previously built by the BuildConfig object (4). The whole template takes two input parameters: IMAGE_TAG and NAMESPACE (5).

ind: Template
apiVersion: template.openshift.io/v1
metadata:
  name: sample-spring-boot-web-tmpl
objects:
  - kind: BuildConfig
    apiVersion: build.openshift.io/v1
    metadata:
      name: sample-spring-boot-web-bc
      labels:
        build: sample-spring-boot-web-bc
    spec:
      # (1)
      output:
        to:
          kind: DockerImage
          name: 'image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/sample-spring-boot-web:${IMAGE_TAG}'
      # (2)
      strategy:
        type: Source
        sourceStrategy:
          from:
            kind: ImageStreamTag
            namespace: openshift
            name: 'openjdk-21:stable'
      # (3)
      source:
        type: Git
        git:
          uri: 'https://github.com/piomin/sample-spring-boot-web.git'
  - kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: sample-spring-boot-web
      labels:
        app: sample-spring-boot-web
        app.kubernetes.io/component: sample-spring-boot-web
        app.kubernetes.io/instance: sample-spring-boot-web
    spec:
      replicas: 1
      selector:
        matchLabels:
          deployment: sample-spring-boot-web
      template:
        metadata:
          labels:
            deployment: sample-spring-boot-web
        spec:
          containers:
            - name: sample-spring-boot-web
              # (4)
              image: 'image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/sample-spring-boot-web:${IMAGE_TAG}'
              ports:
                - containerPort: 8080
                  protocol: TCP
                - containerPort: 8443
                  protocol: TCP
# (5)
parameters:
  - name: IMAGE_TAG
    displayName: Image tag
    description: The output image tag
    value: v1.0
    required: true
  - name: NAMESPACE
    displayName: Namespace
    description: The OpenShift Namespace where the ImageStream resides
    value: openshift

Currently, OpenShift doesn’t provide OpenJDK 21 image by default. So we need to manually import it to our cluster before running the pipeline.

$ oc import-image openjdk-21:stable \
  --from=registry.access.redhat.com/ubi9/openjdk-21:1.20-2.1725851045 \
  --confirm

Now, we can create a pipeline in Azure DevOps. In order to do it, we need to go to the Pipelines section, and then click the New pipeline button. After that, we just need to pass the address of our repository with the pipeline definition.

Our pipeline requires the OCP_PASSWORD input parameter with the OpenShift admin user password. We can set it as the pipeline secret variable. In order to do that, we need to edit the pipeline and then click the Variables button.

Run the Pipeline

Finally, we can run our pipeline. If everything finishes successfully, the status of the job is Success. It takes around one minute to execute all the steps defined in the pipeline.

We can see detailed logs for each step.

Each time, we run a pipeline a new build on OpenShift starts. Note, that the pipeline updates the BuildConfig object with the new version of the output image.

We can take a look at detailed logs of each build. Within such a build OpenShift starts a new pod, which performs the whole process. It uses the S2I approach for building the image from the source code and pushing that image to the internal OpenShift registry.

Finally, let’s take a look at the Deployment. As you see, it was reloaded with the latest version of the image and worked fine.

$ oc describe deploy sample-spring-boot-web -n myapp
Name:                   sample-spring-boot-web
Namespace:              myapp
CreationTimestamp:      Wed, 11 Sep 2024 16:09:11 +0200
Labels:                 app=sample-spring-boot-web
                        app.kubernetes.io/component=sample-spring-boot-web
                        app.kubernetes.io/instance=sample-spring-boot-web
Annotations:            deployment.kubernetes.io/revision: 3
Selector:               deployment=sample-spring-boot-web
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  deployment=sample-spring-boot-web
  Containers:
   sample-spring-boot-web:
    Image:        image-registry.openshift-image-registry.svc:5000/myapp/sample-spring-boot-web:v1.0-134
    Ports:        8080/TCP, 8443/TCP
    Host Ports:   0/TCP, 0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  sample-spring-boot-web-78559697c9 (0/0 replicas created), sample-spring-boot-web-7985d6b844 (0/0 replicas created)
NewReplicaSet:   sample-spring-boot-web-5584447f5d (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  19m   deployment-controller  Scaled up replica set sample-spring-boot-web-5584447f5d to 1
  Normal  ScalingReplicaSet  17m   deployment-controller  Scaled down replica set sample-spring-boot-web-7985d6b844 to 0 from 1

By the way, Azure Pipelines jobs are “load-balanced” between the agents. Here’s the name of the agent used for the first pipeline run.

Here’s the name of the agent used for the second pipeline run.

There are three instances of the agent running on OpenShift. So, Azure DevOps can process max 3 pipeline runs. By the way, we could enable autoscaling for Blue Agent based on KEDA.

Final Thoughts

In this article, I showed the simplest and most OpenShift-native approach to building CI/CD pipelines on Azure DevOps. We just need to use the oc client on the agent image and the OpenShift BuildConfig object to orchestrate the whole process of building and deploying the Spring Boot app on the cluster. Of course, we could implement the same process in several different. For example, we could leverage the plugins for Kubernetes and completely omit the tools provided by OpenShift. On the other hand, it is possible to use Argo CD for the delivery phase, and Azure Pipelines for the integration phase.

The post Azure DevOps with OpenShift appeared first on Piotr's TechBlog.

You can find several other articles about Quarkus on my blog. For example, you can read about advanced testing techniques with Quarkus here. There is also an interesting article about Quarkus the Testcontainer’s support in the local development with Kafka.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. It contains three microservices written in Quarkus. I migrated them from Quarkus 2 to 3, to the latest version of the Pact Quarkus extension, and from Java 17 to 21. In order to proceed with the exercise, you need to clone my GitHub repository. Then you should just follow my instructions.

Let’s do a quick recap before proceeding. We are implementing several contact tests with Pact to verify interactions between our three microservices: employee-service, department-service, and organization-service. We use Pact Broker to store and share contract definitions between the microservices. Here’s the diagram that illustrates the described architecture

Update to Java 21

There are no issues with migration to Java 21 in Quarkus. We need to change the version of Java used in the Maven compilation inside the pom.xml file. However, the situation is more complicated with the CircleCI build. Firstly, we use the ubuntu-2204 machine in the builds to access the Docker daemon. We need Docker to run the container with the Pact broker. Although CircleCI provides the image for OpenJDK 21, there is still Java 17 installed on the latest version of ubuntu-2204. This situation will probably change during the next months. But now, we need to install OpenJDK 21 on that machine. After that, we may run Pact broker and JUnit tests using the latest Java LTS version. Here’s the CircleCI config.yaml file:

version: 2.1

jobs:
  analyze:
    executor:
      name: docker/machine
      image: ubuntu-2204:2024.01.2
    steps:
      - checkout
      - run:
          name: Install OpenJDK 21
          command: |
            java -version
            sudo apt-get update && sudo apt-get install openjdk-21-jdk
            sudo update-alternatives --set java /usr/lib/jvm/java-21-openjdk-amd64/bin/java
            sudo update-alternatives --set javac /usr/lib/jvm/java-21-openjdk-amd64/bin/javac
            java -version
            export JAVA_HOME=/usr/lib/jvm/java-21-openjdk-amd64
      - docker/install-docker-compose
      - maven/with_cache:
          steps:
            - run:
                name: Build Images
                command: mvn package -DskipTests -Dquarkus.container-image.build=true
      - run:
          name: Run Pact Broker
          command: docker-compose up -d
      - maven/with_cache:
          steps:
            - run:
                name: Run Tests
                command: mvn package pact:publish -Dquarkus.container-image.build=false
      - maven/with_cache:
          steps:
            - run:
                name: Sonar Analysis
                command: mvn package sonar:sonar -DskipTests -Dquarkus.container-image.build=false


orbs:
  maven: circleci/maven@1.4.1
  docker: circleci/docker@2.6.0

workflows:
  maven_test:
    jobs:
      - analyze:
          context: SonarCloud

Here’s the root Maven pom.xml. It declares the Maven plugin responsible for publishing contracts to the Pact broker. Each time the Pact JUnit is executed successfully, it tries to publish the JSON pacts to the broker. The ordering of Maven modules is not random. The organization-service generates and publishes pacts for verifying contracts with department-service and employee-service, so it has to be run at the beginning. As you see, we use the current latest version of Quarkus – 3.9.3.

<properties>
  <java.version>21</java.version>
  <surefire-plugin.version>3.2.5</surefire-plugin.version>
  <quarkus.version>3.9.3</quarkus.version>
  <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
  <maven.compiler.source>${java.version}</maven.compiler.source>
  <maven.compiler.target>${java.version}</maven.compiler.target>
</properties>

<modules>
  <module>organization-service</module>
  <module>department-service</module>
  <module>employee-service</module>
</modules>

<build>
  <plugins>
    <plugin>
      <groupId>au.com.dius.pact.provider</groupId>
      <artifactId>maven</artifactId>
        <version>4.6.9</version>
      <configuration>
        <pactBrokerUrl>http://localhost:9292</pactBrokerUrl>
      </configuration>
    </plugin>
  </plugins>
</build>

Here’s the part of the docker-compose.yml responsible for running a Pact broker. It requires a Postgres database.

version: "3.7"
services:
  postgres:
    container_name: postgres
    image: postgres
    environment:
      POSTGRES_USER: pact
      POSTGRES_PASSWORD: pact123
      POSTGRES_DB: pact
    ports:
      - "5432"
  pact-broker:
    container_name: pact-broker
    image: pactfoundation/pact-broker
    ports:
      - "9292:9292"
    depends_on:
      - postgres
    links:
      - postgres
    environment:
      PACT_BROKER_DATABASE_USERNAME: pact
      PACT_BROKER_DATABASE_PASSWORD: pact123
      PACT_BROKER_DATABASE_HOST: postgres
      PACT_BROKER_DATABASE_NAME: pact

Update Quarkus and Pact

Dependencies

Firstly, let’s take a look at the list of dependencies. With the latest versions of Quarkus, we should take care of the REST provider and client used in our app. For example, if we use the quarkus-resteasy-jackson module to expose REST services, we should also use the quarkus-resteasy-client-jackson module to call the services. On the other hand, if we use quarkus-rest-jackson on the server side, we should also use quarkus-rest-client-jackson on the client side. In order to implement Pact tests in our app, we need to include the quarkus-pact-consumer module for the contract consumer and the quarkus-pact-provider on the contract provider side. Finally, we will use Wiremock to replace a Pact mock server.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-resteasy-jackson</artifactId>
</dependency>
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-resteasy-client-jackson</artifactId>
</dependency>
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-junit5</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-junit5-mockito</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>io.rest-assured</groupId>
  <artifactId>rest-assured</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>io.quarkiverse.pact</groupId>
  <artifactId>quarkus-pact-consumer</artifactId>
  <version>1.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>io.quarkiverse.pact</groupId>
  <artifactId>quarkus-pact-provider</artifactId>
  <version>1.3.0</version>
  <scope>provided</scope>
</dependency>
<dependency>
  <groupId>com.github.tomakehurst</groupId>
  <artifactId>wiremock-jre8</artifactId>
  <version>3.0.1</version>
  <scope>test</scope>
</dependency>

Tests Implementation with Quarkus 3 and Pact Consumer

In that exercise, I’m simplifying tests as much as possible. Therefore we will use the REST client directly to verify the contract on the consumer side. However, if you are looking for more advanced examples please go to that repository. Coming back to our exercise, let’s take a look at the example of a declarative REST client used in the department-service.

@ApplicationScoped
@Path("/employees")
@RegisterRestClient(configKey = "employee")
public interface EmployeeClient {

    @GET
    @Path("/department/{departmentId}")
    @Produces(MediaType.APPLICATION_JSON)
    List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId);

}

There are some significant changes in the Pact tests on the consumer side. Some of them were forced by the error related to migration to Quarkus 3 described here. I found a smart workaround for that problem proposed by one of the contributors (1). This workaround replaces the Pact built-in mock server with Wiremock. We will start Wiremock on the dynamic port (2). We also need to implement @QuarkusTestResource to start the Wiremock container before the tests and shut it down after the tests (3). Then, we can switch to the latest version of Pact API by returning the V4Pact object (4) in the @Pact method and updating the @PactTestFor annotation accordingly (5). Finally, instead of the Pact MockServer, we use the wrapper PactMockServer dedicated to Wiremock (6).

@QuarkusTest
@ExtendWith(PactConsumerTestExt.class)
@ExtendWith(PactMockServerWorkaround.class) // (1)
@MockServerConfig(port = "0") // (2)
@QuarkusTestResource(WireMockQuarkusTestResource.class) // (3)
public class EmployeeClientContractTests extends PactConsumerTestBase {

    @Pact(provider = "employee-service", consumer = "department-service")
    public V4Pact callFindDepartment(PactDslWithProvider builder) { // (4)
        DslPart body = PactDslJsonArray.arrayEachLike()
                .integerType("id")
                .stringType("name")
                .stringType("position")
                .numberType("age")
                .closeObject();
        return builder.given("findByDepartment")
                .uponReceiving("findByDepartment")
                    .path("/employees/department/1")
                    .method("GET")
                .willRespondWith()
                    .status(200)
                    .body(body).toPact(V4Pact.class);
    }

    @Test
    // (5)
    @PactTestFor(providerName = "employee-service", pactVersion = PactSpecVersion.V4)
    public void verifyFindDepartmentPact(final PactMockServer mockServer) { // (6)
        EmployeeClient client = RestClientBuilder.newBuilder()
                .baseUri(URI.create(mockServer.getUrl()))
                .build(EmployeeClient.class);
        List<Employee> employees = client.findByDepartment(1L);
        System.out.println(employees);
        assertNotNull(employees);
        assertTrue(employees.size() > 0);
        assertNotNull(employees.get(0).getId());
    }
}

Here’s our PactMockServer wrapper:

public class PactMockServer {

    private final String url;
    private final int port;

    public PactMockServer(String url, int port) {
        this.url = url;
        this.port = port;
    }

    public String getUrl() {
        return url;
    }

    public int getPort() {
        return port;
    }
}

Implement Mock Server with Wiremock

In the first step, we need to provide an implementation of the QuarkusTestResourceLifecycleManager for starting the Wiremock server during the tests.

public class WireMockQuarkusTestResource implements 
        QuarkusTestResourceLifecycleManager {
        
    private static final Logger LOGGER = Logger
       .getLogger(WireMockQuarkusTestResource.class);

    private WireMockServer wireMockServer;

    @Override
    public Map<String, String> start() {
        final HashMap<String, String> result = new HashMap<>();

        this.wireMockServer = new WireMockServer(options()
                .dynamicPort()
                .notifier(createNotifier(true)));
        this.wireMockServer.start();

        return result;
    }

    @Override
    public void stop() {
        if (this.wireMockServer != null) {
            this.wireMockServer.stop();
            this.wireMockServer = null;
        }
    }

    @Override
    public void inject(final TestInjector testInjector) {
        testInjector.injectIntoFields(wireMockServer,
          new TestInjector.AnnotatedAndMatchesType(InjectWireMock.class, 
                                                   WireMockServer.class));
    }

    private static Notifier createNotifier(final boolean verbose) {
        final String prefix = "[WireMock] ";
        return new Notifier() {

            @Override
            public void info(final String s) {
                if (verbose) {
                    LOGGER.info(prefix + s);
                }
            }

            @Override
            public void error(final String s) {
                LOGGER.warn(prefix + s);
            }

            @Override
            public void error(final String s, final Throwable throwable) {
                LOGGER.warn(prefix + s, throwable);
            }
        };
    }
}

Let’s create the annotation:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface InjectWireMock {
}

I’m not very sure it is required. But here’s the test base extended by our tests.

public class PactConsumerTestBase {

   @InjectWireMock
   protected WireMockServer wiremock;

   @BeforeEach
   void initWiremockBeforeEach() {
      wiremock.resetAll();
      configureFor(new WireMock(this.wiremock));
   }

   protected void forwardToPactServer(final PactMockServer wrapper) {
      wiremock.resetAll();  
      stubFor(any(anyUrl())
         .atPriority(1)
         .willReturn(aResponse().proxiedFrom(wrapper.getUrl()))
      );
   }

}

Here’s the workaround implementation used as the test extension included with the @ExtendWith annotation:

public class PactMockServerWorkaround implements ParameterResolver {
    
  @Override
  public boolean supportsParameter(ParameterContext parameterContext, 
                                   ExtensionContext extensionContext)
      throws ParameterResolutionException {

     return parameterContext.getParameter().getType() == PactMockServer.class;
  }

  @Override
  @SuppressWarnings("unchecked")
  public Object resolveParameter(ParameterContext parameterContext, 
                                 ExtensionContext extensionContext)
      throws ParameterResolutionException {

      final ExtensionContext.Store store = extensionContext
           .getStore(ExtensionContext.Namespace.create("pact-jvm"));

      if (store.get("providers") == null) {
         return null;
      }

      final List<Pair<ProviderInfo, List<String>>> providers = store
         .get("providers", List.class);
      var pair = providers.get(0);
      final ProviderInfo providerInfo = pair.getFirst();

      var mockServer = store.get("mockServer:" + providerInfo.getProviderName(),
                MockServer.class);

      return new PactMockServer(mockServer.getUrl(), mockServer.getPort());
   }
}

I intentionally do not comment on this workaround. Maybe it could be somehow improved. I wish that everything would work fine just after migrating the Pact extension to Quarkus 3 without any workarounds. However, thanks to the workaround, I was able to run my Pact tests successfully and then update all the required dependencies to the latest.

Final Thoughts

This article guides you through the changes required to migrate your microservices and Pact contract tests from Qaurkus 2 to 3. For me, it is important to automatically update all the dependencies in my demo projects to be up-to-date as described here. I’m using Renovate to automatically scan and update Maven pom.xml dependencies. Once it updates the version of the dependency it runs all the JUnit tests for the verification. The process is automatically performed on the CircleCI. You can view the history of builds of the sample repository used in that article.

The post Pact with Quarkus 3 appeared first on Piotr's TechBlog.

We will do the same thing as described in that article, but fully with the GitOps approach applied by Argo CD. The main goal here is to integrate Postgres with the Vault database secrets engine to generate database credentials dynamically and initialize the DB schema for the sample Spring Boot app. In order to achieve these goals, we are going to install two Kubernetes operators: Atlas and Vault Config. Atlas is a tool for managing the database schema as code. Its Kubernetes Operator allows us to define the schema and apply it to our database using the CRD objects. The Vault Config Operator provided by the Red Hat Community of Practice does a very similar thing but for Hashicorp Vault.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions

How It Works

Before we start, let’s describe our sample scenario. Thanks to the database secrets engine Vault integrates with Postgres and generates its credentials dynamically based on configured roles. On the other hand, our sample Spring Boot app integrates with Vault and uses its database engine to authenticate against Postgres. All the aspects of that scenario are managed in the GitOps style. Argo CD installs Vault, Postgres, and additional operators on Kubernetes via their Helm charts. Then, it applies all the required CRD objects to configure both Vault and Postgres. We keep the whole configuration in a single Git repository in the form of YAML manifests.

Argo CD prepares the configuration on Vault and creates a table on Postgres for the sample Spring Boot app. Our app integrates with Vault through the Spring Cloud Vault project. It also uses Spring Data JPA to interact with the database. Here’s the illustration of our scenario.

Install Argo CD on Kubernetes

Traditionally, we need to start our GitOps exercise by installing Argo CD on the Kubernetes cluster. Of course, we can do it using the Helm chart. In the first step, we need to add the following repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

We will add one parameter to the argo-cm ConfigMap to ignore the MutatingWebhookConfiguration kind. This step is not necessary. It allows us to ignore the specific resource generated by one of the Helm charts used in the further steps. Thanks to that we will have everything in Argo CD in the “green” color Here’s the Helm values.yaml file with the required configuration:

configs:
  cm:
    resource.exclusions: |
      - apiGroups:
        - admissionregistration.k8s.io
        kinds:
        - MutatingWebhookConfiguration
        clusters:
        - "*"

Now, we can install the Argo CD in the argocd namespace using the configuration previously defined in the values.yml file:

$ helm install argo-cd argo/argo-cd \
    --version 6.7.8 \
    -n argo \
    --create-namespace

That’s not all. Since the Atlas operator is available in the OCI-type Helm repository, we need to apply the following Secret in the argocd namespace. By default, Argo CD doesn’t allow the OCI-type repo, so we need to include the enableOCI parameter in the definition.

apiVersion: v1
kind: Secret
metadata:
  name: ghcr-io-helm-oci
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: ariga
  url: ghcr.io/ariga
  enableOCI: "true"
  type: helm

Let’s take a look at the list of repositories in the Argo CD UI dashboard. You should see the “Successful” connection status.

Prepare Configuration Manifests for Argo CD

Config Repository Structure

Let me first explain the structure of our Git config repository. The additional configuration is stored in the apps directory. It includes the CRD objects required to initialize the database schema or Vault engines. In the bootstrap directory, we keep the values.yaml file for each Helm chart managed by Argo CD. It’s all that we need. The bootstrap-via-appset/bootstrap.yaml contains the definition of Argo CD ApplicationSet we need to apply to the Kubernetes cluster. This ApplicationSet will generate all required Argo CD applications responsible for installing the charts and creating CRD objects.

.
├── apps
│   ├── postgresql
│   │   ├── database.yaml
│   │   ├── policies.yaml
│   │   ├── roles.yaml
│   │   └── schema.yaml
│   └── vault
│       └── job.yaml
├── bootstrap
│   ├── values
│   │   ├── atlas
│   │   │   └── values.yaml
│   │   ├── cert-manager
│   │   │   └── values.yaml
│   │   ├── postgresql
│   │   │   └── values.yaml
│   │   ├── vault
│   │   │   └── values.yaml
│   │   └── vault-config-operator
│   │       └── values.yaml
└── bootstrap-via-appset
    └── bootstrap.yaml

Bootstrap with the Argo CD ApplicationSet

Let’s take a look at the ApplicationSet. It’s pretty interesting (I hope :)). I’m using here some relatively new Argo CD features like multiple sources (Argo CD 2.6) or application sets template patch (Argo CD 2.10). We need to generate an Argo CD Application per each tool we want to install on Kubernetes (1). In the generators section, we define parameters for Vault, PostgreSQL, Atlas Operator, Vault Config Operator, and Cert Manager (which is required by the Vault Config Operator). In the templatePatch section, we prepare a list of source repositories used by each Argo CD Application (2). There is always a Helm chart repo, which refers to our Git repository containing dedicated values.yaml files. For the Vault and PostgreSQL charts, we include another source containing CRDs or additional Kubernetes objects. We will discuss it later.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bootstrap-config
  namespace: argocd
spec:
  goTemplate: true
  generators:
  - list:
      elements:
        - chart: vault
          name: vault
          repo: https://helm.releases.hashicorp.com
          revision: 0.27.0
          namespace: vault
          postInstall: true
        - chart: postgresql
          name: postgresql
          repo: https://charts.bitnami.com/bitnami
          revision: 12.12.10
          namespace: default
          postInstall: true
        - chart: cert-manager
          name: cert-manager
          repo: https://charts.jetstack.io
          revision: v1.14.4
          namespace: cert-manager
          postInstall: false
        - chart: vault-config-operator
          name: vault-config-operator
          repo: https://redhat-cop.github.io/vault-config-operator
          revision: v0.8.25
          namespace: vault-config-operator
          postInstall: false
        - chart: charts/atlas-operator
          name: atlas
          repo: ghcr.io/ariga
          revision: 0.4.2
          namespace: atlas
          postInstall: false
  template:
    metadata:
      name: '{{.name}}'
      annotations:
        argocd.argoproj.io/sync-wave: "1"
    spec:
      syncPolicy:
        automated: {}
        syncOptions:
          - CreateNamespace=true
      destination:
        namespace: '{{.namespace}}'
        server: https://kubernetes.default.svc
      project: default
  templatePatch: |
    spec:
      sources:
        - repoURL: '{{ .repo }}'
          chart: '{{ .chart }}'
          targetRevision: '{{ .revision }}'
          helm:
            valueFiles:
              - $values/bootstrap/values/{{ .name }}/values.yaml
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          ref: values
        {{- if .postInstall }}
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          path: apps/{{ .name }}
        {{- end }}