Argo CD ApplicationSet supports several different generators. In this article, we will focus on the Git generator type. It generates Argo CD Applications based on directory structure or branch changes in a Git repository. It contains two subtypes: the Git directory generator and the Git file generator. If you are interested in other Argo CD ApplicationSet generators you can find some articles on my blog. For example, the following post shows how to use List Generator to promote images between environments. You can also find a post about the Cluster Decision Resource generator, which shows how to spread applications dynamically between multiple Kubernetes clusters.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. You must go to the appset-helm-demo directory, which contains the whole configuration required for that exercise. Then you should only follow my instructions.

Argo CD Installation

Argo CD is the only tool we need to install on our Kubernetes cluster for that exercise. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

helm install my-argo-cd argo/argo-cd -n argocd

I use OpenShift in that exercise. With the OpenShift Console, I can easily install ArgoCD on the cluster using the OpenShift GitOps operator.

Once we installed it we can easily access the Argo CD dashboard.

We can sign in there using OpenShift credentials.

Motivation

Our goal in this exercise is to deploy and run some applications (a simple Java app and Postgres database) on Kubernetes with minimal source code effort. Those two applications only show how to create a standard that can be easily applied to any application type deployed on our cluster. In this standard, a directory structure determines how and where our applications are deployed on Kubernetes. My example configuration is stored in a single Git repository. However, we can easily extend it with multiple repositories, where Argoc CD switches between the central repository and other Git repositories containing a configuration for concrete applications.

Here’s a directory structure and files for deploying our two applications. Both the custom app and Postgres database are deployed in three environments: dev, test, and prod. We use Helm charts for deploying them. Each environment directory contains a Helm values file with installation parameters. The configuration distinguishes two different types of installation: apps and components. Each app is installed using the same Helm chart dedicated to a standard deployment. Each component is installed using a custom Helm chart provided by that component. For example, for Postgres, we will use the following Bitnami chart.

.
├── apps
│   ├── aaa-1
│   │   └── basic
│   │       ├── prod
│   │       │   └── values.yaml
│   │       ├── test
│   │       │   └── values.yaml
│   │       ├── uat
│   │       │   └── values.yaml
│   │       └── values.yaml
│   ├── aaa-2
│   └── aaa-3
└── components
    └── aaa-1
        └── postgresql
            ├── prod
            │   ├── config.yaml
            │   └── values.yaml
            ├── test
            │   ├── config.yaml
            │   └── values.yaml
            └── uat
                ├── config.yaml
                └── values.yaml

Before deploying the application, we should prepare namespaces with quotas, Argo CD projects, and ApplicationSet generators for managing application deployments. Here’s the structure of a global configuration repository. It also uses Helm chart to apply that part of manifests to the Kubernetes cluster. Each directory inside the projects directory determines our project name. On the other hand, a project contains several Kubernetes namespaces. Each project may contain several different Kubernetes Deployments.

.
└── projects
    ├── aaa-1
    │   └── values.yaml
    ├── aaa-2
    │   └── values.yaml
    └── aaa-3
        └── values.yaml

Prepare Global Cluster Configuration

Helm Template for Namespaces and Quotas

Here’s the Helm template for creating namespaces and quotas for each namespace. We will create the project namespace per each environment (stage).

{{- range .Values.stages }}
---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ $.Values.projectName }}-{{ .name }}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: {{ $.Values.projectName }}-{{ .name }}
spec:
  hard:
    {{- if .config }}
    {{- with .config.quotas }}
    pods: {{ .pods | default "10" }}
    requests.cpu: {{ .cpuRequest | default "2" }}
    requests.memory: {{ .memoryRequest | default "2Gi" }}
    limits.cpu: {{ .cpuLimit | default "8" }}
    limits.memory: {{ .memoryLimit | default "8Gi" }}
    {{- end }}
    {{- else }}
    pods: "10"
    requests.cpu: "2"
    requests.memory: "2Gi"
    limits.cpu: "8"
    limits.memory: "8Gi"
    {{- end }}
{{- end }}

Helm Template for the Argo CD AppProject

Helm chart will also create a dedicated Argo CD AppProject object per our project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: {{ .Values.projectName }}
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'

Helm Template for Argo CD ApplicationSet

After that, we can proceed to the most tricky part of our exercise. Helm chart also defines a template for creating the Argo CD ApplicationSet. This ApplicationSet must analyze the repository structure, which contains the configuration of apps and components. We define two ApplicationSets per each project. The first uses the Git Directory generator to determine the structure of the apps catalog and deploy the apps in all environments using my custom spring-boot-api-app chart. The chart parameters can be overridden with Helm values placed in each app directory.

The second ApplicationSet uses the Git Files generator to determine the structure of the components catalog. It reads the contents of the config.yaml file in each directory. The config.yaml file sets the repository, name, and version of the Helm chart that must be used to install the component on Kubernetes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-apps-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          {{- range .Values.stages }}
          - path: appset-helm-demo/apps/{{ $.Values.projectName }}/*/{{ .name }}
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: spring-boot-api-app
          repoURL: 'https://piomin.github.io/helm-charts/'
          targetRevision: 0.3.8
          helm:
            valueFiles:
              - $values/appset-helm-demo/apps/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-components-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        files:
          {{- range .Values.stages }}
          - path: appset-helm-demo/components/{{ $.Values.projectName }}/*/{{ .name }}/config.yaml
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: '{{`{{ .chart.name }}`}}'
          repoURL: '{{`{{ .chart.repository }}`}}'
          targetRevision: '{{`{{ .chart.version }}`}}'
          helm:
            valueFiles:
              - $values/appset-helm-demo/components/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

There are several essential elements in this configuration, which we should pay attention to. Both Helm and ApplicationSet use templating engines based on {{ ... }} placeholders. So to avoid conflicts we should escape Argo CD ApplicationSet templating elements from the Helm templating elements. The following part of the template responsible for generating the Argo CD Application name is a good example of that approach: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'. First, we use the AppliocationSet Git generator parameter index .path.segments 3 that returns the name of the third part of the directory path. Those elements are escaped with the ` char so Helm doesn’t try to analyze it.

Helm Chart Structure

Our ApplicationSets use the “Multiple Sources for Application” feature to read parameters from Helm values files and inject them into the Helm chart from a remote repository. Thanks to that, our configuration repositories for apps and components contain only values.yaml files in the standardized directory structure. The only chart we store in the sample repository has been described above and is responsible for creating the configuration required to run app Deployments on the cluster.

.
└── chart
    ├── Chart.yaml
    ├── templates
    │   ├── additional.yaml
    │   ├── applicationsets.yaml
    │   ├── appproject.yaml
    │   └── namespaces.yaml
    └── values.yaml

By default, each project defines three environments (stages): test, uat, prod.

stages:
  - name: test
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

We can override a default behavior for the specific project in Helm values. Each project directory contains the values.yaml file. Here are Helm parameters for the aaa-3 project that override CPU request quota from 2 CPUs to 4 CPUs only for the test environment.

stages:
  - name: test
    config:
      quotas:
        cpuRequest: 4
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

Run the Synchronization Process

Generate Global Structure on the Cluster

To start a process we must create the ApplicationSet that reads the structure of the projects directory. Each subdirectory in the projects directory indicates the name of our project. Our ApplicationSet uses a Git directory generator to create an Argo CD Application per each project. Its name contains the name of the subdirectory and the config suffix. Each generated Application uses the previously described Helm chart to create all namespaces, quotas, and other resources requested by the project. It also leverages the “Multiple Sources for Application” feature to allow us to override default Helm chart settings. It reads a project name from the directory name and passes it as a parameter to the generated Argo CD Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-config
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          - path: appset-helm-demo/projects/*
  template:
    metadata:
      name: '{{.path.basename}}-config'
    spec:
      destination:
        namespace: '{{.path.basename}}'
        server: 'https://kubernetes.default.svc'
      project: default
      sources:
        - path: appset-helm-demo/chart
          repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          helm:
            valueFiles:
              - $values/appset-helm-demo/projects/{{.path.basename}}/values.yaml
            parameters:
              - name: projectName
                value: '{{.path.basename}}'
              - name: argoNamespace
                value: openshift-gitops
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Once we create the global-config ApplicationSet object magic happens. Here’s the list of Argo CD Applications generated from our directories in Git configuration repositories.

First, there are three Argo CD Applications with the projects’ configuration. That’s happening because we defined 3 subdirectories in the projects directory with names aaa-1, aaa-2 and aaa-3.

The configuration applied by those Argo CD Applications is pretty similar since they are using the same Helm chart. We can look at the list of resources managed by the aaa-3-config Application. There are three namespaces (aaa-3-test, aaa-3-uat, aaa-3-prod) with resource quotas, a single Argo CD AppProject, and two ApplicationSet objects responsible for generating Argo CD Applications for apps and components directories.

In this configuration, we can verify if the value of the request.cpu ResourceQuota object has been overridden from 2 CPUs to 4 CPUs.

Let’s analyze what happened. Here’s a list of Argo CD ApplicationSets. The global-config ApplicationSet generated Argo CD Application per each detected project inside the projects directory. Then, each of these Applications applied two ApplicationSet objects to cluster using the Helm template.

$ kubectl get applicationset
NAME                      AGE
aaa-1-components-config   29m
aaa-1-apps-config         29m
aaa-2-components-config   29m
aaa-2-apps-config         29m
aaa-3-components-config   29m
aaa-3-apps-config         29m
global-config             29m

There’s also a list of created namespaces:

$ kubectl get ns
NAME                                               STATUS   AGE
aaa-1-prod                                         Active   34m
aaa-1-test                                         Active   34m
aaa-1-uat                                          Active   34m
aaa-2-prod                                         Active   34m
aaa-2-test                                         Active   34m
aaa-2-uat                                          Active   34m
aaa-3-prod                                         Active   34m
aaa-3-test                                         Active   34m
aaa-3-uat                                          Active   34m

Generate and Apply Deployments

Our sample configuration contains only two Deployments. We defined the basic subdirectory in the apps directory and the postgres subdirectory in the components directory inside the aaa-1 project. The aaa-2 and aaa-3 projects don’t contain any Deployments for simplification. However, the more subdirectories with the values.yaml file we create there, the more applications will be deployed on the cluster. Here’s a typical values.yaml file for a simple app deployed with a standard Helm chart. It defines the image repository, name, and tag. It also set the Deployment name and environment.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: prod

For the postgres component we must set more parameters in Helm values. Here’s the final list:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

The following Argo CD Application has been generated by the aaa-1-apps-config ApplicationSet. It detected the basic subdirectory in the apps directory. The basic subdirectory contained 3 subdirectories: test, uat and prod with values.yaml file. As a result, we have Argo CD per environment responsible for deploying the basic app in the target namespaces.

Here’s a list of resources managed by the basic-prod Application. It uses my custom Helm chart and applies Deployment and Service objects to the cluster.

The following Argo CD Application has been generated by the aaa-1-components-config ApplicationSet. It detected the basic subdirectory in the components directory. The postgres subdirectory contained 3 subdirectories: test, uat and prod with values.yaml and config.yaml files. The ApplicationSet Files generator reads the repository, name, and version from the configuration in the config.yaml file.

Here’s the config.yaml file with the Bitnami Postgres chart settings. We could place here any other chart we want to install something else on the cluster.

chart:
  repository: https://charts.bitnami.com/bitnami
  name: postgresql
  version: 15.5.38

Here’s the list of resources installed by the Bitnami Helm chart used by the generated Argo CD Applications.

Final Thoughts

This article proves that Argo CD ApplicationSet and Helm templates can be used together to create advanced configuration structures. It shows how to use ApplicationSet Git Directory and Files generators to analyze the structure of directories and files in the Git config repository. With that approach, we can propose a standardization in the configuration structure across the whole organization and propagate it similarly for all the applications deployed in the Kubernetes clusters. Everything can be easily managed at the cluster admin level with the single global Argo CD ApplicationSet that accesses many different repositories with configuration.

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

• Argo CD
• Istio
• Apache Kafka
• Postgres
• HashiCorp Vault
• Prometheus
• Redis
• Cert Manager

Finally, we will migrate our sample Spring Boot app. I will also show you how to build such an app on Kubernetes and OpenShift in the same way using the Shipwright tool. However, before we start, let’s discuss some differences between “vanilla” Kubernetes and OpenShift.

Introduction

What are the key differences between Kubernetes and OpenShift? That’s probably the first question you will ask yourself when considering migration from Kubernetes. Today, I will focus only on those aspects that impact running the apps from our list. First of all, OpenShift is built on top of Kubernetes and is fully compatible with Kubernetes APIs and resources. If you can do something on Kubernetes, you can do it on OpenShift in the same way unless it doesn’t compromise security policy. OpenShift comes with additional security policies out of the box. For example, by default, it won’t allow you to run containers with the root user.

Apart from security reasons, only the fact that you can do something doesn’t mean that you should do it in that way. So, you can run images from Docker Hub, but Red Hat provides many supported container images built from Red Hat Enterprise Linux. You can find a full list of supported images here. Although you can install popular software on OpenShift using Helm charts, Red Hat provides various supported Kubernetes operators for that. With those operators, you can be sure that the installation will go without any problems and the solution might be integrated with OpenShift better. We will analyze all those things based on the examples from the tools list.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions.

Install Argo CD

Use Official Helm Chart

In the first step, we will install Argo CD on OpenShift. I’m assuming that on Kubernetes, you’re using the official Helm chart for that. In order to install that chart, we need to add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

Then, we can install the Argo CD in the argocd namespace on OpenShift with the following command. The Argo CD Helm chart provides some parameters dedicated to OpenShift. We need to enable arbitrary uid for the repo server by setting the openshift.enabled property to true. If we want to access the Argo CD dashboard outside of the cluster we should expose it as the Route. In order to do that, we need to enable the server.route.enabled property and set the hostname using the server.route.hostname parameter (piomin.eastus.aroapp.io is my OpenShift domain).

$ helm install argocd argo/argo-cd -n argocd --create-namespace \
    --set openshift.enabled=true \
    --set server.route.enabled=true \
    --set server.route.hostname=argocd.apps.piomin.eastus.aroapp.io

After that, we can access the Argo CD dashboard using the Route address as shown below. The admin user password may be taken from the argocd-initial-admin-secret Secret generated by the Helm chart.

Use the OpenShift GitOps Operator (Recommended Way)

The solution presented in the previous section works fine. However, it is not the optimal approach for OpenShift. In that case, the better idea is to use OpenShift GitOps Operator. Firstly, we should find the “Red Hat GitOps Operator” inside the “Operator Hub” section in the OpenShift Console. Then, we have to install the operator.

During the installation, the operator automatically creates the Argo CD instance in the openshift-gitops namespace.

OpenShift GitOps operator automatically exposes the Argo CD dashboard through the Route. It is also integrated with OpenShift auth, so we can use cluster credentials to sign in there.

Install Redis, Postgres and Apache Kafka

OpenShift Support in Bitnami Helm Charts

Firstly, let’s assume that we use Bitnami Helm charts to install all three tools from the chapter title (Redis, Postgres, Kafka) on Kubernetes. Fortunately, the latest versions of Bitnami Helm charts provide out-of-the-box compatibility with the OpenShift platform. Let’s analyze what it means.

Beginning from the 4.11 version OpenShift introduces new Security Context Constraints (SCC) called restricted-v2. In OpenShift, security context constraints allow us to control permissions assigned to the pods. The restricted-v2 SCC includes a minimal set of privileges usually required for a generic workload to run. It is the most restrictive policy that matches the current pod security standards. As I mentioned before, the latest version of the most popular Bitnami Helm charts supports the restricted-v2 SCC. We can check which of the charts support that feature by checking if they provide the global.compatibility.openshift.adaptSecurityContext parameter. The default value of that parameter is auto. It means that it is applied only if the detected running cluster is Openshift.

So, in short, we don’t have to change anything in the Helm chart configuration used on Kubernetes to make it work also on OpenShift. However, it doesn’t mean that we won’t change that configuration. Let’s analyze it tool after tool.

Install Redis on OpenShift with Helm Chart

In the first step, let’s add the Bitnami Helm repository with the following command:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Then, we can install and run a three-node Redis cluster with a single master node in the redis namespace using the following command:

$ helm install redis bitnami/redis -n redis --create-namespace

After installing the chart we can display a list of pods running the redis namespace:

$ oc get po
NAME               READY   STATUS    RESTARTS   AGE
redis-master-0     1/1     Running   0          5m31s
redis-replicas-0   1/1     Running   0          5m31s
redis-replicas-1   1/1     Running   0          4m44s
redis-replicas-2   1/1     Running   0          4m3s

Let’s take a look at the securityContext section inside one of the Redis cluster pods. It contains characteristic fields for the restricted-v2 SCC, which removes runAsUser, runAsGroup and fsGroup and let the platform use their allowed default IDs.

However, let’s stop for a moment to analyze the current situation. We installed Redis on OpenShift using the Bitnami Helm chart. By default, this chart is based on the Redis Debian image provided by Bitnami in the Docker Hub.

On the other hand, Red Hat provides its build of Redis image based on RHEL 9. Consequently, this image would be more suitable for running on OpenShift.

In order to use a different Redis image with the Bitnami Helm chart, we need to override the registry, repository, and tag fields in the image section. The full address of the current latest Red Hat Redis image is registry.redhat.io/rhel9/redis-7:1-16. In order to make the Bitnami chart work with that image, we need to override the default data path to /var/lib/redis/data and disable the container’s Security Context read-only root filesystem for the slave pods.

image:
  tag: 1-16
  registry: registry.redhat.io
  repository: rhel9/redis-7

master:
  persistence:
    path: /var/lib/redis/data

replica:
  persistence:
    path: /var/lib/redis/data
  containerSecurityContext:
    readOnlyRootFilesystem: false

Install Postgres on OpenShift with Helm Chart

With Postgres, we have every similar as before with Redis. The Bitnami Helm chart also supports OpenShift restricted-v2 SCC and Red Hat provide the Postgres image based on RHEL 9. Once again, we need to override some chart parameters to adapt to a different image than the default one provided by Bitnami.

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

Of course, we can consider switching to one of the available Postgres operators. From the “Operator Hub” section we can install e.g. Postgres using Crunchy or EDB operators. However, these are not operators provided by Red Hat. Of course, you can use them on “vanilla” Kubernetes as well. In that case, the migration to OpenShift also won’t be complicated.

Install Kafka on OpenShift with the Strimzi Operator

The situation is slightly different in the case of Apache Kafka. Of course, we can use the Kafka Helm chart provided by Bitnami. However, Red Hat provides a supported version of Kafka through the Strimzi operator. This operator is a part of the Red Hat product ecosystem and is available commercially as the AMQ Streams. In order to install Kafka with AMQ Streams on OpenShift, we need to install the operator first.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-streams
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: amq-streams
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Once we install the operator with the Strimzi CRDs we can provision the Kafka instance on OpenShift. In order to do that, we need to define the Kafka object. The name of the cluster is my-cluster. We should install it after a successful installation of the operator CRD, so we set the higher value of the Argo CD sync-wave parameter than for the amq-streams Subscription object. Argo CD should also ignore missing CRDs installed by the operator during sync thanks to the SkipDryRunOnMissingResource option.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.6'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    version: 3.6.0
    replicas: 3
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      deleteClaim: true
      size: 2Gi
    replicas: 3

GitOps Strategy for Kubernetes and OpenShift

In this section, we will focus on comparing differences in the GitOps manifest between Kubernetes and Openshift. We will use Kustomize to configure two overlays: openshift and kubernetes. Here’s the structure of our configuration repository:

.
├── base
│   ├── kustomization.yaml
│   └── namespaces.yaml
└── overlays
    ├── kubernetes
    │   ├── kustomization.yaml
    │   ├── namespaces.yaml
    │   ├── values-cert-manager.yaml
    │   └── values-vault.yaml
    └── openshift
        ├── cert-manager-operator.yaml
        ├── kafka-operator.yaml
        ├── kustomization.yaml
        ├── service-mesh-operator.yaml
        ├── values-postgres.yaml
        ├── values-redis.yaml
        └── values-vault.yaml

Configuration for Kubernetes

In addition to the previously discussed tools, we will also install “cert-manager”, Prometheus, and Vault using Helm charts. Kustomize allows us to define a list of managed charts using the helmCharts section. Here’s the kustomization.yaml file containing a full set of installed charts:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - namespaces.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
  - name: kafka
    repo: https://charts.bitnami.com/bitnami
    releaseName: kafka
    namespace: kafka
  - name: cert-manager
    repo: https://charts.jetstack.io
    releaseName: cert-manager
    namespace: cert-manager
    valuesFile: values-cert-manager.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml
  - name: prometheus
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: prometheus
    namespace: prometheus
  - name: istio
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: istio
    namespace: istio-system

For some of them, we need to override default Helm parameters. Here’s the values-vault.yaml file with the parameters for Vault. We enable development mode and UI dashboard:

server:
  dev:
    enabled: true
ui:
  enabled: true

Let’s also customize the default behavior of the “cert-manager” chart with the following values:

installCRDs: true
startupapicheck:
  enabled: false

Configuration for OpenShift

Then, we can switch to the configuration for Openshift. Vault has to be installed with the Helm chart, but for “cert-manager” we can use the operator provided by Red Hat. Since Openshift comes with built-in Prometheus, we don’t need to install it. We will also replace the Helm chart with Istio with the Red Hat-supported OpenShift Service Mesh operator. Here’s the kustomization.yaml for OpenShift:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - kafka-operator.yaml
  - cert-manager-operator.yaml
  - service-mesh-operator.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
    valuesFile: values-redis.yaml
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
    valuesFile: values-postgres.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml

For Vault we should enable integration with Openshift and support for the Route object. Red Hat provides a Vault image based on UBI in the registry.connect.redhat.com/hashicorp/vault registry. Here’s the values-vault.yaml file for OpenShift:

server:
  dev:
    enabled: true
  route:
    enabled: true
    host: ""
    tls: null
  image:
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.16.1-ubi"
global:
  openshift: true
injector:
  enabled: false

In order to install operators we need to define at least the Subscription object. Here’s the subscription for the OpenShift Service Mesh. After installing the operator we can create a control plane in the istio-system namespace using the ServiceMeshControlPlane CRD object. In order to apply the CRD after installing the operator, we need to use the Argo CD sync waves and define the SkipDryRunOnMissingResource parameter:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: servicemeshoperator
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: servicemeshoperator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
  name: basic
  namespace: istio-system
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  tracing:
    type: None
    sampling: 10000
  policy:
    type: Istiod
  addons:
    grafana:
      enabled: false
    jaeger:
      install:
        storage:
          type: Memory
    kiali:
      enabled: false
    prometheus:
      enabled: false
  telemetry:
    type: Istiod
  version: v2.5

Since the “cert-manager” operator is installed in a different namespace than openshift-operators, we also need to define the OperatorGroup object.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  targetNamespaces:
    - cert-manager

Finally, OpenShift comes with built-in Prometheus monitoring, so we don’t need to install it.

Apply the Configuration with Argo CD

Here’s the Argo CD Application responsible for installing our sample configuration on OpenShift. We should create it in the openshift-gitops namespace.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: install
  namespace: openshift-gitops
spec:
  destination:
    server: 'https://kubernetes.default.svc'
  project: default
  source:
    path: overlays/openshift
    repoURL: 'https://github.com/piomin/kubernetes-to-openshift-argocd.git'
    targetRevision: HEAD

Before that, we need to enable the use of the Helm chart inflator generator with Kustomize in Argo CD. In order to do that, we can add the kustomizeBuildOptions parameter in the openshift-gitops ArgoCD object as shown below.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  # ...
  kustomizeBuildOptions: '--enable-helm'

After creating the Argo CD Application and triggering the sync process, the installation starts on OpenShift.

Build App Images

We installed several software solutions including the most popular databases, message brokers, and security tools. However, now we want to build and run our own apps. How to migrate them from Kubernetes to OpenShift? Of course, we can run the app images exactly in the same way as in Kubernetes. On the other hand, we can build them on OpenShift using the Shipwright project. We can install it on OpenShift using the “Builds for Red Hat OpenShift Operator”.

After that, we need to create the ShiwrightBuild object. It needs to contain the name of the target namespace for running Shipwright in the targetNamespace field. In my case, the target namespace is builds-demo. For a detailed description of the Shipwright build, you can refer to that article on my blog.

apiVersion: operator.shipwright.io/v1alpha1
kind: ShipwrightBuild
metadata:
  name: openshift-builds
spec:
  targetNamespace: builds-demo

With Shipwright we can easily switch between multiple build strategies on Kubernetes, and on OpenShift as well. For example, on OpenShift we can use a built-in source-to-image (S2I) strategy, while on Kubernetes e.g. Kaniko or Cloud Native Buildpacks.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: sample-spring-kotlin-build
  namespace: builds-demo
spec:
  output:
    image: quay.io/pminkows/sample-kotlin-spring:1.0-shipwright
    pushSecret: pminkows-piomin-pull-secret
  source:
    git:
      url: https://github.com/piomin/sample-spring-kotlin-microservice.git
  strategy:
    name: source-to-image
    kind: ClusterBuildStrategy

Final Thoughts

Migration from Kubernetes to Openshift is not a painful process. Many popular Helm charts support OpenShift restricted-v2 SCC. Thanks to that, in some cases, you don’t need to change anything. However, sometimes it’s worth switching to the version of the particular tool supported by Red Hat.

The post Migrate from Kubernetes to OpenShift in the GitOps Way appeared first on Piotr's TechBlog.

You can find many other articles about Apache Kafka on my blog. For example, to read about concurrency with Spring Kafka please refer to the following post. There is also an article about Kafka transactions available here.

Prerequisites

In order to proceed with the exercise, you need to have a Kubernetes cluster. This cluster should have at least three worker nodes since I’m going to show you the approach with Kafka brokers spread across several nodes. We can easily simulate multiple Kubernetes nodes locally with Kind. You need to install the kind CLI tool and start Docker on your laptop. Here’s the Kind configuration manifest containing a definition of a single control plane and 4 worker nodes:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
- role: worker
- role: worker
- role: worker

Then, we need to create the Kubernetes cluster based on the manifest visible above with the following kind command:

$ kind create cluster --name c1 --config cluster.yaml

The name of our Kind cluster is c1. It corresponds to the kind-c1 Kubernetes context, which is automatically set as default after creating the cluster. After that, we can display a list of Kubernetes nodes using the following kubectl command:

$ kubectl get node
NAME               STATUS   ROLES           AGE  VERSION
c1-control-plane   Ready    control-plane   1m   v1.27.3
c1-worker          Ready    <none>          1m   v1.27.3
c1-worker2         Ready    <none>          1m   v1.27.3
c1-worker3         Ready    <none>          1m   v1.27.3
c1-worker4         Ready    <none>          1m   v1.27.3

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. After that, go to the kafka directory. There are two Spring Boot apps inside the producer and consumer directories. The required Kubernetes manifests are available inside the k8s directory. You can apply them with kubectl or using the Skaffold CLI tool. The repository is already configured to work with Skaffold and Kind. To proceed with the exercise just follow my instructions in the next sections.

Architecture

Let’s analyze our main goals in this exercise. Of course, we want to run a Kafka cluster on Kubernetes as simple as possible. There are several requirements for the cluster:

1. It should automatically expose broker metrics in the Prometheus format. Then we will use Prometheus mechanisms to get the metrics and store them for visualization.
2. It should consist of at least 3 brokers. Each broker has to run on a different Kubernetes worker node.
3. Our Kafka needs to work in the Zookeeper-less mode. Therefore, we need to enable the KRaft protocol between the brokers.
4. Once we scale up the Kafka cluster, we must automatically rebalance it to reassign partition replicas to the new broker. In order to do that, we will use the Cruise Control support in Strimzi.

Here’s the diagram that visualizes the described architecture. We will also run two simple Spring Boot apps on Kubernetes that connect the Kafka cluster and use it to send/receive messages.

1. Install Monitoring Stack on Kubernetes

In the first step, we will install the monitoring on our Kubernetes cluster. We are going to use the kube-prometheus-stack Helm chart for that. It provides preconfigured instances of Prometheus and Grafana. It also comes with several CRD objects that allow us to easily customize monitoring mechanisms according to our needs. Let’s add the following Helm repository:

$ helm repo add prometheus-community \
    https://prometheus-community.github.io/helm-charts

Then, we can install the chart in the monitoring namespace. We can leave the default configuration.

$ helm install kube-prometheus-stack \
    prometheus-community/kube-prometheus-stack \
    --version 52.1.0 -n monitoring --create-namespace

2. Install Strimzi Operator on Kubernetes

In the next step, we will install the Strimzi operator on Kubernetes using Helm chart. The same as before we need to add the Helm repository:

$ helm repo add strimzi https://strimzi.io/charts

Then, we can proceed to the installation. This time we will override some configuration settings. The Strimzi Helm chart comes with a set of Grafana dashboards to visualize metrics exported by Kafka brokers and some other components managed by Strimzi. We place those dashboards inside the monitoring namespace. By default, the Strimzi chart doesn’t add the dashboards, so we also need to enable that feature in the values YAML file. That’s not all. Because we want to run Kafka in the KRaft mode, we need to enable it using feature gates. Enabling the UseKRaft feature gate requires the KafkaNodePools feature gate to be enabled as well. Then when we deploy a Kafka cluster in KRaft mode, we also must use the KafkaNodePool resources. Here’s the full list of overridden Helm chart values:

dashboards:
  enabled: true
  namespace: monitoring
featureGates: +UseKRaft,+KafkaNodePools,+UnidirectionalTopicOperator

Finally, let’s install the operator in the strimzi namespace using the following command:

$ helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator \
    --version 0.38.0 \
    -n strimzi --create-namespace \
    -f strimzi-values.yaml

3. Run Kafka in the KRaft Mode

In the current version of Strimzi KRaft mode support is still in the alpha phase. This will probably change soon but for now, we have to deal with some inconveniences. In the previous section, we enabled three feature gates required to run Kafka in KRaft mode. Thanks to that we can finally define our Kafka cluster. In the first step, we need to create a node pool. This new Strimzi object is responsible for configuring brokers and controllers in the cluster. Controllers are responsible for coordinating operations and maintaining the cluster’s state. Fortunately, a single node in the poll can act as a controller and a broker at the same time.

Let’s create the KafkaNodePool object for our cluster. As you see it defines two roles: broker and controller (1). We can also configure storage for the cluster members (2). One of our goals is to avoid sharing the same Kubernetes node between Kafka brokers. Therefore, we will define the podAntiAffinity section (3). Setting the topologyKey to kubernetes.io/hostname indicates that the selected pods are not scheduled on nodes with the same hostname (4).

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaNodePool
metadata:
  name: dual-role
  namespace: strimzi
  labels:
    strimzi.io/cluster: my-cluster
spec:
  replicas: 3
  roles: # (1)
    - controller
    - broker
  storage: # (2)
    type: jbod
    volumes:
      - id: 0
        type: persistent-claim
        size: 20Gi
        deleteClaim: false
  template:
    pod:
      affinity:
        podAntiAffinity: # (3)
          requiredDuringSchedulingIgnoredDuringExecution:
            - labelSelector:
                matchExpressions:
                  - key: strimzi.io/name
                    operator: In
                    values:
                      - my-cluster-kafka
              topologyKey: "kubernetes.io/hostname" # (4)

Once we create a node pool, we can proceed to the Kafka object creation. We need to enable Kraft mode and node pools for the particular cluster by annotating it with strimzi.io/kraft and strimzi.io/node-pools (1). The sections like storage (2) or zookeeper (5) are not used in the KRaft mode but are still required by the CRD. We should also configure the cluster metrics exporter (3) and enable the Cruise Control component (4). Of course, our cluster is exposing API for the client connection under the 9092 port.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: strimzi
  annotations: # (1)
    strimzi.io/node-pools: enabled
    strimzi.io/kraft: enabled
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.6'
    storage: # (2)
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    version: 3.6.0
    replicas: 3
    metricsConfig: # (3)
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml
  entityOperator:
    topicOperator: {}
    userOperator: {}
  cruiseControl: {} # (4)
  # (5)
  zookeeper:
    storage:
      type: persistent-claim
      deleteClaim: true
      size: 2Gi
    replicas: 3

The metricsConfig section in the Kafka object took the ConfigMap as the configuration source. This ConfigMap contains a single kafka-metrics-config.yml entry with the Prometheus rules definition.

kind: ConfigMap
apiVersion: v1
metadata:
  name: kafka-metrics
  namespace: strimzi
  labels:
    app: strimzi
data:
  kafka-metrics-config.yml: |
    lowercaseOutputName: true
    rules:
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        topic: "$4"
        partition: "$5"
    - pattern: kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value
      name: kafka_server_$1_$2
      type: GAUGE
      labels:
        clientId: "$3"
        broker: "$4:$5"
    - pattern: kafka.server<type=(.+), cipher=(.+), protocol=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_tls_info
      type: GAUGE
      labels:
        cipher: "$2"
        protocol: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: kafka.server<type=(.+), clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections
      name: kafka_server_$1_connections_software
      type: GAUGE
      labels:
        clientSoftwareName: "$2"
        clientSoftwareVersion: "$3"
        listener: "$4"
        networkProcessor: "$5"
    - pattern: "kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+):"
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    - pattern: kafka.server<type=(.+), listener=(.+), networkProcessor=(.+)><>(.+)
      name: kafka_server_$1_$4
      type: GAUGE
      labels:
        listener: "$2"
        networkProcessor: "$3"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>MeanRate
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+)Percent\w*, (.+)=(.+)><>Value
      name: kafka_$1_$2_$3_percent
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*, (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)PerSec\w*><>Count
      name: kafka_$1_$2_$3_total
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Value
      name: kafka_$1_$2_$3
      type: GAUGE
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
        "$6": "$7"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        "$6": "$7"
        quantile: "0.$8"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
      labels:
        "$4": "$5"
    - pattern: kafka.(\w+)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        "$4": "$5"
        quantile: "0.$6"
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>Count
      name: kafka_$1_$2_$3_count
      type: COUNTER
    - pattern: kafka.(\w+)<type=(.+), name=(.+)><>(\d+)thPercentile
      name: kafka_$1_$2_$3
      type: GAUGE
      labels:
        quantile: "0.$4"
    - pattern: "kafka.server<type=raft-metrics><>(.+-total|.+-max):"
      name: kafka_server_raftmetrics_$1
      type: COUNTER
    - pattern: "kafka.server<type=raft-metrics><>(.+):"
      name: kafka_server_raftmetrics_$1
      type: GAUGE
    - pattern: "kafka.server<type=raft-channel-metrics><>(.+-total|.+-max):"
      name: kafka_server_raftchannelmetrics_$1
      type: COUNTER
    - pattern: "kafka.server<type=raft-channel-metrics><>(.+):"
      name: kafka_server_raftchannelmetrics_$1
      type: GAUGE
    - pattern: "kafka.server<type=broker-metadata-metrics><>(.+):"
      name: kafka_server_brokermetadatametrics_$1
      type: GAUGE

4. Interacting with Kafka on Kubernetes

Once we apply the KafkaNodePool and Kafka objects to the Kubernetes cluster, Strimzi starts provisioning. As a result, you should see the broker pods, a single pod related to Cruise Control, and a metrics exporter pod. Each Kafka broker pod is running on a different Kubernetes node:

Clients can connect Kafka using the my-cluster-kafka-bootstrap Service under the 9092 port:

$ kubectl get svc
NAME                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                        AGE
my-cluster-cruise-control    ClusterIP   10.96.108.204   <none>        9090/TCP                                       4m10s
my-cluster-kafka-bootstrap   ClusterIP   10.96.155.136   <none>        9091/TCP,9092/TCP,9093/TCP                     4m59s
my-cluster-kafka-brokers     ClusterIP   None            <none>        9090/TCP,9091/TCP,8443/TCP,9092/TCP,9093/TCP   4m59s

In the next step, we will deploy our two apps for producing and consuming messages. The producer app sends one message per second to the target topic:

@SpringBootApplication
@EnableScheduling
public class KafkaProducer {

   private static final Logger LOG = LoggerFactory
      .getLogger(KafkaProducer.class);

   public static void main(String[] args) {
      SpringApplication.run(KafkaProducer.class, args);
   }

   AtomicLong id = new AtomicLong();
   @Autowired
   KafkaTemplate<Long, Info> template;

   @Value("${POD:kafka-producer}")
   private String pod;
   @Value("${NAMESPACE:empty}")
   private String namespace;
   @Value("${CLUSTER:localhost}")
   private String cluster;
   @Value("${TOPIC:test}")
   private String topic;

   @Scheduled(fixedRate = 1000)
   public void send() {
      Info info = new Info(id.incrementAndGet(), 
                           pod, namespace, cluster, "HELLO");
      CompletableFuture<SendResult<Long, Info>> result = template
         .send(topic, info.getId(), info);
      result.whenComplete((sr, ex) ->
         LOG.info("Sent({}): {}", sr.getProducerRecord().key(), 
         sr.getProducerRecord().value()));
   }
}

Here’s the Deployment manifest for the producer app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: producer
spec:
  selector:
    matchLabels:
      app: producer
  template:
    metadata:
      labels:
        app: producer
    spec:
      containers:
      - name: producer
        image: piomin/producer
        resources:
          requests:
            memory: 200Mi
            cpu: 100m
        ports:
        - containerPort: 8080
        env:
          - name: KAFKA_URL
            value: my-cluster-kafka-bootstrap
          - name: CLUSTER
            value: c1
          - name: TOPIC
            value: test-1
          - name: POD
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

Before running the app we can create the test-1 topic with the Strimzi CRD:

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: test-1
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 12
  replicas: 3
  config:
    retention.ms: 7200000
    segment.bytes: 1000000

The consumer app is listening for incoming messages. Here’s the bean responsible for receiving and logging messages:

@SpringBootApplication
@EnableKafka
public class KafkaConsumer {

   private static final Logger LOG = LoggerFactory
      .getLogger(KafkaConsumer.class);

   public static void main(String[] args) {
      SpringApplication.run(KafkaConsumer.class, args);
   }

   @Value("${app.in.topic}")
   private String topic;

   @KafkaListener(id = "info", topics = "${app.in.topic}")
   public void onMessage(@Payload Info info,
      @Header(name = KafkaHeaders.RECEIVED_KEY, required = false) Long key,
      @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
      LOG.info("Received(key={}, partition={}): {}", key, partition, info);
   }
}

Here’s the Deployment manifest for the consumer app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer
spec:
  selector:
    matchLabels:
      app: consumer
  template:
    metadata:
      labels:
        app: consumer
    spec:
      containers:
      - name: consumer
        image: piomin/consumer
        resources:
          requests:
            memory: 200Mi
            cpu: 100m
        ports:
        - containerPort: 8080
        env:
          - name: TOPIC
            value: test-1
          - name: KAFKA_URL
            value: my-cluster-kafka-bootstrap

We can run both Spring Boot apps using Skaffold. Firstly, we need to go to the kafka directory in our repository. Then let’s run the following command:

$ skaffold run -n strimzi --tail

Finally, we can verify the logs printed by our apps. As you see, all the messages sent by the producer app are received by the consumer app.

5. Kafka Metrics in Prometheus

Once we installed the Strimzi Helm chart with the dashboard.enabled=true and dashboard.namespace=monitoring, we have several Grafana dashboard manifests placed in the monitoring namespace. Each dashboard is represented as a ConfigMap. Let’s display a list of ConfigMaps installed by the Strimzi Helm chart:

$ kubectl get cm -n monitoring | grep strimzi
strimzi-cruise-control                                    1      2m
strimzi-kafka                                             1      2m
strimzi-kafka-bridge                                      1      2m
strimzi-kafka-connect                                     1      2m
strimzi-kafka-exporter                                    1      2m
strimzi-kafka-mirror-maker-2                              1      2m
strimzi-kafka-oauth                                       1      2m
strimzi-kraft                                             1      2m
strimzi-operators                                         1      2m
strimzi-zookeeper                                         1      2m

Since Grafana is also installed in the monitoring namespace, it automatically imports all the dashboards from ConfigMaps annotated with grafana_dashboard. Consequently, after logging into Grafana (admin / prom-operator), we can easily switch between all the Kafka-related dashboards.

The only problem is that Prometheus doesn’t scrape the metrics exposed by the Kafka pods. Since we have already configured metrics exporting on the Strimzi Kafka CRD, Kafka pods expose the /metric endpoint for Prometheus under the 9404 port. Let’s take a look at the Kafka broker pod details:

In order to force Prometheus to scrape metrics from Kafka pods, we need to create the PodMonitor object. We should place it in the monitoring (1) namespace and set the release=kube-prometheus-stack label (2). The PodMonitor object filters all the pods from the strimzi namespace (3) that contains the strimzi.io/kind label having one of the values: Kafka, KafkaConnect, KafkaMirrorMaker, KafkaMirrorMaker2 (4). Also, it has to query the /metrics endpoint under the port with the tcp-prometheus name (5).

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: kafka-resources-metrics
  namespace: monitoring
  labels:
    app: strimzi
    release: kube-prometheus-stack
spec:
  selector:
    matchExpressions:
      - key: "strimzi.io/kind"
        operator: In
        values: ["Kafka", "KafkaConnect", "KafkaMirrorMaker", "KafkaMirrorMaker2"]
  namespaceSelector:
    matchNames:
      - strimzi
  podMetricsEndpoints:
  - path: /metrics
    port: tcp-prometheus
    relabelings:
    - separator: ;
      regex: __meta_kubernetes_pod_label_(strimzi_io_.+)
      replacement: $1
      action: labelmap
    - sourceLabels: [__meta_kubernetes_namespace]
      separator: ;
      regex: (.*)
      targetLabel: namespace
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_name]
      separator: ;
      regex: (.*)
      targetLabel: kubernetes_pod_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_node_name]
      separator: ;
      regex: (.*)
      targetLabel: node_name
      replacement: $1
      action: replace
    - sourceLabels: [__meta_kubernetes_pod_host_ip]
      separator: ;
      regex: (.*)
      targetLabel: node_ip
      replacement: $1
      action: replace

Finally, we can display the Grafana dashboard with Kafka metrics visualization. Let’s choose the dashboard with the “Strimzi Kafka” name. Here’s the general view:

There are several other diagrams available. For example, we can take a look at the statistics related to the incoming and outgoing messages.

6. Rebalancing Kafka with Cruise Control

Let’s analyze the typical scenario around Kafka related to increasing the number of brokers in the cluster. Before we do it, we will generate more incoming traffic to the test-1 topic. In order to do it, we can use the Grafana k6 tool. The k6 tool provides several extensions for load testing – including the Kafka plugin. Here’s the Deployment manifest that runs k6 with the Kafka extension on Kubernetes.

kind: ConfigMap
apiVersion: v1
metadata:
  name: load-test-cm
  namespace: strimzi
data:
  load-test.js: |
    import {
      Writer,
      SchemaRegistry,
      SCHEMA_TYPE_JSON,
    } from "k6/x/kafka";
    const writer = new Writer({
      brokers: ["my-cluster-kafka-bootstrap.strimzi:9092"],
      topic: "test-1",
    });
    const schemaRegistry = new SchemaRegistry();
    export default function () {
      writer.produce({
        messages: [
          {
            value: schemaRegistry.serialize({
              data: {
                id: 1,
                source: "test",
                space: "strimzi",
                cluster: "c1",
                message: "HELLO"
              },
              schemaType: SCHEMA_TYPE_JSON,
            }),
          },
        ],
      });
    }
    
    export function teardown(data) {
      writer.close();
    }
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: k6-test
  namespace: strimzi
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: k6-test
  template:
    metadata:
      labels:
        app.kubernetes.io/name: k6-test
    spec:
      containers:
        - image: mostafamoradian/xk6-kafka:latest
          name: xk6-kafka
          command:
            - "k6"
            - "run"
            - "--vus"
            - "1"
            - "--duration"
            - "720s"
            - "/tests/load-test.js"
          env:
            - name: KAFKA_URL
              value: my-cluster-kafka-bootstrap
            - name: CLUSTER
              value: c1
            - name: TOPIC
              value: test-1
            - name: POD
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          volumeMounts:
            - mountPath: /tests
              name: test
      volumes:
        - name: test
          configMap:
            name: load-test-cm

Let’s apply the manifest to the strimzi namespace with the following command:

$ kubectl apply -f k8s/k6.yaml

After that, we can take a look at the k6 Pod logs. As you see, it generates and sends a lot of messages to the test-1 topic on our Kafka cluster:

Now, let’s increase the number of Kafka brokers in our cluster. We can do it by changing the value of the replicas field in the KafkaNodePool object:

$ kubectl scale kafkanodepool dual-role --replicas=4 -n strimzi

After a while, Strimzi will start a new pod with another Kafka broker. Although we have a new member of the Kafka cluster, all the partitions are still distributed only across three previous brokers. The situation would be different for the new topic. However, the partitions related to the existing topics won’t be automatically migrated to the new broker instance. Let’s verify the current partition structure for the test-1 topic with kcat CLI (I’m exposing Kafka API locally with kubectl port-forward):

$ kcat -b localhost:9092 -L -t test-1
Metadata for test-1 (from broker -1: localhost:9092/bootstrap):
 4 brokers:
  broker 0 at my-cluster-dual-role-0.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 1 at my-cluster-dual-role-1.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 2 at my-cluster-dual-role-2.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 3 at my-cluster-dual-role-3.my-cluster-kafka-brokers.strimzi.svc:9092 (controller)
 1 topics:
  topic "test-1" with 12 partitions:
    partition 0, leader 0, replicas: 0,1,2, isrs: 1,0,2
    partition 1, leader 1, replicas: 1,2,0, isrs: 1,0,2
    partition 2, leader 2, replicas: 2,0,1, isrs: 1,0,2
    partition 3, leader 0, replicas: 0,1,2, isrs: 1,0,2
    partition 4, leader 1, replicas: 1,2,0, isrs: 1,0,2
    partition 5, leader 2, replicas: 2,0,1, isrs: 1,0,2
    partition 6, leader 0, replicas: 0,1,2, isrs: 1,0,2
    partition 7, leader 1, replicas: 1,2,0, isrs: 1,0,2
    partition 8, leader 2, replicas: 2,0,1, isrs: 1,0,2
    partition 9, leader 0, replicas: 0,2,1, isrs: 1,0,2
    partition 10, leader 2, replicas: 2,1,0, isrs: 1,0,2
    partition 11, leader 1, replicas: 1,0,2, isrs: 1,0,2

Here comes Cruise Control. Cruise Control makes managing and operating Kafka much easier. For example, it allows us to move partitions across brokers after scaling up the cluster. Let’s see how it works. We have already enabled Cruise Control in the Strimzi Kafka CRD. In order to begin a rebalancing procedure, we should create the KafkaRebalance object. This object is responsible for asking Cruise Control to generate an optimization proposal.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaRebalance
metadata:
  name: my-rebalance
  labels:
    strimzi.io/cluster: my-cluster
spec: {}

If the optimization proposal is ready you will see the ProposalReady value in the Status.Conditions.Type field. I won’t get into the details of Cruise Control. It suggested moving 58 partition replicas between separate brokers in the cluster.

Let’s accept the proposal by annotating the KafkaRebalance object with strimzi.io/rebalance=approve:

$ kubectl annotate kafkarebalance my-rebalance \   
    strimzi.io/rebalance=approve -n strimzi

Finally, we can run the kcat command on the test-1 topic once again. Now, as you see, partition replicas are spread across all the brokers.

$ kcat -b localhost:9092 -L -t test-1
Metadata for test-1 (from broker -1: localhost:9092/bootstrap):
 4 brokers:
  broker 0 at my-cluster-dual-role-0.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 1 at my-cluster-dual-role-1.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 2 at my-cluster-dual-role-2.my-cluster-kafka-brokers.strimzi.svc:9092
  broker 3 at my-cluster-dual-role-3.my-cluster-kafka-brokers.strimzi.svc:9092 (controller)
 1 topics:
  topic "test-1" with 12 partitions:
    partition 0, leader 2, replicas: 2,1,3, isrs: 1,2,3
    partition 1, leader 1, replicas: 1,2,0, isrs: 1,0,2
    partition 2, leader 2, replicas: 0,2,1, isrs: 1,0,2
    partition 3, leader 0, replicas: 0,2,3, isrs: 0,2,3
    partition 4, leader 1, replicas: 3,2,1, isrs: 1,2,3
    partition 5, leader 2, replicas: 2,3,0, isrs: 0,2,3
    partition 6, leader 0, replicas: 0,1,2, isrs: 1,0,2
    partition 7, leader 1, replicas: 3,1,0, isrs: 1,0,3
    partition 8, leader 2, replicas: 2,0,1, isrs: 1,0,2
    partition 9, leader 0, replicas: 0,3,1, isrs: 1,0,3
    partition 10, leader 2, replicas: 2,3,0, isrs: 0,2,3
    partition 11, leader 1, replicas: 1,0,3, isrs: 1,0,3

Final Thoughts

Strimzi allows us not only to install and manage Kafka but also the whole ecosystem around it. In this article, I showed how to export metrics to Prometheus and use the Cruise Control tool to rebalance a cluster after scale-up. We also ran Kafka in KRaft mode and then connected two simple Java apps with the cluster through Kubernetes Service.

The post Apache Kafka on Kubernetes with Strimzi appeared first on Piotr's TechBlog.

Let me clarify some things before we begin. In this article, I’m not promoting or recommending Terraform as the best tool for managing OpenShift or Kubernetes clusters at scale. Usually, I prefer the GitOps approach for that. If you are interested in how to leverage such tools like ACM (Advanced Cluster Management for Kubernetes) and Argo CD for managing multiple clusters with the GitOps approach read that article. It describes the idea of a cluster continuous management. From my perspective, Terraform fits better for one-time actions, like for example, creating and configuring OpenShift for the demo or PoC and then removing it. We can also use Terraform to install Argo CD and then delegate all the next steps there.

Anyway, let’s focus on our scenario. We will widely use those two Terraform providers: Azure and Kubernetes. So, it is worth at least taking a look at the documentation to familiarize yourself with the basics.

Prerequisites

Of course, you don’t have to perform that exercise on Azure with ARO. If you already have OpenShift running you can skip the part related to the cluster creation and just run the Terraform script responsible for installing operators and apps. For the whole exercise, you need to install:

1. Azure CLI (instructions) – once you install the login to your Azure account and create the subscription. To check if all works run the following command: az account show
2. Terraform CLI (instructions) – once you install the Terraform CLI you can verify it with the following command: terraform version

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then you should follow my instructions

Terraform Providers

The Terraform scripts for cluster creation are available inside the aro directory, while the script for cluster configuration inside the servicemesh directory. Here’s the structure of our repository:

Firstly, let’s take a look at the list of Terraform providers used in our exercise. In general, we need providers to interact with Azure and OpenShift through the Kubernetes API. In most cases, the official Hashicorp Azure Provider for Azure Resource Manager will be enough (1). However, in a few cases, we will have to interact directly with Azure REST API (for example to create an OpenShift cluster object) through the azapi provider (2). The Hashicorp Random Provider will be used to generate a random domain name for our cluster (3). The rest of the providers allow us to interact with OpenShift. Once again, the official Hashicorp Kubernetes Provider is valid in most cases (4). We will also use the kubectl provider (5) and Helm for installing the Postgres database (6) used by the sample apps.

terraform {
  required_version = ">= 1.0"
  required_providers {
    // (1)
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.3.0"
    }
    // (2)
    azapi = {
      source  = "Azure/azapi"
      version = ">=1.0.0"
    }
    // (3)
    random = {
      source = "hashicorp/random"
      version = "3.5.1"
    }
    local = {
      source = "hashicorp/local"
      version = "2.4.0"
    }
  }
}

provider "azurerm" {
  features {}
}

provider "azapi" {
}

provider "random" {}
provider "local" {}

Here’s the list of providers used in the Re Hat Service Mesh installation:

terraform {
  required_version = ">= 1.0"
  required_providers {
    // (4)
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.23.0"
    }
    // (5)
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.13.0"
    }
    // (6)
    helm = {
      source = "hashicorp/helm"
      version = "2.11.0"
    }
  }
}

provider "kubernetes" {
  config_path = "aro/kubeconfig"
  config_context = var.cluster-context
}

provider "kubectl" {
  config_path = "aro/kubeconfig"
  config_context = var.cluster-context
}

provider "helm" {
  kubernetes {
    config_path = "aro/kubeconfig"
    config_context = var.cluster-context
  }
}

In order to install providers, we need to run the following command (you don’t have to do it now):

$ terraform init

Create Azure Red Hat OpenShift Cluster with Terraform

Unfortunately, there is no dedicated, official Terraform provider for creating OpenShift clusters on Azure ARO. There are some discussions about such a feature (you can find it here), but still without a final effect. Maybe it will change in the future. However, creating an ARO cluster is not such a complicated thing since we may use existing providers listed in the previous section. You can find an interesting guide in the Microsoft docs here. It was also a starting point for my work. I improved several things there, for example, to avoid using the az CLI in the scripts and have the full configuration in Terraform HCL.

Let’s analyze our Terraform manifest step by step. Here’s a list of the most important elements we need to place in the HCL file:

1. We have to read some configuration data from the Azure client
2. I have an existing resource group with the openenv prefix, but you can put there any name you want. That’s our main resource group
3. ARO requires a different resource group than a main resource group
4. We need to create a virtual network for Openshift. There is a dedicated subnet for master nodes and another one for worker nodes. All the parameters visible there are required. You can change the IP address range as long as it doesn’t allow for conflicts between the master and worker nodes
5. ARO requires the dedicated service principal to create a cluster. Let’s create the Azure application, and then the service principal with the password. The password is auto-generated by Azure.
6. The newly created service principal requires some privileges. Let’s assign the “User Access Administrator” and network “Contributor”. Then, we need to search the service principal created by Azure under the “Azure Red Hat OpenShift RP” name and also assign a network “Contributor” there.
7. All the required objects have already been created. There is no dedicated resource for the ARO cluster. In order to define the cluster resource we need to leverage the azapi provider.
8. The definition of the OpenShift cluster is available inside the body section. All the fields you see there are required to successfully create the cluster.

// (1)
data "azurerm_client_config" "current" {}
data "azuread_client_config" "current" {}

// (2)
data "azurerm_resource_group" "my_group" {
  name = "openenv-${var.guid}"
}

resource "random_string" "random" {
  length           = 10
  numeric          = false
  special          = false
  upper            = false
}

// (3)
locals {
  resource_group_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/resourceGroups/aro-${random_string.random.result}-${data.azurerm_resource_group.my_group.location}"
  domain            = random_string.random.result
}

// (4)
resource "azurerm_virtual_network" "virtual_network" {
  name                = "aro-vnet-${var.guid}"
  address_space       = ["10.0.0.0/22"]
  location            = data.azurerm_resource_group.my_group.location
  resource_group_name = data.azurerm_resource_group.my_group.name
}
resource "azurerm_subnet" "master_subnet" {
  name                 = "master_subnet"
  resource_group_name  = data.azurerm_resource_group.my_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes     = ["10.0.0.0/23"]
  service_endpoints    = ["Microsoft.ContainerRegistry"]
  private_link_service_network_policies_enabled  = false
  depends_on = [azurerm_virtual_network.virtual_network]
}
resource "azurerm_subnet" "worker_subnet" {
  name                 = "worker_subnet"
  resource_group_name  = data.azurerm_resource_group.my_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes     = ["10.0.2.0/23"]
  service_endpoints    = ["Microsoft.ContainerRegistry"]
  depends_on = [azurerm_virtual_network.virtual_network]
}

// (5)
resource "azuread_application" "aro_app" {
  display_name = "aro_app"
  owners       = [data.azuread_client_config.current.object_id]
}
resource "azuread_service_principal" "aro_app" {
  application_id               = azuread_application.aro_app.application_id
  app_role_assignment_required = false
  owners                       = [data.azuread_client_config.current.object_id]
}
resource "azuread_service_principal_password" "aro_app" {
  service_principal_id = azuread_service_principal.aro_app.object_id
}

// (6)
resource "azurerm_role_assignment" "aro_cluster_service_principal_uaa" {
  scope                = data.azurerm_resource_group.my_group.id
  role_definition_name = "User Access Administrator"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
resource "azurerm_role_assignment" "aro_cluster_service_principal_network_contributor_pre" {
  scope                = data.azurerm_resource_group.my_group.id
  role_definition_name = "Contributor"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
resource "azurerm_role_assignment" "aro_cluster_service_principal_network_contributor" {
  scope                = azurerm_virtual_network.virtual_network.id
  role_definition_name = "Contributor"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
data "azuread_service_principal" "aro_app" {
  display_name = "Azure Red Hat OpenShift RP"
  depends_on = [azuread_service_principal.aro_app]
}
resource "azurerm_role_assignment" "aro_resource_provider_service_principal_network_contributor" {
  scope                = azurerm_virtual_network.virtual_network.id
  role_definition_name = "Contributor"
  principal_id         = data.azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}

// (7)
resource "azapi_resource" "aro_cluster" {
  name      = "aro-cluster-${var.guid}"
  parent_id = data.azurerm_resource_group.my_group.id
  type      = "Microsoft.RedHatOpenShift/openShiftClusters@2023-07-01-preview"
  location  = data.azurerm_resource_group.my_group.location
  timeouts {
    create = "75m"
  }
  // (8)
  body = jsonencode({
    properties = {
      clusterProfile = {
        resourceGroupId      = local.resource_group_id
        pullSecret           = file("~/Downloads/pull-secret-latest.txt")
        domain               = local.domain
        fipsValidatedModules = "Disabled"
        version              = "4.12.25"
      }
      networkProfile = {
        podCidr              = "10.128.0.0/14"
        serviceCidr          = "172.30.0.0/16"
      }
      servicePrincipalProfile = {
        clientId             = azuread_service_principal.aro_app.application_id
        clientSecret         = azuread_service_principal_password.aro_app.value
      }
      masterProfile = {
        vmSize               = "Standard_D8s_v3"
        subnetId             = azurerm_subnet.master_subnet.id
        encryptionAtHost     = "Disabled"
      }
      workerProfiles = [
        {
          name               = "worker"
          vmSize             = "Standard_D8s_v3"
          diskSizeGB         = 128
          subnetId           = azurerm_subnet.worker_subnet.id
          count              = 3
          encryptionAtHost   = "Disabled"
        }
      ]
      apiserverProfile = {
        visibility           = "Public"
      }
      ingressProfiles = [
        {
          name               = "default"
          visibility         = "Public"
        }
      ]
    }
  })
  depends_on = [
    azurerm_subnet.worker_subnet,
    azurerm_subnet.master_subnet,
    azuread_service_principal_password.aro_app,
    azurerm_role_assignment.aro_resource_provider_service_principal_network_contributor
  ]
}

output "domain" {
  value = local.domain
}

Save Kubeconfig

Once we successfully create the OpenShift cluster, we need to obtain and save the kubeconfig file. It will allow Terraform to interact with the cluster through the master API. In order to get the kubeconfig content we need to call the Azure listAdminCredentials REST endpoint. It is the same as calling the az aro get-admin-kubeconfig command using CLI. It will return JSON with base64-encoded content. After decoding from JSON and Base64 we save the content inside the kubeconfig file in the current directory.

resource "azapi_resource_action" "test" {
  type        = "Microsoft.RedHatOpenShift/openShiftClusters@2023-07-01-preview"
  resource_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/resourceGroups/openenv-${var.guid}/providers/Microsoft.RedHatOpenShift/openShiftClusters/aro-cluster-${var.guid}"
  action      = "listAdminCredentials"
  method      = "POST"
  response_export_values = ["*"]
}

output "kubeconfig" {
  value = base64decode(jsondecode(azapi_resource_action.test.output).kubeconfig)
}

resource "local_file" "kubeconfig" {
  content  =  base64decode(jsondecode(azapi_resource_action.test.output).kubeconfig)
  filename = "kubeconfig"
  depends_on = [azapi_resource_action.test]
}

Install OpenShift Operators with Terraform

Finally, we can interact with the existing OpenShift cluster via the kubeconfig file. In the first step, we will deploy some operators. In OpenShift operators are a preferred way of installing more advanced apps (for example consisting of several Deployments). Red Hat comes with a set of supported operators that allows us to extend OpenShift functionalities. It can be, for example, a service mesh, a clustered database, or a message broker.

Let’s imagine we want to install a service mesh on OpenShift. There are some dedicated operators for that. The OpenShift Service Mesh operator is built on top of the open-source project Istio. We will also install the OpenShift Distributed Tracing (Jaeger) and Kiali operators. In order to do that we need to define the Subscription CRD object. Also, if we install an operator in a different namespace than openshift-operators we have to create the OperatorGroup CRD object. Here’s the Terraform HCL script that installs our operators.

// (1)
resource "kubernetes_namespace" "openshift-distributed-tracing" {
  metadata {
    name = "openshift-distributed-tracing"
  }
}
resource "kubernetes_manifest" "tracing-group" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1"
    "kind"       = "OperatorGroup"
    "metadata"   = {
      "name"      = "openshift-distributed-tracing"
      "namespace" = "openshift-distributed-tracing"
    }
    "spec" = {
      "upgradeStrategy" = "Default"
    }
  }
}
resource "kubernetes_manifest" "tracing" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata" = {
      "name"      = "jaeger-product"
      "namespace" = "openshift-distributed-tracing"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "jaeger-product"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (2)
resource "kubernetes_manifest" "kiali" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata" = {
      "name"      = "kiali-ossm"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "kiali-ossm"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (3)
resource "kubernetes_manifest" "ossm" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata"   = {
      "name"      = "servicemeshoperator"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "servicemeshoperator"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (4)
resource "kubernetes_manifest" "ossmconsole" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata"   = {
      "name"      = "ossmconsole"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "candidate"
      "installPlanApproval" = "Automatic"
      "name"                = "ossmconsole"
      "source"              = "community-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

After installing the operators we may proceed to the service mesh configuration (1). We need to use CRD objects installed by the operators. Kubernetes Terraform provider won’t be a perfect choice for that since it verifies the existence of an object before applying the whole script. Therefore we will switch to the kubectl provider that just applies the object without any initial verification. We need to create an Istio control plane using the ServiceMeshControlPlane object (2). As you see, it also enables distributed tracing with Jaeger and a dashboard with Kiali. Once a control plane is ready we may proceed to the next steps. We will create all the objects responsible for Istio configuration including VirtualService, DestinatioRule, and Gateway (3).

resource "kubernetes_namespace" "istio" {
  metadata {
    name = "istio"
  }
}

// (1)
resource "time_sleep" "wait_120_seconds" {
  depends_on = [kubernetes_manifest.ossm]

  create_duration = "120s"
}

// (2)
resource "kubectl_manifest" "basic" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio]
  yaml_body = <<YAML
kind: ServiceMeshControlPlane
apiVersion: maistra.io/v2
metadata:
  name: basic
  namespace: istio
spec:
  version: v2.4
  tracing:
    type: Jaeger
    sampling: 10000
  policy:
    type: Istiod
  telemetry:
    type: Istiod
  addons:
    jaeger:
      install:
        storage:
          type: Memory
    prometheus:
      enabled: true
    kiali:
      enabled: true
    grafana:
      enabled: true
YAML
}

resource "kubectl_manifest" "console" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio]
  yaml_body = <<YAML
kind: OSSMConsole
apiVersion: kiali.io/v1alpha1
metadata:
  name: ossmconsole
  namespace: istio
spec:
  kiali:
    serviceName: ''
    serviceNamespace: ''
    servicePort: 0
    url: ''
YAML
}

resource "time_sleep" "wait_60_seconds_2" {
  depends_on = [kubectl_manifest.basic]

  create_duration = "60s"
}

// (3)
resource "kubectl_manifest" "access" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
  name: default
  namespace: istio
spec:
  members:
    - demo-apps
YAML
}

resource "kubectl_manifest" "gateway" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
  namespace: demo-apps
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - quarkus-insurance-app.apps.${var.domain}
        - quarkus-person-app.apps.${var.domain}
YAML
}

resource "kubectl_manifest" "quarkus-insurance-app-vs" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-insurance-app-vs
  namespace: demo-apps
spec:
  hosts:
    - quarkus-insurance-app.apps.${var.domain}
  gateways:
    - microservices-gateway
  http:
    - match:
        - uri:
            prefix: "/insurance"
      rewrite:
        uri: " "
      route:
        - destination:
            host: quarkus-insurance-app
          weight: 100
YAML
}

resource "kubectl_manifest" "quarkus-person-app-dr" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: quarkus-person-app-dr
  namespace: demo-apps
spec:
  host: quarkus-person-app
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
YAML
}

resource "kubectl_manifest" "quarkus-person-app-vs-via-gw" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-person-app-vs-via-gw
  namespace: demo-apps
spec:
  hosts:
    - quarkus-person-app.apps.${var.domain}
  gateways:
    - microservices-gateway
  http:
    - match:
      - uri:
          prefix: "/person"
      rewrite:
        uri: " "
      route:
        - destination:
            host: quarkus-person-app
            subset: v1
          weight: 100
        - destination:
            host: quarkus-person-app
            subset: v2
          weight: 0
YAML
}

resource "kubectl_manifest" "quarkus-person-app-vs" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-person-app-vs
  namespace: demo-apps
spec:
  hosts:
    - quarkus-person-app
  http:
    - route:
        - destination:
            host: quarkus-person-app
            subset: v1
          weight: 100
        - destination:
            host: quarkus-person-app
            subset: v2
          weight: 0
YAML
}

Finally, we will run our sample Quarkus apps that communicate through the Istio mesh and connect to the Postgres database. The script is quite large. All the apps are running in the demo-apps namespace (1). They are connecting with the Postgres database installed using the Terraform Helm provider from Bitnami chart (2). Finally, we are creating the Deployments for two apps: person-service and insurance-service (3). There are two versions per microservice. Don’t focus on the features of the apps. They are here just to show the subsequent layers of the installation process. We are starting with the operators and CRDs, then moving to Istio configuration, and finally installing our custom apps.

// (1)
resource "kubernetes_namespace" "demo-apps" {
  metadata {
    name = "demo-apps"
  }
}

resource "kubernetes_secret" "person-db-secret" {
  depends_on = [kubernetes_namespace.demo-apps]
  metadata {
    name      = "person-db"
    namespace = "demo-apps"
  }
  data = {
    postgres-password = "123456"
    password          = "123456"
    database-user     = "person-db"
    database-name     = "person-db"
  }
}

resource "kubernetes_secret" "insurance-db-secret" {
  depends_on = [kubernetes_namespace.demo-apps]
  metadata {
    name      = "insurance-db"
    namespace = "demo-apps"
  }
  data = {
    postgres-password = "123456"
    password          = "123456"
    database-user     = "insurance-db"
    database-name     = "insurance-db"
  }
}

// (2)
resource "helm_release" "person-db" {
  depends_on = [kubernetes_namespace.demo-apps]
  chart            = "postgresql"
  name             = "person-db"
  namespace        = "demo-apps"
  repository       = "https://charts.bitnami.com/bitnami"

  values = [
    file("manifests/person-db-values.yaml")
  ]
}
resource "helm_release" "insurance-db" {
  depends_on = [kubernetes_namespace.demo-apps]
  chart            = "postgresql"
  name             = "insurance-db"
  namespace        = "demo-apps"
  repository       = "https://charts.bitnami.com/bitnami"

  values = [
    file("manifests/insurance-db-values.yaml")
  ]
}

// (3)
resource "kubernetes_deployment" "quarkus-insurance-app" {
  depends_on = [helm_release.insurance-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-insurance-app"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-insurance-app"
        version = "v1"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-insurance-app"
          version = "v1"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-insurance-app"
          image = "piomin/quarkus-insurance-app:v1"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "insurance-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "insurance-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "insurance-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_service" "quarkus-insurance-app" {
  depends_on = [helm_release.insurance-db, time_sleep.wait_60_seconds_2]
  metadata {
    name = "quarkus-insurance-app"
    namespace = "demo-apps"
    labels = {
      app = "quarkus-insurance-app"
    }
  }
  spec {
    type = "ClusterIP"
    selector = {
      app = "quarkus-insurance-app"
    }
    port {
      port = 8080
      name = "http"
    }
  }
}

resource "kubernetes_deployment" "quarkus-person-app-v1" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-person-app-v1"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-person-app"
        version = "v1"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-person-app"
          version = "v1"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-person-app"
          image = "piomin/quarkus-person-app:v1"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "person-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_deployment" "quarkus-person-app-v2" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-person-app-v2"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-person-app"
        version = "v2"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-person-app"
          version = "v2"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-person-app"
          image = "piomin/quarkus-person-app:v2"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "person-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_service" "quarkus-person-app" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name = "quarkus-person-app"
    namespace = "demo-apps"
    labels = {
      app = "quarkus-person-app"
    }
  }
  spec {
    type = "ClusterIP"
    selector = {
      app = "quarkus-person-app"
    }
    port {
      port = 8080
      name = "http"
    }
  }
}

Applying Terraform Scripts

Finally, we can apply the whole Terraform configuration described in the article. Here’s the aro-with-servicemesh.sh script responsible for running required Terraform commands. It is placed in the repository root directory. In the first step, we go to the aro directory to apply the script responsible for creating the Openshift cluster. The domain name is automatically generated by Terraform, so we will export it using the terraform output command. After that, we may apply the scripts with operators and Istio configuration. In order to do everything automatically we pass the location of the kubeconfig file and the generated domain name as variables.

#! /bin/bash

cd aro
terraform init
terraform apply -auto-approve
domain="apps.$(terraform output -raw domain).eastus.aroapp.io"

cd ../servicemesh
terraform init
terraform apply -auto-approve -var kubeconfig=../aro/kubeconfig -var domain=$domain

Let’s run the aro-with-service-mesh.sh script. Once you will do it you should have a similar output as visible below. In the beginning, Terraform creates several objects required by the ARO cluster like a virtual network or service principal. Once those resources are ready, it starts the main part – ARO installation.

Let’s switch to Azure Portal. As you see the installation is in progress. There are several other newly created resources. Of course, there is also the resource representing the OpenShift cluster.

Now, arm yourself with patience. You can easily go get a coffee…

You can verify the progress, e.g. by displaying a list of virtual machines. If you see all the 3 master and 3 worker VMs running it means that we are slowly approaching the end.

It may take even more than 40 minutes. That’s why I overridden a default timeout for azapi resource to 75 minutes. Once the cluster is ready, Terraform will connect to the instance of OpenShift to install operators there. In the meantime, we can switch to Azure Portal and see the details about the ARO cluster. It displays, among others, the OpenShift Console URL. Let’s log in to the console.

In order to obtain the admin password we need to run the following command (for my cluster and resource group name):

$ az aro list-credentials -n aro-cluster-p2pvg -g openenv-p2pvg

Here’s our OpenShift console:

Let’s back to the installation process. The first part has been just finished. Now, the script executes terraform commands in the servicemesh directory. As you see, it installed our operators.

Let’s check out how it looks in the OpenShift Console. Go to the Operators -> Installed Operators menu item.

Of course, the installation is continued in the background. After installing the operators, it created the Istio Control Plane using the CRD object.

Let’s switch to the OpenShift Console once again. Go to the istio project. In the list of installed operators find Red Hat OpenShift Service Mesh and then go to the Istio Service Mesh Control Plane tab. You should see the basic object. As you see all 9 required components, including Istio, Kiali, and Jaeger instances, are successfully installed.

And finally the last part of our exercise. Installation is finished. Terraform applied deployment with our Postgres databases and some Quarkus apps.

In order to see the list of apps we can go to the Topology view in the Developer perspective. All the pods are running. As you see there is also a Kiali console available. We can click that link.

In the Kiali dashboard, we can see a detailed view of our service mesh. For example, there is a diagram showing a graphical visualization of traffic between the services.

Final Thoughts

If you use Terraform for managing your cloud infrastructure this article is for you. Did you already have doubts is it possible to easily create and configure the OpenShift cluster with Terraform? This article should dispel your doubts. You can also easily create your ARO cluster just by cloning this repository and running a single script on your cloud account. Enjoy

The post Manage OpenShift with Terraform appeared first on Piotr's TechBlog.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then you should just follow my instructions.

Install Loki Stack on Kubernetes

In the first step, we will install Loki Stack on Kubernetes. The most convenient way to do it is through the Helm chart. Fortunately, there is a single Helm chart that installs and configures all the tools required in our exercise: Loki, Promtail, and Grafana. Let’s add the following Helm repository:

$ helm repo add grafana https://grafana.github.io/helm-charts

Then, we can install the loki-stack chart. By default, it does not install Grafana. In order to enable Grafana we need to set the grafana.enabled parameter to true. Our Loki Stack is installed in the loki-stack namespace:

$ helm install loki grafana/loki-stack \
  -n loki-stack \
  --set grafana.enabled=true \
  --create-namespace

Here’s a list of running pods in the loki-stack namespace:

$ kubectl get po -n loki-stack
NAME                           READY   STATUS    RESTARTS   AGE
loki-0                         1/1     Running   0          78s
loki-grafana-bf598db67-czcds   2/2     Running   0          93s
loki-promtail-vt25p            1/1     Running   0          30s

Let’s enable port forwarding to access the Grafana dashboard on the local port:

$ kubectl port-forward svc/loki-grafana 3000:80 -n loki-stack

Helm chart automatically generates a password for the admin user. We can obtain it with the following command:

$ kubectl get secret -n loki-stack loki-grafana \
    -o jsonpath="{.data.admin-password}" | \
    base64 --decode ; echo

Once we login into the dashboard we will see the auto-configured Loki datasource. We can use it to get the latest logs from the Kubernetes cluster:

Create and Deploy Quarkus App on Kubernetes

In the next step, we will install our sample Quarkus app on Kubernetes. It connects to the Postgres database. Therefore, we will also install Postgres with the Bitnami Helm chart:

$ helm install person-db bitnami/postgresql -n sample-quarkus \
  --set auth.username=quarkus  \
  --set auth.database=quarkus  \
  --set fullnameOverride=person-db \
  --create-namespace

With Quarkus we can easily change the logs format to JSON. We just need to include the following Maven dependency:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-logging-json</artifactId>
</dependency>

And also enable JSON logging in the application properties:

quarkus.log.console.json = true

Besides the static logging fields, we will include a single dynamic field. We will use the MDC mechanism for that (1) (2). That field indicates the id of the person for whom we make the GET or POST request. Here’s the code of the REST controller:

@Path("/persons")
public class PersonResource {

    private PersonRepository repository;
    private Logger logger;

    public PersonResource(PersonRepository repository, Logger logger) {
        this.repository = repository;
        this.logger = logger;
    }

    @POST
    @Transactional
    public Person add(Person person) {
        repository.persist(person);
        MDC.put("personId", person.id); // (1)
        logger.infof("IN -> add(%s)", person);
        return person;
    }

    @GET
    @APIResponseSchema(Person.class)
    public List<Person> findAll() {
        logger.info("IN -> findAll");
        return repository.findAll()
                .list();
    }

    @GET
    @Path("/{id}")
    public Person findById(@PathParam("id") Long id) {
        MDC.put("personId", id); // (2)
        logger.infof("IN -> findById(%d)", id);
        return repository.findById(id);
    }
}

Here’s the sample log for the GET endpoint. Now, our goal is to parse and index it properly in Loki with Promtail.

Now, we need to deploy our sample app on Kubernetes. Fortunately, with Quarkus we can build and deploy the app using the single Maven command. We just need to activate the following custom profile which includes quarkus-kubernetes dependency and enables deployment with the quarkus.kubernetes.deploy property. It also activates image build using the Jib Maven Plugin.

<profile>
  <id>kubernetes</id>
  <activation>
    <property>
      <name>kubernetes</name>
    </property>
  </activation>
  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-container-image-jib</artifactId>
    </dependency>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-kubernetes</artifactId>
    </dependency>
  </dependencies>
  <properties>
    <quarkus.kubernetes.deploy>true</quarkus.kubernetes.deploy>
  </properties>
</profile>

Let’s build and deploy the app:

$ mvn clean package -DskipTests -Pkubernetes

Here’s the list of running pods (database and app):

$ kubectl get po -n sample-quarkus
NAME                             READY   STATUS    RESTARTS   AGE
person-db-0                      1/1     Running   0          48s
person-service-9f67b6d57-gvbs6   1/1     Running   0          18s

Configure Promptail to Parse JSON Logs

Let’s take a look at the Promtail configuration. We can find it inside the loki-promtail Secret. As you see it uses only the cri component.

server:
  log_level: info
  http_listen_port: 3101


clients:
  - url: http://loki:3100/loki/api/v1/push

positions:
  filename: /run/promtail/positions.yaml

scrape_configs:
  - job_name: kubernetes-pods
    pipeline_stages:
      - cri: {}
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels:
          - __meta_kubernetes_pod_controller_name
        regex: ([0-9a-z-.]+?)(-[0-9a-f]{8,10})?
        action: replace
        target_label: __tmp_controller_name
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_name
          - __meta_kubernetes_pod_label_app
          - __tmp_controller_name
          - __meta_kubernetes_pod_name
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: app
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_instance
          - __meta_kubernetes_pod_label_release
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: instance
      - source_labels:
          - __meta_kubernetes_pod_label_app_kubernetes_io_component
          - __meta_kubernetes_pod_label_component
        regex: ^;*([^;]+)(;.*)?$
        action: replace
        target_label: component
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_node_name
        target_label: node_name
      - action: replace
        source_labels:
          - __meta_kubernetes_namespace
        target_label: namespace
      - action: replace
        replacement: $1
        separator: /
        source_labels:
          - namespace
          - app
        target_label: job
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_name
        target_label: pod
      - action: replace
        source_labels:
          - __meta_kubernetes_pod_container_name
        target_label: container
      - action: replace
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_uid
          - __meta_kubernetes_pod_container_name
        target_label: __path__
      - action: replace
        regex: true/(.*)
        replacement: /var/log/pods/*$1/*.log
        separator: /
        source_labels:
          - __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
          - __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
          - __meta_kubernetes_pod_container_name
        target_label: __path__

The result for our app is quite inadequate. Loki stores the full Kubernetes pods’ log lines and doesn’t recognize our logging fields.

In order to change that behavior we will parse data using the json component. This action will be limited just to our sample application (1). We will label the log records with level, sequence, and the personId MDC field (2) after extracting them from the Kubernetes log line. The mdc field contains a list of objects, so we need to perform additional JSON parsing (3) to extract the personId field. As the output, Promtail should return the log message field (4). Here’s the required transformation in the configuration file:

- job_name: kubernetes-pods
  pipeline_stages:
    - cri: {}
    - match:
        selector: '{app="person-service"}' # (1)
        stages:
          - json:
              expressions:
                log:
          - json: # (2)
              expressions:
                sequence: sequence
                message: message
                level: level
                mdc:
              source: log
          - json: # (3)
              expressions:
                personId: personId
              source: mdc
          - labels:
              sequence:
              level:
              personId:
          - output: # (4)
              source: message

After setting a new value of the loki-promtail Secret we should restart the Promtail pod. Let’s also restart our app and perform some test calls of the REST API:

$ curl http://localhost:8080/persons/1

$ curl http://localhost:8080/persons/6

$ curl -X 'POST' http://localhost:8080/persons \
  -H 'Content-Type: application/json' \
  -d '{
  "name": "John Wick",
  "age": 18,
  "gender": "MALE",
  "externalId": 100,
  "address": {
    "street": "Test Street",
    "city": "Warsaw",
    "flatNo": 18,
    "buildingNo": 100
  }
}'

Let’s see how it looks in Grafana:

As you see, the log record for the GET request is labeled with level, sequence and the personId MDC field. That’s what we exactly wanted to achieve!

Now, we are able to filter results using the fields from our JSON log line:

Distributed Installation of Loki Stack

In the previously described installation method, we run a single instance of Loki. In order to use a more cloud-native and scalable approach we should switch to the loki-distributed Helm chart. It decides a single Loki instance into several independent components. That division also separates read and write streams. Let’s install it in the loki-distributed namespace with the following command:

$ helm install loki grafana/loki-distributed \
  -n loki-distributed --create-namespace

When installing Promtail we should modify the default address of the write endpoint. We use the Loki gateway component for that. In our case the name of the gateway Service is loki-loki-distributed-gateway. That component listens on the 80 port.

config:
  clients:
  - url: http://loki-loki-distributed-gateway/loki/api/v1/push

Let’s install Promtail using the following command:

$ helm install promtail grafana/promtail -n loki-distributed \
  -f values.yml

Finally, we should install Grafana. The same as before we will use a dedicated Helm chart:

$ helm install grafana grafana/grafana -n loki-distributed

Here’s a list of running pods:

$ kubectl get pod -n loki-distributed
NAME                                                    READY   STATUS    RESTARTS   AGE
grafana-6cd56666b9-6hvqg                                1/1     Running   0          42m
loki-loki-distributed-distributor-59767b5445-n59bq      1/1     Running   0          48m
loki-loki-distributed-gateway-7867bc8ddb-kgdfk          1/1     Running   0          48m
loki-loki-distributed-ingester-0                        1/1     Running   0          48m
loki-loki-distributed-querier-0                         1/1     Running   0          48m
loki-loki-distributed-query-frontend-86c944647c-vl2bz   1/1     Running   0          48m
promtail-c6dxj                                          1/1     Running   0          37m

After logging in to Grafana, we should add the Loki data source (we could also do it during the installation with Helm values). This time we have to connect to the query-frontend component available under the address loki-loki-distributed-query-frontend:3100.

Final Thoughts

Loki Stack is an interesting alternative to Elastic Stack for collecting and aggregating logs on Kubernetes. Loki has been designed to be very cost-effective and easy to operate. Since it does not index the contents of the logs, the usage of such resources as disk space or RAM memory is lower than for Elasticsearch. In this article, I showed you how to install Loki Stack on Kubernetes and how to configure it to analyze app logs in practice.

The post Logging in Kubernetes with Loki appeared first on Piotr's TechBlog.

In this article, I’m describing several more advanced Argo CD features. If you looking for the basics you can find a lot of other articles about Argo CD on my blog. For example, you may about Kubernetes CI/CD with Tekton and ArgoCD in the following article.

Introduction

The main goal of this exercise is to run the scenario, in which we can automatically install and use operators on Kubernetes in the GitOps way. Therefore, the state of the Git repository should be automatically applied to the target Kubernetes cluster. We will define a single Argo CD Application that performs all the required steps. In the first step, it will trigger the operator installation process. It may take some time since we need to install the controller application and Kubernetes CRDs. Then we may define some CR objects to run our apps on the cluster.

We cannot create a CR object before installing an operator. Fortunately, with ArgoCD we can divide the sync process into multiple separate phases. This ArgoCD feature is called sync waves. In order to proceed to the next phase, ArgoCD first needs to finish the previous sync wave. ArgoCD checks the health checks of all objects created during the particular phase. If all of those checks reply with success the phase is considered to be finished. Argo CD provides some built-in health check implementations for several standard Kubernetes types. However, in this exercise, we will have to override the health check for the main operator CR – the Subscription object.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. After that go to the global directory. Then you should just follow my instructions. Let’s begin.

Prerequisites

Before starting the exercise you need to have a running Kubernetes cluster with ArgoCD and Operator Lifecycle Manager (OLM) installed. You can install Argo CD using Helm chart or with the operator. In order to read about the installation details please refer to the Argo CD docs.

Install Operators with Argo CD

In the first step, we will define templates responsible for operators’ installation. If you have OLM installed on the Kubernetes cluster that process comes to the creation of the Subscription object (1). In some cases, we have to create the OperatorGroup object. It provides multitenant configuration to OLM-installed Operators. An Operator group selects target namespaces in which to generate required RBAC access for its members. Before installing in a different namespace than openshift-operators, we have to create the OperatorGroup in that namespace (2). We use the argocd.argoproj.io/sync-wave annotation to configure sync phases (3). The lower value of that parameter is – the highest priority for the object (before OperatorGroup we need to create the namespace).

{{- range .Values.subscriptions }}
apiVersion: operators.coreos.com/v1alpha1 # (1)
kind: Subscription
metadata:
  name: {{ .name }}
  namespace: {{ .namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "2" # (3)
spec:
  channel: {{ .channel }}
  installPlanApproval: Automatic
  name: {{ .name }}
  source: {{ .source }}
  sourceNamespace: openshift-marketplace
---
{{- if ne .namespace "openshift-operators" }}
apiVersion: v1
kind: Namespace
metadata:
  name: {{ .namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "1" # (3)
---
apiVersion: operators.coreos.com/v1alpha2 # (2)
kind: OperatorGroup
metadata:
  name: {{ .name }}
  namespace: {{ .namespace }}
  annotations:
    argocd.argoproj.io/sync-wave: "2" # (3)
spec: {}
---
{{- end }}
{{- end }}

I’m using Helm for templating the YAML manifests. Thanks to that we can use it to apply several Subscription and OperatorGroup objects. Our Helm templates iterate over the subscriptions list. In order to define a list of operators we just need to provide a similar configuration in the values.yaml file visible below. There are operators installed with that example: Kiali, Service Mesh (Istio), AMQ Streams (Strimzi Kafka), Patch Operator, and Serverless (Knative).

subscriptions:
  - name: kiali-ossm
    namespace: openshift-operators
    channel: stable
    source: redhat-operators
  - name: servicemeshoperator
    namespace: openshift-operators
    channel: stable
    source: redhat-operators
  - name: amq-streams
    namespace: openshift-operators
    channel: stable
    source: redhat-operators
  - name: patch-operator
    namespace: patch-operator
    channel: alpha
    source: community-operators
  - name: serverless-operator
    namespace: openshift-serverless
    channel: stable
    source: redhat-operators

Override Argo CD Health Check

As I mentioned before, we need to override the default Argo CD health check for the Subscription CR. Normally, Argo CD just creates the Subscription objects and doesn’t wait until the operator is installed on the cluster. In order to do that, we need to verify the value of the status.state field. If it equals the AtLatestKnown value, it means that the operator has been successfully installed. In that case, we can set the value of the Argo CD health check to Healthy. We can also override the default health check description to display the current version of the operator (the status.currentCSV field). If you installed Argo CD using Helm chart you can provide your health check implementation directly in the argocd-cm ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
  labels:
    app.kubernetes.io/name: argocd-cm
    app.kubernetes.io/part-of: argocd
data:
  resource.customizations: |
    operators.coreos.com/Subscription:
      health.lua: |
        hs = {}
        hs.status = "Progressing"
        hs.message = ""
        if obj.status ~= nil then
          if obj.status.state ~= nil then
            if obj.status.state == "AtLatestKnown" then
              hs.message = obj.status.state .. " - " .. obj.status.currentCSV
              hs.status = "Healthy"
            end
          end
        end
        return hs

For those of you, who installed Argo CD using the operator (including me) there is another way to override the health check. We need to provide it inside the extraConfig field in the ArgoCD CR.

apiVersion: argoproj.io/v1alpha1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  ...
  extraConfig:
    resource.customizations: |
      operators.coreos.com/Subscription:
        health.lua: |
          hs = {}
          hs.status = "Progressing"
          hs.message = ""
          if obj.status ~= nil then
            if obj.status.state ~= nil then
              if obj.status.state == "AtLatestKnown" then
                hs.message = obj.status.state .. " - " .. obj.status.currentCSV
                hs.status = "Healthy"
              end
            end
          end
          return hs

After the currently described steps, we achieved two things. We divided our sync process into multiple phases with the Argo CD waves feature. We also forced Argo CD to wait before going to the next phase until the operator installation process is finished. Let’s proceed to the next step – defining CRDs.

Create Custom Resources with Argo CD

In the previous steps, we successfully installed Kubernetes operators with ArgoCD. Now, it is time to use them. We will do everything in a single synchronization process. In the previous phase (wave=2), we installed the Kafka operator (Strimzi). In this phase, we will run the Kafka cluster using CRD provided by the Strimzi project. To be sure that we apply it after the Strimzi operator installation, we will do it in the third phase (1). That’s not all. Since our CRD has been created by the operator, it is not part of the sync process. By default, Argo CD tries to find the CRD in the sync and will fail with the error the server could not find the requested resource. To avoid it we will skip the dry run for missing resource types (2) during sync.

apiVersion: v1
kind: Namespace
metadata:
  name: kafka
  annotations:
    argocd.argoproj.io/sync-wave: "1"
---
apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
  annotations:
    argocd.argoproj.io/sync-wave: "3" # (1)
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true # (2)
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.2'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    version: 3.2.3
    replicas: 3
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      deleteClaim: true
      size: 2Gi
    replicas: 3

We can also install Knative Serving on our cluster since we previously installed the Knative operator. The same as before we are setting the wave=3 and skipping the dry run on missing resources during the sync.

apiVersion: operator.knative.dev/v1beta1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec: {}

Finally, let’s create the Argo CD Application that manages all the defined manifests and automatically applies them to the Kubernetes cluster. We need to define the source Git repository and the directory containing our YAMLs (global).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cluster-config
spec:
  destination:
    server: 'https://kubernetes.default.svc'
  project: default
  source:
    path: global
    repoURL: 'https://github.com/piomin/openshift-cluster-config.git'
    targetRevision: HEAD
    helm:
      valueFiles:
        - values.yaml
  syncPolicy:
    automated:
      selfHeal: true

Helm Unit Testing

Just to ensure that we defined all the Helm templates properly we can include some unit tests. We can use helm-unittest for that. We will place the test sources inside the global/tests directory. Here’s our test defined in the subscription_tests.yaml file:

suite: test main
values:
  - ./values/test.yaml
templates:
  - templates/subscriptions.yaml
chart:
  version: 1.0.0+test
  appVersion: 1.0
tests:
  - it: subscription default ns
    template: templates/subscriptions.yaml
    documentIndex: 0
    asserts:
      - equal:
          path: metadata.namespace
          value: openshift-operators
      - equal:
          path: metadata.name
          value: test1
      - equal:
          path: spec.channel
          value: ch1
      - equal:
          path: spec.source
          value: src1
      - isKind:
          of: Subscription
      - isAPIVersion:
          of: operators.coreos.com/v1alpha1
  - it: subscription custom ns
    template: templates/subscriptions.yaml
    documentIndex: 1
    asserts:
      - equal:
          path: metadata.namespace
          value: custom-ns
      - equal:
          path: metadata.name
          value: test2
      - equal:
          path: spec.channel
          value: ch2
      - equal:
          path: spec.source
          value: src2
      - isKind:
          of: Subscription
      - isAPIVersion:
          of: operators.coreos.com/v1alpha1
  - it: custom ns
    template: templates/subscriptions.yaml
    documentIndex: 2
    asserts:
      - equal:
          path: metadata.name
          value: custom-ns
      - isKind:
          of: Namespace
      - isAPIVersion:
          of: v1

We need to define test values:

subscriptions:
  - name: test1
    namespace: openshift-operators
    channel: ch1
    source: src1
  - name: test2
    namespace: custom-ns
    channel: ch2
    source: src2

We can prepare a build process for our repository. Here’s a sample Circle CI configuration for that. If you are interested in more details about Helm unit testing and releasing please refer to my article.

version: 2.1

orbs:
  helm: circleci/helm@2.0.1

jobs:
  build:
    docker:
      - image: cimg/base:2023.04
    steps:
      - checkout
      - helm/install-helm-client
      - run:
          name: Install Helm unit-test
          command: helm plugin install https://github.com/helm-unittest/helm-unittest
      - run:
          name: Run unit tests
          command: helm unittest global

workflows:
  helm_test:
    jobs:
      - build

Synchronize Configuration with Argo CD

Once we create a new Argo CD Application responsible for synchronization our process is starting. In the first step, Argo CD creates the required namespaces. Then, it proceeds to the operators’ installation phase. It may take some time.

Once ArgoCD installs all the Kubernetes operators you can verify their health checks. Here’s the value of a health check during the installation phase.

Here’s the result after successful installation.

Now, Argo CD is proceeding to the CRDs creation phase. It runs the Kafka cluster and enables Knative. Let’s switch to the Openshift cluster console. We can display a list of installed operators:

We can also verify if the Kafka cluster is running in the kafka namespace:

Final Thoughts

With Argo CD we can configure the whole Kubernetes cluster configuration. It supports Helm charts, but there is another way for installing apps on Kubernetes – operators. I focused on the features and approach that allow us to install and manage operators in the GitOps way. I showed a practical example of how to use sync waves and apply CRDs not managed directly by Argo CD. With all mechanisms, we can easily handle Kubernetes operators with ArgoCD.

The post Manage Kubernetes Operators with ArgoCD appeared first on Piotr's TechBlog.

If you are interested in the Helm chart and CI/CD process you may refer to the following article. It shows how to design your continuous development process on Kubernetes and use Helm charts in the GitOps approach.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions.

Create Helm Chart

In this part of the exercise, we will use the helm CLI. Helm installed locally is not required in our whole process, but helps you understand what will happen in the next steps. Therefore, it is worth installing it. Please refer to the official Helm docs to find an installation approach suitable for your needs.

In the first step, we are going to create a sample chart. It is a typical chart for web apps. For example, it exposes the 8080 port outside of the containers or allows us to define liveness and readiness probes checking HTTP endpoints. This Helm chart should not be too complicated, but also not too simple, since we want to create automated tests for it.

Here’s our Deployment template. It adds some standard labels to the Deployment manifest (1). It also sets resource requests and limits (2). As I mentioned before, our chart is adding liveness probe (3), readiness probe (4), and exposes port 8080 outside of the container (5). We may also set environment variables (6), or inject them from ConfigMap (7) and Secret (8).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
  labels: # (1)
    app: {{ .Values.app.name }}
    env: {{ .Values.app.environment }}
    owner: {{ .Values.app.owner }}
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
        env: {{ .Values.app.environment }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.repository }}:{{ .Values.image.tag }}
          resources: # (2)
            {{- toYaml .Values.resources | nindent 12 }}
          livenessProbe: # (3)
            initialDelaySeconds: {{ .Values.liveness.initialDelaySeconds }}
            httpGet:
              port: {{ .Values.liveness.port }}
              path: {{ .Values.liveness.path }}
            failureThreshold: {{ .Values.liveness.failureThreshold }}
            successThreshold: {{ .Values.liveness.successThreshold }}
            timeoutSeconds: {{ .Values.liveness.timeoutSeconds }}
            periodSeconds: {{ .Values.liveness.periodSeconds }}
          readinessProbe: # (4)
            initialDelaySeconds: {{ .Values.readiness.initialDelaySeconds }}
            httpGet:
              port: {{ .Values.readiness.port }}
              path: {{ .Values.readiness.path }}
            failureThreshold: {{ .Values.readiness.failureThreshold }}
            successThreshold: {{ .Values.readiness.successThreshold }}
            timeoutSeconds: {{ .Values.readiness.timeoutSeconds }}
            periodSeconds: {{ .Values.readiness.periodSeconds }}
          ports: # (5)
          {{- range .Values.ports }}
          - containerPort: {{ .value }}
            name: {{ .name }}
          {{- end }}
          {{- if .Values.envs }}
          env: # (6)
          {{- range .Values.envs }}
          - name: {{ .name }}
            value: {{ .value }}
          {{- end }}
          {{- end }}
          {{- if or .Values.extraEnvVarsConfigMap .Values.extraEnvVarsSecret }}
          envFrom:
          {{- if or .Values.extraEnvVarsConfigMap }}
          - configMapRef:
              name: {{ .Values.extraEnvVarsConfigMap }}
          {{- end }}
          {{- if or .Values.extraEnvVarsSecret }}
          - secretRef:
              name: {{ .Values.extraEnvVarsSecret }}
          {{- end }}
          {{- end }}
          securityContext:
            runAsNonRoot: true

We also have a template for the Service object.

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.app.name }}
  labels:
    app: {{ .Values.app.name }}
    env: {{ .Values.app.environment }}
    owner: {{ .Values.app.owner }}
spec:
  type: {{ .Values.service.type }}
  selector:
    app: {{ .Values.app.name }}
  ports:
  {{- range .Values.ports }}
  - port: {{ .value }}
    name: {{ .name }}
  {{- end }}

Now, we can fill our templates with the default values. The following values.yaml file is available in the sample repository.

replicaCount: 1

app:
  name: sample-spring-boot-api
  environment: dev
  owner: default

image:
  repository: piomin/sample-spring-kotlin-microservice
  tag: "1.1"

nameOverride: ""
fullnameOverride: ""

service:
  type: ClusterIP

ports:
  - name: http
    value: 8080

resources:
  limits:
    cpu: 1000m
    memory: 512Mi
  requests:
    cpu: 100m
    memory: 256Mi

liveness:
  initialDelaySeconds: 10
  port: http
  path: /actuator/health/liveness
  failureThreshold: 3
  successThreshold: 1
  timeoutSeconds: 3
  periodSeconds: 5

readiness:
  initialDelaySeconds: 10
  port: http
  path: /actuator/health/readiness
  failureThreshold: 3
  successThreshold: 1
  timeoutSeconds: 3
  periodSeconds: 5

envs:
  - name: INFO
    value: Spring Boot REST API

You can easily test the newly created templates with the helm CLI. In order to do that, just execute the following command in the repository root directory. As a result, you will see the YAML manifests created from our sample templates.

$ helm template charts/spring-boot-api-app

Such a testing method is fine… but just to run it locally during chart development. Assuming we need to create a delivery pipeline, we need a more advanced tool.

Unit Testing of Helm Charts

From my perspective, the most important thing in the CI/CD pipeline is automated testing. Without it, we are releasing unverified software, which may potentially result in many complications. The single Helm chart can be by several apps, so we should make every effort to carefully test it. Fortunately, there are some tools dedicated to Helm chart testing. My choice fell on the helm-unittest. It allows us to write unit tests file in pure YAML. We can install it as a Helm plugin or run it inside the Docker container. Let’s just it locally to verify our test work before pushing it to the Git repository:

$ helm plugin install https://github.com/helm-unittest/helm-unittest

We should place the unit test inside the test directory in our chart. Here’s the structure of our chart repository:

In the first step, we are creating the unit test file. As mentioned before, we can create a test using the YAML notation. It is pretty intuitive. We need to pass a location of the values file (1) and a location of the tested Helm template (2). In the test section, we have to define a list of asserts (3). I will not get into the details of the helm-unittest tool – for more information please refer to its docs. The important thing is that I can easily test each path of the YAML manifest. It can be an exact comparison or regex. It also supports JsonPath for mappings and arrays. Here’s our test in the deployment_test.yaml:

suite: test deployment
values:
  - ./values/test.yaml # (1)
templates:
  - templates/deployment.yaml # (2)
chart:
  version: 0.3.4+test
  appVersion: 1.0.0
tests:
  - it: should pass all kinds of assertion
    template: templates/deployment.yaml
    documentIndex: 0
    asserts: # (3)
      - equal:
          path: spec.template.spec.containers[?(@.name == "sample-spring-boot-api")].image
          value: piomin/sample-spring-kotlin-microservice:1.1
      - equal:
          path: metadata.labels.app
          value: sample-spring-boot-api
      - equal:
          path: metadata.labels.env
          value: dev
      - equal:
          path: metadata.labels.owner
          value: default
      - matchRegex:
          path: metadata.name
          pattern: ^.*-api$
      - contains:
          path: spec.template.spec.containers[?(@.name == "sample-spring-boot-api")].ports
          content:
            containerPort: 8080
            name: http
      - notContains:
          path: spec.template.spec.containers[?(@.name == "sample-spring-boot-api")].ports
          content:
            containerPort: 80
      - isNotEmpty:
          path: spec.template.spec.containers[?(@.name == "sample-spring-boot-api")].livenessProbe
      - isNotEmpty:
          path: spec.template.spec.containers[?(@.name == "sample-spring-boot-api")].readinessProbe
      - isKind:
          of: Deployment
      - isAPIVersion:
          of: apps/v1

Now, we can verify the test locally by executing the following command from the project root directory:

$ helm unittest charts/*

Currently, we have only a single chart in the charts directory. Assuming we would have more, it runs the tests for all charts. Here’s our result:

We can change something in the test to break it. Now, the result of the same will look like that:

Helm Chart Release Pipeline in CircleCI

Once we have a chart and tests created we may proceed to the delivery pipeline. In our CircleCI pipeline, we have to do not only the same steps as before, but also need to include a release part. First of all, we will use GitHub Releases and GitHub Pages to release and host our charts. In order to simplify the process, we may use a dedicated tool for releasing Helm charts – Chart Releaser.

We also need to create a personal token to pass to the Helm Chart Release workflow. Visit Settings > Developer Settings > Personal Access Token. Generate Personal the token with the repo scope. Then, we should put this token into the CircleCI context. You can choose any name for the context, but the name of the environment variable has to be CR_TOKEN. That name is required by the Chart Releaser. The name of my context is GitHub.

Here’s the list of steps we need to do in our pipeline:

1. Install the helm CLI on the machine (we will use the cimg/base image as the tests executor)
2. Install the Helm unit-test plugin
3. Run unit tests
4. Only if we make a change in the master branch, then we proceed to the release part. In the first step, we need to package the chart with helm package command
5. Install Chart Releaser
6. Release the chart in GitHub with the Chart Releaser upload command
7. Generate chart index.yaml and publish it on GitHub Pages

Let’s define our CircleCI pipeline. First, we need to create the .circleci directory in the repository root and place the config.yml file there. We can use the helm orb to simplify the process of the helm CLI installation (1) (2). Once we install the helm CLI, we can install the unit-test plugin and run the unit tests (3). Then we define a rule for filtering the master branch (4). If the change is pushed to the master branch we package the chart as the TAR archive and place it in the .deploy directory (5). Then we install Chart Releaser and create a GitHub release (6). In the last step, we generate the index.yaml file using Chart Releaser and commit it to the gh-pages branch (7).

version: 2.1

orbs:
  helm: circleci/helm@2.0.1 # (1)

jobs:
  build:
    docker:
      - image: cimg/base:2023.02
    steps:
      - checkout
      - helm/install-helm-client # (2)
      - run:
          name: Install Helm unit-test
          command: helm plugin install https://github.com/helm-unittest/helm-unittest
      - run: # (3)
          name: Run unit tests
          command: helm unittest charts/*
      - when:
          condition: # (4)
            equal: [ master, << pipeline.git.branch >> ]
          steps:
            - run:
                name: Package chart # (5)
                command: helm package charts/* -d .deploy
            - run:
                name: Install chart releaser
                command: |
                  curl -L -o /tmp/cr.tgz https://github.com/helm/chart-releaser/releases/download/v1.5.0/chart-releaser_1.5.0_linux_amd64.tar.gz
                  tar -xv -C /tmp -f /tmp/cr.tgz
                  mv /tmp/cr ~/bin/cr
            - run:
                name: Release chart # (6)
                command: cr upload -o piomin -r helm-charts -p .deploy
            - run:
                name: Create index on GitHub pages # (7)
                command: |
                  git config user.email "job@circleci.com"
                  git config user.name "CircleCI"
                  git checkout --orphan gh-pages
                  cr index -i ./index.yaml -p .deploy -o piomin -r helm-charts
                  git add index.yaml
                  git commit -m "New release"
                  git push --set-upstream --force origin gh-pages

workflows:
  helm_test:
    jobs:
      - build:
          context: GitHub

Execute Helm Chart Release Pipeline

Once we push a change to the helm-charts repository our pipeline is starting. Here is our result. As you see the pipeline finishes with success. We were releasing the 0.3.5 version of our chart.

Let’s see a list of GitHub releases. As you see, the 0.3.5 version has already been released.

How to access our Helm repository. In order to check it go to the repository Settings > Pages. The address of the GitHub Pages for that repository is the address of our Helm repository. We publish there the index.yaml file that contains a definition of charts inside the repository. As you see, the address of the Helm repository is piomin.github.io/helm-charts.

We can see the structure of the index.yaml file just by calling the following URL: https://piomin.github.io/helm-charts/index.yaml. Here’s the fragment of index.yaml for the currently published version.

apiVersion: v1
entries:
  spring-boot-api-app:
  - apiVersion: v2
    appVersion: 1.0.0
    created: "2023-02-28T13:06:28.835693321Z"
    description: A Helm chart for Kubernetes
    digest: b308cbdf9f93f79baf5b39de8a7c509834d7b858f33d79d0c76b528e0cd7ca11
    name: spring-boot-api-app
    type: application
    urls:
    - https://github.com/piomin/helm-charts/releases/download/spring-boot-api-app-0.3.5/spring-boot-api-app-0.3.5.tgz
    version: 0.3.5

Assuming we want to use our Helm chart we can easily access it. First, let’s add the Helm repository using CLI:

$ helm repo add piomin https://piomin.github.io/helm-charts/

Then, we can verify a list of Helm charts existing inside the repository:

$ helm search repo piomin
NAME                            CHART VERSION   APP VERSION     DESCRIPTION                
piomin/spring-boot-api-app      0.3.5           1.0.0           A Helm chart for Kubernetes

Publish Helm Chart to Artifact Hub

In order to publish your Helm repository and charts on Artifact Hub you need to go to that site and create an account. Once you do it, you can add a new repository just by clicking the button. Then you just need to choose the name of your repo and put the right address.

Now, we can find our spring-boot-api-app chart on the list of packages.

We can see its details. It’s worth publishing documentation in the README.md file. Once you do it, you can view it in the chart details on Artifact Hub.

Finally, we can easily use the chart and deploy the Spring Boot app, e.g. with Argo CD.

The post Create and Release Your Own Helm Chart appeared first on Piotr's TechBlog.

It’s not my first article about Devtron. You can read more about the GitOps approach with Devtron in this article. Today, I’m going to focus more on the developer-friendly features around Helm charts support.

Install Devtron on Kubernetes

In the first step, we will install Devtron on Kubernetes. There are two options for installation: with CI/CD module or without it. We won’t build a CI/CD process today, but there are some important features for our scenario included in this module. Firstly, let’s add the Devtron Helm repository:

$ helm repo add devtron https://helm.devtron.ai

Then, we have to execute the following Helm command:

$ helm install devtron devtron/devtron-operator \
    --create-namespace --namespace devtroncd \
    --set installer.modules={cicd}

For detailed installation instructions please refer to the Devtron documentation available here.

Create Kubernetes Cluster with Kind

In order to prepare a multi-cluster environment on the local machine, we will use Kind. Let’s create the second Kubernetes cluster c1 by executing the following command:

$ kind create cluster --name c1

The second cluster is available as the kind-c1 context. It becomes a default context after you create a Kind cluster.

Now, our goal is to add the newly created Kind cluster as a managed cluster in Devtron. A single instance of Devtron can manage multiple Kubernetes clusters. Of course, by default, it just manages a local cluster. Before we add our Kind cluster to the Devtron dashboard, we should first configure privileges on that cluster. The following script will generate a bearer token for authentication purposes so that Devtron is able to communicate with the target cluster:

$ curl -O https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/kubernetes_export_sa.sh && bash kubernetes_export_sa.sh cd-user devtroncd https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/clusterrole.yaml

The bearer token is printed in the output of that command. Just copy it.

We will also have to provide an URL of the master API of a target cluster. Since I’m running Kubernetes on Kind I need to get an internal address of the Docker container that contains Kind. In order to obtain it we need to run the following command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' c1-control-plane

Here’s the IP address of my Kind cluster:

Now, we have all the required data to add a new managed cluster in the Devtron dashboard. In order to do that let’s navigate to the “Global Configuration” section. Then we need to choose the “Clusters and Environments” item and click the “Add cluster” button. We need to put the Kind cluster URL and previously generated bearer token.

If everything works fine, you should see the second cluster on the managed clusters list. Now, you also need to install the Devtron agent on Kind according to the message visible below:

Create Environments

In the next step, we will define three environments. In Devtron environment is assigned to the cluster. We will create a single environment on the local cluster (local), and another two on the Kind cluster (remote-dev, remote-devqa). Each environment has a target namespace. In order to simplify, the name of the namespace is the same as the name environment. Of course, you may set any names you want.

Now, let’s switch to the “Clusters” view.

As you see there are two clusters connected to Devtron:

We can take a look at the details of each cluster. Here you can see a detailed view for the kind-c1 cluster:

Add Custom Helm Repository

One of the most important Devtron features is support for Helm charts. We can deploy charts individually or by creating a group of charts. By default, there are several Helm repositories available in Devtron including bitnami or elastic. It is also possible to add a custom repository. That’s something that we are going to do. We have our own custom Helm repository with a chart for deploying the Spring Boot app. I have already published it on GitHub under the address https://piomin.github.io/helm-charts/. The name of our chart is spring-boot-api-app, and the latest version is 0.3.2.

In order to add the custom repository in Devtron, we need to go to the “Global Configurations” section once again. Then go to the “Chart repositories” menu item, and click the “Add repository” button. As you see below, I added a new repository under the name piomin.

Once you created a repository you can go to the “Chart Store” section to verify if the new chart is available.

Deploy the Spring Boot App with Devtron

Now, we can proceed to the most important part of our exercise – application deployment. Our sample Spring Boot app is available in the following repository on GitHub. It is a simple REST app written in Kotlin. It exposes some HTTP endpoints for adding and returning persons and uses an in-memory store. Here’s our Spring @RestController:

@RestController
@RequestMapping("/persons")
class PersonController(val repository: PersonRepository) {

   val log: Logger = LoggerFactory.getLogger(PersonController::class.java)

   @GetMapping("/{id}")
   fun findById(@PathVariable id: Int): Person? {
      log.info("findById({})", id)
      return repository.findById(id)
   }

   @GetMapping("/age/{age}")
   fun findByAge(@PathVariable age: Int): List<Person> {
      log.info("findByAge({})", age)
      return repository.findByAge(age)
   }

   @GetMapping
   fun findAll(): List<Person> = repository.findAll()

   @PostMapping
   fun add(@RequestBody person: Person): Person = repository.save(person)

   @PutMapping
   fun update(@RequestBody person: Person): Person = repository.update(person)

   @DeleteMapping("/{id}")
   fun remove(@PathVariable id: Int): Boolean = repository.removeById(id)

}

Let’s imagine we are just working on the latest version of that, and we want to deploy it on Kubernetes to perform some development tests. In the first step, we will build the app locally and push the image to the container registry using Jib Maven Plugin. Here’s the required configuration:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.3.0</version>
  <configuration>
    <to>
      <image>piomin/sample-spring-kotlin-microservice</image>
      <tags>
        <tag>1.1</tag>
      </tags>
    </to>
    <container>
      <user>999</user>
    </container>
  </configuration>
</plugin>

Let’s build and push the image to the container registry using the following command:

$ mvn clean compile jib:build -Pjib,tomcat

Besides YAML templates our Helm repository also contains a JSON schema for values.yaml validation. Thanks to that schema we would be able to take advantage of Devtron GUI for creating apps from the chart. Let’s see how it works. Once you click on our custom chart you will be redirected to the page with the details. The latest version of the chart is 0.3.2. Just click the Deploy button.

On the next page, we need to provide a configuration of our app. The target environment is local, which exists on the main cluster. Thanks to Devtron support for Helm values.schema.json we define all values using the GUI form. For example, we can increase change the value of the image to the latest – 1.1.

Once we deploy the app we may verify its status:

Let’s make some test calls. Our sample Spring Boot exposes Swagger UI, so we can easily send HTTP requests. To interact with the app running on Kubernetes we should enable port-forwarding for our service kubectl port-forward svc/sample-spring-boot-api 8080:8080. After executing that command you can access the Swagger UI under the address http://localhost:8080/swagger-ui.html.

Devtron allows us to view pod logs. We can “grep” them with our criteria. Let’s display the logs related to our test calls.

Deploy App to the Remote Cluster

Now, we will deploy our sample Spring Boot app to the remote cluster. In order to do that go to the same page as before, but instead of the local environment choose remote-dev. It is related to the kind-c1 cluster.

Now, there are two same applications running on two different clusters. We can do the same thing for the app running on the Kind cluster as for the local cluster, e.g. verify its status or check logs.

Deploy Group of Apps

Let’s assume we would like to deploy the app that connects to the database. We can do it in a single step using the Devtron feature called “Chart Group”. With that feature, we can place our Helm chart for Spring Boot and the chart for e.g. Postgres inside the same logical group. Then, we can just deploy the whole group into the target environment. In order to create a chart group go to the Chart Store menu and then click the “Create Group” button. You should set the name of the group and choose the charts that will be included. For me, these are bitnami/postgresql and my custom Helm chart.

After creating a group you will see it on the main “Chart Store” page. Now, just click on it to deploy the apps.

After you click the tile with the chart group, you will be predicted to the deploy page.

After you click the “Deploy to…” button Devtron will redirect you to the next page. You can set there a target project and environment for all member charts of the group. We will deploy them to the remote-devqa environment from the kind-c1 cluster. We can use the image from my Docker account: piomin/person:1.1. By default, it tries to connect to the database postgres on the postgres host. The only thing we need to inject into the app container is the postgres user password. It is available inside the postgresql Secret generated by the Bitnami Helm chart. To inject envs defined in that secret use the extraEnvVarsSecret parameter in our custom Spring Boot chart. Finally, let’s deploy both Spring Boot and Postgres in the remove-devqa namespace by clicking the “Deploy” button.

Here’s the final list of apps we have already deployed during this exercise:

Final Thoughts

With Devtron you can easily deploy applications across multiple Kubernetes clusters using Helm chart support. Devtron simplifies development on Kubernetes. You can deploy all required applications just with a “single click” with the chart group feature. Then you can manage and monitor them using a GUI dashboard. In general, you can do everything in the dashboard without passing any YAML manifests by yourself or executing kubectl commands.

The post Development on Kubernetes Multicluster with Devtron appeared first on Piotr's TechBlog.

With Datree you can automatically detect misconfigurations in Kubernetes manifests. Argo CD allows you to manage the continuous delivery process on Kubernetes declaratively using the GitOps approach. It can apply the manifests stored in a Git repository to a Kubernetes cluster. But, what about validation? Argo CD doesn’t have any built-in mechanisms for validating configuration taken from Git. This is where Datree comes in. Let’s see how to integrate both of these tools into your CD process.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that you need to clone my GitHub repository. This repository contains sample Helm charts, which Argo CD will use as Deployment templates. After that, you should just follow my instructions.

Introduction

We will deploy our sample image using two different Helm charts. Therefore, we will have two applications defined in ArgoCD per each Deployment. Our goal is to configure Argo CD to automatically use Datree as a Kubernetes manifest validation tool for all applications. To achieve that, we need to create the ArgoCD plugin for Datree. It is quite a similar approach to the case described in this article. However, there we used a ready plugin for integration between Argo CD and HashiCorp’s Vault.

There are two different ways to run a custom plugin in Argo CD. We can add the plugin config to the Argo CD main ConfigMap or run it as a sidecar to the argocd-repo-server Pod. In this article, we will use the first approach. This approach requires us to ensure that relevant binaries are available inside the argocd-repo-server pod. They can be added via volume mounts or using a custom image. We are going to create a custom image of the Argo CD that contains the Datree binaries.

Build a custom Argo CD that contains Datree

We will use the official Argo CD Helm chart for installation on Kubernetes. This chart is based on the following Argo CD image: quay.io/argoproj/argocd. In fact, the image is used by several components including argocd-server and argocd-repo-server. We are going to replace the default image for the argocd-repo-server:

In the first step, we will create a Dockerfile that extends a base image. We will install the Datree CLI there. Since Datree requires curl and unzip we also need to add them to the final image. Here’s our Dockerfile based on the latest version of the Argo CD image:

FROM quay.io/argoproj/argocd:v2.4.11

USER root

RUN apt-get update
RUN apt-get install -y curl
RUN apt-get install -y unzip
RUN apt-get clean
RUN curl https://get.datree.io | /bin/bash

LABEL release=1-with-curl

USER 999

I have already built and published the image on my Docker Hub account. You can pull it from the repository piomin/argocd under the v2.4.11-datree tag. If you would like to build it by yourself just run the following command:

$ docker build -t piomin/argocd:v2.4.11-datree .

Here’s the output:

Install Argo CD with the Datree Plugin on Kubernetes

We will use Helm to install Argo CD on Kubernetes. Instead of the default image, we need to use our custom Argo CD image built in the previous step. We also need to enable and configure our plugin in the Argo CD ConfigMap. You would also have to set the DATREE_TOKEN environment variable containing your Datree token. Fortunately, we can set all those things in a single place. Here’s our Helm values.yaml that overrides default configuration settings for Argo CD.

server:
  config:
    configManagementPlugins: |
      - name: datree
        generate:
          command: ["bash", "-c"]
          args: ['if [[ $(helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . > result.yaml | datree test -o json result.yaml) && $? -eq 0 ]]; then cat result.yaml; else exit 1; fi']

repoServer:
  image:
    repository: piomin/argocd
    tag: v2.4.11-datree
  env:
    - name: DATREE_TOKEN
      value: <YOUR_DATREE_TOKEN>

You can obtain the token from the Datree dashboard after login. Just go to your account settings:

Then, you need to add the Helm repository with Argo CD charts:

$ helm repo add argo https://argoproj.github.io/argo-helm

After that, just install it with the custom settings provided in values.yaml file:

$ helm install argocd argo/argo-cd \
    --version 5.1.0 \
    --values values.yaml \
    -n argocd

Once you installed Argo CD in the argocd namespace, you can display the argocd-cm ConfigMap.

$ kubectl get cm argocd-cm -n argocd -o yaml

The argocd-cm ConfigMap contains the definition of our plugin. Inside the args parameter, we need to pass the command for creating Deployment manifests. Each time we call the helm template command we pass the generated YAML file to the Datree CLI, which will run a policy check against it. If all of the rules pass the datree test command will return exit code 0. Otherwise, it will return a value other than 0. If the Datree policy check finishes successfully we need to send the YAML manifest to the output. Thanks to that, Argo CD will apply it to the Kubernetes cluster.

Create Helm charts

The instance of ArgoCD with a plugin for Datree is ready and we can now proceed to the app deployment phase. Firstly, let’s create Helm templates. Here’s a very basic Helm template for our sample app. It just exposes ports outside the container and sets environment variables. It is available under the simple-with-envs directory.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  replicas: {{ .Values.app.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
          ports:
          {{- range .Values.app.ports }}
            - containerPort: {{ .value }}
              name: {{ .name }}
          {{- end }}
          {{- if .Values.app.envs }}
          env:
          {{- range .Values.app.envs }}
            - name: {{ .name }}
              value: {{ .value }}
          {{- end }}
          {{- end }}

We can test it locally with Helm and Datree CLI. Here’s a set of test values:

image:
  registry: quay.io
  repository: pminkows/sample-kotlin-spring
  tag: "1.0"

app:
  name: sample-spring-boot-kotlin
  replicas: 1
  ports:
    - name: http
      value: 8080
  envs:
    - name: PASS
      value: example

The following command generates the YAML manifest using test-values.yaml and performs a Datree policy check.

$ helm template --values test-values.yaml . > result.yaml | \
  datree test result.yaml

Here’s the result of our test analysis. As you can see, there are a lot of violations reported by Datree:

For example, we should add liveness and readiness probes, and disable root access to the container. Here is another Helm template that fixes all the problems reported by Datree:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
  labels:
    app: {{ .Values.app.name }}
    env: {{ .Values.app.environment }}
    owner: {{ .Values.app.owner }}
spec:
  replicas: {{ .Values.app.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
        env: {{ .Values.app.environment }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
          resources:
            requests:
              memory: {{ .Values.app.resources.memoryRequest }}
              cpu: {{ .Values.app.resources.cpuRequest }}
            limits:
              memory: {{ .Values.app.resources.memoryLimit }}
              cpu: {{ .Values.app.resources.cpuLimit }}
          livenessProbe:
            initialDelaySeconds: {{ .Values.app.liveness.initialDelaySeconds }}
            httpGet:
              port: {{ .Values.app.liveness.port }}
              path: {{ .Values.app.liveness.path }}
            failureThreshold: {{ .Values.app.liveness.failureThreshold }}
            successThreshold: {{ .Values.app.liveness.successThreshold }}
            timeoutSeconds: {{ .Values.app.liveness.timeoutSeconds }}
            periodSeconds: {{ .Values.app.liveness.periodSeconds }}
          readinessProbe:
            initialDelaySeconds: {{ .Values.app.readiness.initialDelaySeconds }}
            httpGet:
              port: {{ .Values.app.readiness.port }}
              path: {{ .Values.app.readiness.path }}
            failureThreshold: {{ .Values.app.readiness.failureThreshold }}
            successThreshold: {{ .Values.app.readiness.successThreshold }}
            timeoutSeconds: {{ .Values.app.readiness.timeoutSeconds }}
            periodSeconds: {{ .Values.app.readiness.periodSeconds }}
          ports:
          {{- range .Values.app.ports }}
            - containerPort: {{ .value }}
              name: {{ .name }}
          {{- end }}
          {{- if .Values.app.envs }}
          env:
          {{- range .Values.app.envs }}
            - name: {{ .name }}
              value: {{ .value }}
          {{- end }}
          {{- end }}
          securityContext:
            runAsNonRoot: true

You can perform the same test as before for the new Helm chart. Just go to the full-compliant directory.

Create Argo CD Applications with Helm charts

Finally, we can create ArgoCD applications that use our Helm charts. Let’s create an app for the simple-with-envs chart (1). Instead of a typical helm type, we should set the plugin type (2). The name of our plugin is datree (3). With this type of ArgoCD app we have to set Helm parameters inside the HELM_VALUES environment variable (4).

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-failed
  namespace: argocd
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: simple-with-envs # (1)
    repoURL: 'https://github.com/piomin/sample-generic-helm-charts.git'
    targetRevision: HEAD
    plugin: # (2)
      name: datree # (3)
      env:
        - name: HELM_VALUES # (4)
          value: |
            image:
              registry: quay.io
              repository: pminkows/sample-kotlin-spring
              tag: "1.4.30"

            app:
              name: sample-spring-boot-kotlin
              replicas: 1
              ports:
                - name: http
                  value: 8080
              envs:
                - name: PASS
                  value: example
  project: default

Here’s the UI of our ArgoCD Application:

Let’s verify what happened:

ArgoCD performs a policy check using Datree. As you probably remember this Helm chart does not meet the rules defined in Datree. Therefore the process exited with error code 1.

Now, let’s create an ArgoCD Application for the second Helm chart:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: full-compliant-ok
  namespace: argocd
spec:
  destination:
    name: ''
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: full-compliant
    repoURL: 'https://github.com/piomin/sample-generic-helm-charts.git'
    targetRevision: HEAD
    plugin:
      name: datree
      env:
        - name: HELM_VALUES
          value: |
            image:
              registry: quay.io
              repository: pminkows/sample-kotlin-spring
              tag: "1.4.30"

            app:
              name: sample-spring-boot-kotlin
              replicas: 2
              environment: test
              owner: piomin
              resources:
                memoryRequest: 128Mi
                memoryLimit: 512Mi
                cpuRequest: 500m
                cpuLimit: 1
              liveness:
                initialDelaySeconds: 10
                port: 8080
                path: /actuator/health/liveness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
              readiness:
                initialDelaySeconds: 10
                port: 8080
                path: /actuator/health/readiness
                failureThreshold: 3
                successThreshold: 1
                timeoutSeconds: 3
                periodSeconds: 5
              ports:
                - name: http
                  value: 8080
              envs:
                - name: PASS
                  value: example
  project: default

This time the ArgoCD Application is created successfully:

That’s because Datree analysis finished successfully:

Finally, we can just synchronize the app to deploy our image on Kubernetes.

Final Thoughts

ArgoCD allows us to install custom plugins. Thanks to that we can integrate the ArgoCD deployment process with the Datree policy check, which is performed each time a new version of Kubernetes manifest is processed by ArgoCD. It applies to all the apps that refer to the plugin.

The post Continuous Delivery on Kubernetes with Argo CD and Datree appeared first on Piotr's TechBlog.

You can use Vault in several different ways on Kubernetes. For example, you may integrate it directly with your Spring Boot app using the Spring Cloud Vault project. To read more about it please refer to that post on my blog.

Prerequisites

In this exercise, we are going to use Helm a lot. Our sample app Deployment is based on the Helm template. We also use Helm to install Vault and ArgoCD on Kubernetes. Finally, we need to customize the ArgoCD Helm chart parameters to enable and configure ArgoCD Vault Plugin. So, before proceeding please ensure you have basic knowledge about Helm. Of course, you should also install it on your laptop. For me, it is possible with homebrew:

$ brew install helm

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

Run and Configure Vault on Kubernetes

In the first step, we are going to install Vault on Kubernetes. We can easily do it using its official Helm chart. In order to simplify our exercise, we run it in development mode with a single server instance. Normally, you would configure it in HA mode. Let’s add the Hashicorp helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

In order to enable development mode the parameter server.dev.enabled should have the value true. We don’t need to override any other default values:

$ helm install vault hashicorp/vault \
    --set "server.dev.enabled=true"

To check if Vault is successfully installed on the Kubernetes cluster we can display a list of running pods:

$ kubectl get pod 
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          25s
vault-agent-injector-9456c6d55-hx2fd   1/1     Running   0          21s

We can configure Vault in several different ways. One of the options is through the UI. To login there we may use the root token generated only in development mode. Vault also exposes the HTTP API. The last available option is with the CLI. CLI is available inside the Vault pod, but as well we can install it locally. For me, it is possible using the brew install vault command. Then we need to enable port-forwarding and export Vault local address as the VAULT_ADDR environment variable:

$ kubectl port-forward vault-0 8200
$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault status

Then just login to the Vault server using the root token:

Enable Kubernetes Authentication on Vault

There are several authentication methods on Vault. However, since we run it on Kubernetes we should the method dedicated to that platform. What is important, this method is also supported by the ArgoCD Vault Plugin. Firstly, let’s enable the Kubernetes Auth method:

$ vault auth enable kubernetes

Then, we need to configure our authentication method. There are three required parameters: the URL of the Kubernetes API server, Kubernetes CA certificate, and a token reviewer JWT.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=<your Kubernetes API address> \
    kubernetes_ca_cert=@ca.crt

In order to easily obtain all those parameters, you can the following three commands. Then you can set them also e.g. using the Vault UI.

$ echo "https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"
$ kubectl exec vault-0 \
  -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
$ echo $(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Once we pass all the required parameters, we may proceed to the named role creation. ArgoCD Vault Plugin will use that role to authenticate against the Vault server. We need to provide the namespace with ArgoCD and the name of the Kubernetes service account used by the ArgoCD Repo Server. Our token expires after 24 hours.

$ vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-repo-server \
  bound_service_account_namespaces=argocd \
  policies=argocd \
  ttl=24h

That’s all for now. We will also need to create a test secret on Vault and configure a policy for the argocd role. Before that, let’s take a look at our sample Spring Boot app and its Helm template.

Helm Template for Spring Boot App

Our app is very simple. It just exposes a single HTTP endpoint that returns the value of the environment variable inside a container. Here’s the REST controller class written in Kotlin.

@RestController
@RequestMapping("/persons")
class PersonController {

    @Value("\${PASS:none}")
    lateinit var pass: String

    @GetMapping("/pass")
    fun printPass() = pass

}

We will use a generic Helm chart to deploy our app on Kubernetes. Our Deployment template contains a list of environment variables defined for the container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  replicas: {{ .Values.app.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
          ports:
          {{- range .Values.app.ports }}
            - containerPort: {{ .value }}
              name: {{ .name }}
          {{- end }}
          {{- if .Values.app.envs }}
          env:
          {{- range .Values.app.envs }}
            - name: {{ .name }}
              value: {{ .value }}
          {{- end }}
          {{- end }}

There is also another file in the templates directory. It contains a definition of the Kubernetes Service.

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.app.name }}
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.app.name }}
  ports:
  {{- range .Values.app.ports }}
  - port: {{ .value }}
    name: {{ .name }}
  {{- end }}

Finally, let’s take a look at the Chart.yaml file.

apiVersion: v2
name: sample-with-envs
description: A Helm chart for Kubernetes
type: application
version: 1.0
appVersion: "1.0"

Our goal is to use this Helm chart to deploy the sample Spring Boot app on Kubernetes with ArgoCD and Vault. Of course, before we do it we need to install ArgoCD.

Install ArgoCD with Vault Plugin on Kubernetes

Normally, it would be a very simple installation. But this time we need to customize the ArgoCD template to install it with the Vault plugin. Or more precisely, we have to customize the configuration of the ArgoCD Repository Server. It is one of the ArgoCD internal services. It maintains the local cache of the Git repository and generates Kubernetes manifests.

There are some different options for installing Vault plugin on ArgoCD. The full list of options is available here. Starting with the version 2.4.0 of ArgoCD it is possible to install it via a sidecar container. We will choose the option based on sidecar and initContainer. You may read more about it here. However, our case would be different a little since we Helm instead of Kustomize for installing ArgoCD. To clarify, we need to do three things to install the Vault plugin on ArgoCD. Let’s analyze those steps:

• define initContainer on the ArgoCD Repository Server Deployment to download argocd-vault-plugin binaries and mount them on the volume
• define the ConfigMap containing the ConfigManagementPlugin CRD overriding a default behavior of Helm on ArgoCD
• customize argocd-repo-server Deployment to mount the volume with argocd-vault-plugin and the ConfigMap created in the previous step

After those steps, we would have to integrate the plugin with the running instance of the Vault server. We will use a previously create Vault argocd role.

Firstly, let’s create the ConfigMap to customize the default behavior of Helm on ArgoCD. After running the helm template command we will also run the argocd-vault-plugin generate command to replace all inline placeholders with the secrets defined in Vault. The address and auth configuration of Vault are defined in the vault-configuration secret.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s vault-configuration -
      lockRepo: false

Here’s the vault-configuration Secret:

apiVersion: v1
kind: Secret
metadata:
  name: vault-configuration
  namespace: argocd 
data:
  AVP_AUTH_TYPE: azhz
  AVP_K8S_ROLE: YXJnb2Nk
  AVP_TYPE: dmF1bHQ=
  VAULT_ADDR: aHR0cDovL3ZhdWx0LmRlZmF1bHQ6ODIwMA==
type: Opaque

To see the values let’s display the Secret in Lens. Vault is running in the default namespace, so its address is http://vault.default:8200. The name of our role in Vault is argocd. We also need to set the auth type as k8s.

Finally, we need to customize the ArgoCD Helm installation. To achieve that let’s define the Helm values.yaml file. It contains the definition of initContainer and sidecar for argocd-repo-server. We also mount the cmp-plugin ConfigMap to the Deployment, and add additional privileges to the argocd-repo-service ServiceAccount to allow reading Secrets.

repoServer:
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.11.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}

In order to install ArgoCD on Kubernetes add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

Let’s install it in the argocd namespace using customized parameters in the values.yaml file:

$ kubectl create ns argocd
$ helm install argocd argo/argo-cd -n argocd -f values.yaml

Sync Vault Secrets with ArgoCD

Once we deployed Vault and ArgoCD on Kubernetes we may proceed to the next step. Now, we are going to create a secret on Vault. Firstly, let’s enable the KV engine:

$ vault secrets enable kv-v2

Then, we can create a sample secret with the argocd name and a single password key:

$ vault kv put kv-v2/argocd password="123456"

ArgoCD Vault Plugin uses the argocd policy to read secrets. So, in the next step, we need to create the following policy to enable reading the previously created secret:

$ vault policy write argocd - <<EOF
path "kv-v2/data/argocd" {
  capabilities = ["read"]
}
EOF

Then, we may define the ArgoCD Application for deploying our Spring Boot app on Kubernetes. The Helm template for Kubernetes manifests is available on the GitHub repository under the simple-with-envs directory (1). As a tool for creating manifests we choose plugin (2). However, we won’t set its name since we use a sidecar container with argocd-vault-plugin. ArgoCD Vault plugin allows passing inline values in the application manifest. It reads the content defined inside the HELM_VALUES environment variable (3) (depending on the environment variable name set inside cmp-plugin ConfigMap). And finally, the most important thing. ArgoCD Vault Plugin is looking for placeholders inside the <> brackets. For inline values, it should have the following structure: <path:path_to_the_vault_secret#name_of_the_key> (4). In our case, we define the environment variable PASS that uses the argocd secret and the password key stored inside the KV engine.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-helm
  namespace: argocd
spec:
  destination:
    name: ''
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: simple-with-envs # (1)
    repoURL: https://github.com/piomin/sample-generic-helm-charts.git 
    targetRevision: HEAD
    plugin: # (2)
      env:
        - name: HELM_VALUES # (3)
          value: |
            image:
              registry: quay.io
              repository: pminkows/sample-kotlin-spring
              tag: "1.4.30"

            app:
              name: sample-spring-boot-kotlin
              replicas: 1
              ports:
                - name: http
                  value: 8080
              envs:
                - name: PASS
                  value: <path:kv-v2/data/argocd#password> # (4)

Finally, we can create the ArgoCD Application. It should have the OutOfSync status:

Let’s synchronize its state with the Git repository. We can do it using e.g. ArgoCD UI. If everything works fine you should see the green tile with your application name.

Then, let’s just verify the structure of our app Deployment. You should see the value 123456 instead of the placeholder defined inside the ArgoCD Application.

It is just a formality, but in the end, you can test the endpoint GET /persons/pass exposed by our Spring Boot app. It prints the value of the PASS environment variable. To do that you should also enable port-forwarding for the app.

$ kubectl port-forward svc/simple-helm 8080:8080
$ curl http://localhost:8080/persons/pass

Final Thoughts

GitOps approach becomes very popular in a Kubernetes-based environment. As always, one of the greatest challenges with that approach is security. Hashicorp Vault is one of the best tools for managing and protecting sensitive data. It can be easily installed on Kubernetes and included in your GitOps process. In this article, I showed how to use it together with other very popular solutions for deploying apps: ArgoCD and Helm.

The post Manage Secrets on Kubernetes with ArgoCD and Vault appeared first on Piotr's TechBlog.