Argo CD ApplicationSet supports several different generators. In this article, we will focus on the Git generator type. It generates Argo CD Applications based on directory structure or branch changes in a Git repository. It contains two subtypes: the Git directory generator and the Git file generator. If you are interested in other Argo CD ApplicationSet generators you can find some articles on my blog. For example, the following post shows how to use List Generator to promote images between environments. You can also find a post about the Cluster Decision Resource generator, which shows how to spread applications dynamically between multiple Kubernetes clusters.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. You must go to the appset-helm-demo directory, which contains the whole configuration required for that exercise. Then you should only follow my instructions.

Argo CD Installation

Argo CD is the only tool we need to install on our Kubernetes cluster for that exercise. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

helm install my-argo-cd argo/argo-cd -n argocd

I use OpenShift in that exercise. With the OpenShift Console, I can easily install ArgoCD on the cluster using the OpenShift GitOps operator.

Once we installed it we can easily access the Argo CD dashboard.

We can sign in there using OpenShift credentials.

Motivation

Our goal in this exercise is to deploy and run some applications (a simple Java app and Postgres database) on Kubernetes with minimal source code effort. Those two applications only show how to create a standard that can be easily applied to any application type deployed on our cluster. In this standard, a directory structure determines how and where our applications are deployed on Kubernetes. My example configuration is stored in a single Git repository. However, we can easily extend it with multiple repositories, where Argoc CD switches between the central repository and other Git repositories containing a configuration for concrete applications.

Here’s a directory structure and files for deploying our two applications. Both the custom app and Postgres database are deployed in three environments: dev, test, and prod. We use Helm charts for deploying them. Each environment directory contains a Helm values file with installation parameters. The configuration distinguishes two different types of installation: apps and components. Each app is installed using the same Helm chart dedicated to a standard deployment. Each component is installed using a custom Helm chart provided by that component. For example, for Postgres, we will use the following Bitnami chart.

.
├── apps
│   ├── aaa-1
│   │   └── basic
│   │       ├── prod
│   │       │   └── values.yaml
│   │       ├── test
│   │       │   └── values.yaml
│   │       ├── uat
│   │       │   └── values.yaml
│   │       └── values.yaml
│   ├── aaa-2
│   └── aaa-3
└── components
    └── aaa-1
        └── postgresql
            ├── prod
            │   ├── config.yaml
            │   └── values.yaml
            ├── test
            │   ├── config.yaml
            │   └── values.yaml
            └── uat
                ├── config.yaml
                └── values.yaml

Before deploying the application, we should prepare namespaces with quotas, Argo CD projects, and ApplicationSet generators for managing application deployments. Here’s the structure of a global configuration repository. It also uses Helm chart to apply that part of manifests to the Kubernetes cluster. Each directory inside the projects directory determines our project name. On the other hand, a project contains several Kubernetes namespaces. Each project may contain several different Kubernetes Deployments.

.
└── projects
    ├── aaa-1
    │   └── values.yaml
    ├── aaa-2
    │   └── values.yaml
    └── aaa-3
        └── values.yaml

Prepare Global Cluster Configuration

Helm Template for Namespaces and Quotas

Here’s the Helm template for creating namespaces and quotas for each namespace. We will create the project namespace per each environment (stage).

{{- range .Values.stages }}
---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ $.Values.projectName }}-{{ .name }}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: {{ $.Values.projectName }}-{{ .name }}
spec:
  hard:
    {{- if .config }}
    {{- with .config.quotas }}
    pods: {{ .pods | default "10" }}
    requests.cpu: {{ .cpuRequest | default "2" }}
    requests.memory: {{ .memoryRequest | default "2Gi" }}
    limits.cpu: {{ .cpuLimit | default "8" }}
    limits.memory: {{ .memoryLimit | default "8Gi" }}
    {{- end }}
    {{- else }}
    pods: "10"
    requests.cpu: "2"
    requests.memory: "2Gi"
    limits.cpu: "8"
    limits.memory: "8Gi"
    {{- end }}
{{- end }}

Helm Template for the Argo CD AppProject

Helm chart will also create a dedicated Argo CD AppProject object per our project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: {{ .Values.projectName }}
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'

Helm Template for Argo CD ApplicationSet

After that, we can proceed to the most tricky part of our exercise. Helm chart also defines a template for creating the Argo CD ApplicationSet. This ApplicationSet must analyze the repository structure, which contains the configuration of apps and components. We define two ApplicationSets per each project. The first uses the Git Directory generator to determine the structure of the apps catalog and deploy the apps in all environments using my custom spring-boot-api-app chart. The chart parameters can be overridden with Helm values placed in each app directory.

The second ApplicationSet uses the Git Files generator to determine the structure of the components catalog. It reads the contents of the config.yaml file in each directory. The config.yaml file sets the repository, name, and version of the Helm chart that must be used to install the component on Kubernetes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-apps-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          {{- range .Values.stages }}
          - path: appset-helm-demo/apps/{{ $.Values.projectName }}/*/{{ .name }}
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: spring-boot-api-app
          repoURL: 'https://piomin.github.io/helm-charts/'
          targetRevision: 0.3.8
          helm:
            valueFiles:
              - $values/appset-helm-demo/apps/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-components-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        files:
          {{- range .Values.stages }}
          - path: appset-helm-demo/components/{{ $.Values.projectName }}/*/{{ .name }}/config.yaml
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: '{{`{{ .chart.name }}`}}'
          repoURL: '{{`{{ .chart.repository }}`}}'
          targetRevision: '{{`{{ .chart.version }}`}}'
          helm:
            valueFiles:
              - $values/appset-helm-demo/components/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

There are several essential elements in this configuration, which we should pay attention to. Both Helm and ApplicationSet use templating engines based on {{ ... }} placeholders. So to avoid conflicts we should escape Argo CD ApplicationSet templating elements from the Helm templating elements. The following part of the template responsible for generating the Argo CD Application name is a good example of that approach: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'. First, we use the AppliocationSet Git generator parameter index .path.segments 3 that returns the name of the third part of the directory path. Those elements are escaped with the ` char so Helm doesn’t try to analyze it.

Helm Chart Structure

Our ApplicationSets use the “Multiple Sources for Application” feature to read parameters from Helm values files and inject them into the Helm chart from a remote repository. Thanks to that, our configuration repositories for apps and components contain only values.yaml files in the standardized directory structure. The only chart we store in the sample repository has been described above and is responsible for creating the configuration required to run app Deployments on the cluster.

.
└── chart
    ├── Chart.yaml
    ├── templates
    │   ├── additional.yaml
    │   ├── applicationsets.yaml
    │   ├── appproject.yaml
    │   └── namespaces.yaml
    └── values.yaml

By default, each project defines three environments (stages): test, uat, prod.

stages:
  - name: test
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

We can override a default behavior for the specific project in Helm values. Each project directory contains the values.yaml file. Here are Helm parameters for the aaa-3 project that override CPU request quota from 2 CPUs to 4 CPUs only for the test environment.

stages:
  - name: test
    config:
      quotas:
        cpuRequest: 4
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}

Run the Synchronization Process

Generate Global Structure on the Cluster

To start a process we must create the ApplicationSet that reads the structure of the projects directory. Each subdirectory in the projects directory indicates the name of our project. Our ApplicationSet uses a Git directory generator to create an Argo CD Application per each project. Its name contains the name of the subdirectory and the config suffix. Each generated Application uses the previously described Helm chart to create all namespaces, quotas, and other resources requested by the project. It also leverages the “Multiple Sources for Application” feature to allow us to override default Helm chart settings. It reads a project name from the directory name and passes it as a parameter to the generated Argo CD Application.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-config
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          - path: appset-helm-demo/projects/*
  template:
    metadata:
      name: '{{.path.basename}}-config'
    spec:
      destination:
        namespace: '{{.path.basename}}'
        server: 'https://kubernetes.default.svc'
      project: default
      sources:
        - path: appset-helm-demo/chart
          repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          helm:
            valueFiles:
              - $values/appset-helm-demo/projects/{{.path.basename}}/values.yaml
            parameters:
              - name: projectName
                value: '{{.path.basename}}'
              - name: argoNamespace
                value: openshift-gitops
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

Once we create the global-config ApplicationSet object magic happens. Here’s the list of Argo CD Applications generated from our directories in Git configuration repositories.

First, there are three Argo CD Applications with the projects’ configuration. That’s happening because we defined 3 subdirectories in the projects directory with names aaa-1, aaa-2 and aaa-3.

The configuration applied by those Argo CD Applications is pretty similar since they are using the same Helm chart. We can look at the list of resources managed by the aaa-3-config Application. There are three namespaces (aaa-3-test, aaa-3-uat, aaa-3-prod) with resource quotas, a single Argo CD AppProject, and two ApplicationSet objects responsible for generating Argo CD Applications for apps and components directories.

In this configuration, we can verify if the value of the request.cpu ResourceQuota object has been overridden from 2 CPUs to 4 CPUs.

Let’s analyze what happened. Here’s a list of Argo CD ApplicationSets. The global-config ApplicationSet generated Argo CD Application per each detected project inside the projects directory. Then, each of these Applications applied two ApplicationSet objects to cluster using the Helm template.

$ kubectl get applicationset
NAME                      AGE
aaa-1-components-config   29m
aaa-1-apps-config         29m
aaa-2-components-config   29m
aaa-2-apps-config         29m
aaa-3-components-config   29m
aaa-3-apps-config         29m
global-config             29m

There’s also a list of created namespaces:

$ kubectl get ns
NAME                                               STATUS   AGE
aaa-1-prod                                         Active   34m
aaa-1-test                                         Active   34m
aaa-1-uat                                          Active   34m
aaa-2-prod                                         Active   34m
aaa-2-test                                         Active   34m
aaa-2-uat                                          Active   34m
aaa-3-prod                                         Active   34m
aaa-3-test                                         Active   34m
aaa-3-uat                                          Active   34m

Generate and Apply Deployments

Our sample configuration contains only two Deployments. We defined the basic subdirectory in the apps directory and the postgres subdirectory in the components directory inside the aaa-1 project. The aaa-2 and aaa-3 projects don’t contain any Deployments for simplification. However, the more subdirectories with the values.yaml file we create there, the more applications will be deployed on the cluster. Here’s a typical values.yaml file for a simple app deployed with a standard Helm chart. It defines the image repository, name, and tag. It also set the Deployment name and environment.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: prod

For the postgres component we must set more parameters in Helm values. Here’s the final list:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

The following Argo CD Application has been generated by the aaa-1-apps-config ApplicationSet. It detected the basic subdirectory in the apps directory. The basic subdirectory contained 3 subdirectories: test, uat and prod with values.yaml file. As a result, we have Argo CD per environment responsible for deploying the basic app in the target namespaces.

Here’s a list of resources managed by the basic-prod Application. It uses my custom Helm chart and applies Deployment and Service objects to the cluster.

The following Argo CD Application has been generated by the aaa-1-components-config ApplicationSet. It detected the basic subdirectory in the components directory. The postgres subdirectory contained 3 subdirectories: test, uat and prod with values.yaml and config.yaml files. The ApplicationSet Files generator reads the repository, name, and version from the configuration in the config.yaml file.

Here’s the config.yaml file with the Bitnami Postgres chart settings. We could place here any other chart we want to install something else on the cluster.

chart:
  repository: https://charts.bitnami.com/bitnami
  name: postgresql
  version: 15.5.38

Here’s the list of resources installed by the Bitnami Helm chart used by the generated Argo CD Applications.

Final Thoughts

This article proves that Argo CD ApplicationSet and Helm templates can be used together to create advanced configuration structures. It shows how to use ApplicationSet Git Directory and Files generators to analyze the structure of directories and files in the Git config repository. With that approach, we can propose a standardization in the configuration structure across the whole organization and propagate it similarly for all the applications deployed in the Kubernetes clusters. Everything can be easily managed at the cluster admin level with the single global Argo CD ApplicationSet that accesses many different repositories with configuration.

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. It is the simple Spring Boot application we are going to deploy on OpenShift. Then you should just follow my instructions.

Before we begin with OpenShift odo

It is important to understand some key concepts related to odo before we begin. If you want to deploy a new application you should create a component. Each component can be run and deployed separately. There are two types of components: Devfile and S2I. The S2I component is based on the Source-To-Image process. Therefore, your application is built on the server-side with the S2I builder. With the Devfile component, you run the application with the mvn spring-boot:run command. Here’s the list of available components. We will use the java component.

You can also check out the list of services by executing the command odo catalog list services. A service is a software that your component links to or depends on. However, I won’t focus on that feature. Currently, the Service Catalog is deprecated in OpenShift. Instead, you can operators with odo the same as you would use templates.

The Spring Boot application

Let’s take a moment on discussing our sample Spring Boot application. It is a REST-based application, which connects to MongoDB. We will use JDK 11 for compilation.

<groupId>pl.piomin.samples</groupId>
<artifactId>sample-spring-boot-on-kubernetes</artifactId>
<version>1.0-SNAPSHOT</version>

<properties>
    <java.version>11</java.version>
</properties>

<dependencies>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-web</artifactId>
   </dependency>
   <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-data-mongodb</artifactId>
   </dependency>
   ...
</dependencies>

The MongoDB connection settings are set inside application.yml. We will use environment variables for getting credentials and database name. We will set these values in the local configuration using odo.

spring:
  application:
    name: sample-spring-boot-on-kubernetes
  data:
    mongodb:
      uri: mongodb://${MONGO_USERNAME}:${MONGO_PASSWORD}@mongodb/${MONGO_DATABASE}

Creating configuration with odo

Ok, let’s by creating a new component. We need to choose java S2I type and then set a name of our component.

$ odo create java sample-spring-boot

After that odo is creating a file config.yaml inside the .odo directory in the project root catalog. It’s pretty simple.

kind: LocalConfig
apiversion: odo.dev/v1alpha1
ComponentSettings:
  Type: java
  SourceLocation: ./
  SourceType: local
  Ports:
  - 8443/TCP
  - 8778/TCP
  - 8080/TCP
  Application: app
  Project: pminkows-workshop
  Name: sample-spring-boot

Then, we will add some additional configuration properties. Firstly, we need to expose our application outside the cluster. The command visible below creates a Route to the Service on OpenShift on the 8080 port.

$ odo url create --port 8080

In the next step, we add environment variables to the odo configuration responsible for establishing MongoDB connection: MONGO_USERNAME, MONGO_PASSWORD, and MONGO_DATABASE. Just to simplify, I created a standalone instance of MongoDB on OpenShift using the template. Here are the values in the mongodb secret.

Let’s set environment variables using odo config command. Of course, all things are still available only in the local configuration.

$ odo config set --env MONGO_USERNAME=userJ5Q
$ odo config set --env MONGO_PASSWORD=UrfgtUKohNOFVqbQ
$ odo config set --env MONGO_DATABASE=sampledb 

Finally, we are ready to deploy our application on OpenShift. To do that, we need to execute odo push command. If you would like to see the logs from the build add option --show-log.

Instead of using environment variables to provide MongoDB connection settings, you may take advantage of odo link command. This feature helps to connect an odo component to a service or another component. However, to use it you first need to install the Service Binding Operator on OpenShift. Then you may install a database like MongoDB with an operator and link to your application component.

Verify S2I build on OpenShift

After running odo push let’s switch to oc client. Our application is running on OpenShift as shown below. The same with MongoDB.

We can also navigate to the OpenShift Management Console. The environment variables set inside odo configuration has been applied to the DeploymentConfig.

Now, let’s say we need to provide some changes in our code. Fortunately, we can execute the command that watches for changes in the directory for current component. After detecting such a change, the new version of application is immediately deployed on OpenShift.

$ odo watch
Waiting for something to change in /Users/pminkows/IdeaProjects/sample-spring-boot-on-kubernetes

Using OpenShift odo in Devfile mode

In opposition to the S2I approach, we may use Devfile mode. In order to do that we will choose the java-springboot component.

As a result, OpenShift odo creates devfile.yaml in the project root directory.

schemaVersion: 2.0.0
metadata:
  name: java-springboot
  version: 1.1.0
starterProjects:
  - name: springbootproject
    git:
      remotes:
        origin: "https://github.com/odo-devfiles/springboot-ex.git"
components:
  - name: tools
    container:
      image: quay.io/eclipse/che-java11-maven:nightly
      memoryLimit: 768Mi
      mountSources: true
      endpoints:
      - name: '8080-tcp'
        targetPort: 8080
      volumeMounts:
        - name: m2
          path: /home/user/.m2
  - name: m2
    volume:
      size: 3Gi
commands:
  - id: build
    exec:
      component: tools
      commandLine: "mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true"
      group:
        kind: build
        isDefault: true
  - id: run
    exec:
      component: tools
      commandLine: "mvn -Dmaven.repo.local=/home/user/.m2/repository spring-boot:run"
      group:
        kind: run
        isDefault: true
  - id: debug
    exec:
      component: tools
      commandLine: "java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=${DEBUG_PORT},suspend=n -jar target/*.jar"
      group:
        kind: debug
        isDefault: true

Of course, all next steps are the same as for S2I component.

Install OpenShift Intellij Plugin

If you do not like command-line tools, you may install the Intellij OpenShift plugin as well. It uses odo for build and deploy. The good news is that the latest version of this plugin supports odo in 2.0.3 version.

Thanks to that plugin you can for example easily create component with OpenShift odo.

Conclusion

With odo you can easily deploy your application on OpenShift in a few seconds. You may also continuously watch for changes in the code, and immediately deploy a new version of application. Moreover, you don’t need to create any Kubernetes YAML manifests.

OpenShift odo is a similar tool to Skaffold. If you would like to compare both these tools used to run the Spring Boot application you may read the following article about Skaffold.

The post Java Development on OpenShift with odo appeared first on Piotr's TechBlog.