It’s not my first article about Devtron. You can read more about the GitOps approach with Devtron in this article. Today, I’m going to focus more on the developer-friendly features around Helm charts support.

Install Devtron on Kubernetes

In the first step, we will install Devtron on Kubernetes. There are two options for installation: with CI/CD module or without it. We won’t build a CI/CD process today, but there are some important features for our scenario included in this module. Firstly, let’s add the Devtron Helm repository:

$ helm repo add devtron https://helm.devtron.ai

Then, we have to execute the following Helm command:

$ helm install devtron devtron/devtron-operator \
    --create-namespace --namespace devtroncd \
    --set installer.modules={cicd}

For detailed installation instructions please refer to the Devtron documentation available here.

Create Kubernetes Cluster with Kind

In order to prepare a multi-cluster environment on the local machine, we will use Kind. Let’s create the second Kubernetes cluster c1 by executing the following command:

$ kind create cluster --name c1

The second cluster is available as the kind-c1 context. It becomes a default context after you create a Kind cluster.

Now, our goal is to add the newly created Kind cluster as a managed cluster in Devtron. A single instance of Devtron can manage multiple Kubernetes clusters. Of course, by default, it just manages a local cluster. Before we add our Kind cluster to the Devtron dashboard, we should first configure privileges on that cluster. The following script will generate a bearer token for authentication purposes so that Devtron is able to communicate with the target cluster:

$ curl -O https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/kubernetes_export_sa.sh && bash kubernetes_export_sa.sh cd-user devtroncd https://raw.githubusercontent.com/devtron-labs/utilities/main/kubeconfig-exporter/clusterrole.yaml

The bearer token is printed in the output of that command. Just copy it.

We will also have to provide an URL of the master API of a target cluster. Since I’m running Kubernetes on Kind I need to get an internal address of the Docker container that contains Kind. In order to obtain it we need to run the following command:

$ docker inspect -f '{{range.NetworkSettings.Networks}}{{.IPAddress}}{{end}}' c1-control-plane

Here’s the IP address of my Kind cluster:

Now, we have all the required data to add a new managed cluster in the Devtron dashboard. In order to do that let’s navigate to the “Global Configuration” section. Then we need to choose the “Clusters and Environments” item and click the “Add cluster” button. We need to put the Kind cluster URL and previously generated bearer token.

If everything works fine, you should see the second cluster on the managed clusters list. Now, you also need to install the Devtron agent on Kind according to the message visible below:

Create Environments

In the next step, we will define three environments. In Devtron environment is assigned to the cluster. We will create a single environment on the local cluster (local), and another two on the Kind cluster (remote-dev, remote-devqa). Each environment has a target namespace. In order to simplify, the name of the namespace is the same as the name environment. Of course, you may set any names you want.

Now, let’s switch to the “Clusters” view.

As you see there are two clusters connected to Devtron:

We can take a look at the details of each cluster. Here you can see a detailed view for the kind-c1 cluster:

Add Custom Helm Repository

One of the most important Devtron features is support for Helm charts. We can deploy charts individually or by creating a group of charts. By default, there are several Helm repositories available in Devtron including bitnami or elastic. It is also possible to add a custom repository. That’s something that we are going to do. We have our own custom Helm repository with a chart for deploying the Spring Boot app. I have already published it on GitHub under the address https://piomin.github.io/helm-charts/. The name of our chart is spring-boot-api-app, and the latest version is 0.3.2.

In order to add the custom repository in Devtron, we need to go to the “Global Configurations” section once again. Then go to the “Chart repositories” menu item, and click the “Add repository” button. As you see below, I added a new repository under the name piomin.

Once you created a repository you can go to the “Chart Store” section to verify if the new chart is available.

Deploy the Spring Boot App with Devtron

Now, we can proceed to the most important part of our exercise – application deployment. Our sample Spring Boot app is available in the following repository on GitHub. It is a simple REST app written in Kotlin. It exposes some HTTP endpoints for adding and returning persons and uses an in-memory store. Here’s our Spring @RestController:

@RestController
@RequestMapping("/persons")
class PersonController(val repository: PersonRepository) {

   val log: Logger = LoggerFactory.getLogger(PersonController::class.java)

   @GetMapping("/{id}")
   fun findById(@PathVariable id: Int): Person? {
      log.info("findById({})", id)
      return repository.findById(id)
   }

   @GetMapping("/age/{age}")
   fun findByAge(@PathVariable age: Int): List<Person> {
      log.info("findByAge({})", age)
      return repository.findByAge(age)
   }

   @GetMapping
   fun findAll(): List<Person> = repository.findAll()

   @PostMapping
   fun add(@RequestBody person: Person): Person = repository.save(person)

   @PutMapping
   fun update(@RequestBody person: Person): Person = repository.update(person)

   @DeleteMapping("/{id}")
   fun remove(@PathVariable id: Int): Boolean = repository.removeById(id)

}

Let’s imagine we are just working on the latest version of that, and we want to deploy it on Kubernetes to perform some development tests. In the first step, we will build the app locally and push the image to the container registry using Jib Maven Plugin. Here’s the required configuration:

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.3.0</version>
  <configuration>
    <to>
      <image>piomin/sample-spring-kotlin-microservice</image>
      <tags>
        <tag>1.1</tag>
      </tags>
    </to>
    <container>
      <user>999</user>
    </container>
  </configuration>
</plugin>

Let’s build and push the image to the container registry using the following command:

$ mvn clean compile jib:build -Pjib,tomcat

Besides YAML templates our Helm repository also contains a JSON schema for values.yaml validation. Thanks to that schema we would be able to take advantage of Devtron GUI for creating apps from the chart. Let’s see how it works. Once you click on our custom chart you will be redirected to the page with the details. The latest version of the chart is 0.3.2. Just click the Deploy button.

On the next page, we need to provide a configuration of our app. The target environment is local, which exists on the main cluster. Thanks to Devtron support for Helm values.schema.json we define all values using the GUI form. For example, we can increase change the value of the image to the latest – 1.1.

Once we deploy the app we may verify its status:

Let’s make some test calls. Our sample Spring Boot exposes Swagger UI, so we can easily send HTTP requests. To interact with the app running on Kubernetes we should enable port-forwarding for our service kubectl port-forward svc/sample-spring-boot-api 8080:8080. After executing that command you can access the Swagger UI under the address http://localhost:8080/swagger-ui.html.

Devtron allows us to view pod logs. We can “grep” them with our criteria. Let’s display the logs related to our test calls.

Deploy App to the Remote Cluster

Now, we will deploy our sample Spring Boot app to the remote cluster. In order to do that go to the same page as before, but instead of the local environment choose remote-dev. It is related to the kind-c1 cluster.

Now, there are two same applications running on two different clusters. We can do the same thing for the app running on the Kind cluster as for the local cluster, e.g. verify its status or check logs.

Deploy Group of Apps

Let’s assume we would like to deploy the app that connects to the database. We can do it in a single step using the Devtron feature called “Chart Group”. With that feature, we can place our Helm chart for Spring Boot and the chart for e.g. Postgres inside the same logical group. Then, we can just deploy the whole group into the target environment. In order to create a chart group go to the Chart Store menu and then click the “Create Group” button. You should set the name of the group and choose the charts that will be included. For me, these are bitnami/postgresql and my custom Helm chart.

After creating a group you will see it on the main “Chart Store” page. Now, just click on it to deploy the apps.

After you click the tile with the chart group, you will be predicted to the deploy page.

After you click the “Deploy to…” button Devtron will redirect you to the next page. You can set there a target project and environment for all member charts of the group. We will deploy them to the remote-devqa environment from the kind-c1 cluster. We can use the image from my Docker account: piomin/person:1.1. By default, it tries to connect to the database postgres on the postgres host. The only thing we need to inject into the app container is the postgres user password. It is available inside the postgresql Secret generated by the Bitnami Helm chart. To inject envs defined in that secret use the extraEnvVarsSecret parameter in our custom Spring Boot chart. Finally, let’s deploy both Spring Boot and Postgres in the remove-devqa namespace by clicking the “Deploy” button.

Here’s the final list of apps we have already deployed during this exercise:

Final Thoughts

With Devtron you can easily deploy applications across multiple Kubernetes clusters using Helm chart support. Devtron simplifies development on Kubernetes. You can deploy all required applications just with a “single click” with the chart group feature. Then you can manage and monitor them using a GUI dashboard. In general, you can do everything in the dashboard without passing any YAML manifests by yourself or executing kubectl commands.

The post Development on Kubernetes Multicluster with Devtron appeared first on Piotr's TechBlog.

If you are interested in more tips about CI/CD on Kubernetes you may also read my article about Tekton and Argo CD.

Prerequisites

Of course, you need a running Kubernetes cluster to start. The best way to install Devtron on Kubernetes is by using Helm. I won’t get into the details. You can find instructions in the Devtron materials here. Once you install it on your cluster you can display a list of running pods in the devtroncd namespace. There are a lot of tools there, but the most important for us are Argo CD and, of course, Devtron.

$ kubectl get pod -n devtroncd

Since there are a lot of apps, you should have sufficient resources for your Kubernetes cluster. I have 12GB of memory intended for the cluster and everything works perfectly fine on the local machine. The first step is to access the Devtron dashboard. For me, it is available at the localhost, and port 80. You can check what is your address by executing the following command:

$ kubectl get svc -n devtroncd devtron-service \
  -o jsonpath='{.status.loadBalancer.ingress}'

Then, you need to log in as an administrator. The default username is admin. In order to obtain a password, you need to display the secret devtron-secret.

$ kubectl -n devtroncd get secret devtron-secret \
  -o jsonpath='{.data.ACD_PASSWORD}' | base64 -d

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. After that, you can follow my instructions.

Configure GitOps on Devtron

Before we start with the application pipeline, we need to configure some things on GitHub and Devtron. I’m using my public account on GitHub. Firstly, we need to create an organization pinned to our GitHub account. The name of my organization for this demo is piomin-devtron-test.

We also need to generate a personal access token on GitHub. In order to do that, go to Settings, then Developer Settings, and Personal access tokens. You should click the button Generate new token. Devtron requires to have write access to the account repositories because it creates a new repository and makes changes there.

Once you did that, you can configure access to the GitHub organization in your Devtron dashboard. Go to Global Configurations in the left-side menu, and then choose the GitOps tab. Finally, you can provide all the required settings as shown below.

Then switch to another tab in the current menu – Clusters & Environments. We will add three environments for our sample app. Like the exercise, we will promote it from the test environment to the stage environment, and finally to the production.

Build Delivery Pipeline with Devtron

Our pipeline consists of five steps. The first two of them are related to a CI process. We need to clone the Git repository and build the image from a Dockerfile. After that, we are deploying the image automatically to the test environment. The promotion to the higher environments (stage, prod) required manual approval. Here’s the screen from the Devtron dashboard that illustrates our pipeline.

We can easily define each pipeline deployment step in Devtron. We need to set the target environment, namespace, and deployment strategy.

In order to switch from the automatic deployment to the manual approval, we need to go to the Advanced Options. In production, I’m also changing the default deployment strategy to the CANARY release.

Deployment Template and Configuration

Let’s take a brief look at our sample Spring Boot application. As I mentioned before, it connects to a Mongo database and exposes API over HTTP. The address and database connection credentials are available for the app as environment variables. There are four variables configured in the Spring application.yml: MONGO_URL, MONGO_USERNAME, MONGO_PASSWORD, MONGO_DATABASE. The default web port is 8080. However, we are also going to expose port 8081 for the management endpoints. It includes health checks or metrics. We will use those endpoints for configuring liveness and readiness endpoints on Kubernetes. Additionally, we may expose a health check under the main web port 8080. Here’s the configuration of our Spring Boot app in the application.yml file:

spring:
  application:
    name: sample-spring-boot-on-kubernetes
  data:
    mongodb:
      host: ${MONGO_URL}
      port: 27017
      username: ${MONGO_USERNAME}
      password: ${MONGO_PASSWORD}
      database: ${MONGO_DATABASE}
      authentication-database: admin

management:
  endpoints:
    web:
      exposure:
        include: "*"
  endpoint.health:
      show-details: always
      group:
        readiness:
          include: mongo
          additional-path: server:/readiness
      probes:
        enabled: true
  server:
    port: 8081

With Spring Boot, we can expose some basic information about the app as an HTTP endpoint. It includes e.g. a version from Maven pom.xml. Then, we will use that information in our tests after releasing a new version of the app. To enable it, we need to include a build-info execution goal for the Spring Boot Maven Plugin:

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
    <executions>
      <execution>
        <goals>
          <goal>build-info</goal>
        </goals>
      </execution>
    </executions>
  <configuration>
</plugin>

Devtron simplifies GitOps for applications running on Kubernetes. After creating a new application with Devtron we will use an example template for deployment. In this template, we do not define Kubernetes objects directly, but just configure the behavior of our deployment. We need to expose both HTTP ports 8080 and 8081 outside the app, define liveness and readiness probes, configure resource limits, and a number of replicas. Here’s the full template for our sample deployment:

ContainerPort:
  - name: app
    port: 8080
    servicePort: 80
  - name: mgmt
    port: 8081
    servicePort: 81
LivenessProbe:
  Path: /actuator/health/liveness
  command: []
  failureThreshold: 3
  initialDelaySeconds: 30
  periodSeconds: 10
  port: 8081
  successThreshold: 1
  tcp: false
  timeoutSeconds: 5
ReadinessProbe:
  Path: /readiness
  command: []
  failureThreshold: 3
  initialDelaySeconds: 30
  periodSeconds: 10
  port: 8080
  successThreshold: 1
  tcp: false
  timeoutSeconds: 5
replicaCount: 2
resources:
  limits:
    cpu: "0.5"
    memory: 512Mi
  requests:
    cpu: "0.05"
    memory: 256Mi
server:
  deployment:
    image: ""
    image_tag: 1-95af053
service:
  type: ClusterIP

In the next step, we should inject environment variables with database address and credentials into the app. Once again, we can easily do it with Devtron. Firstly, go to the Secrets tab inside the App Configuration. Then click the Add Secret button. You can choose any name you want. For me it is mongo-secret.

Inside that secret, we just need to provide a list of environment variables with values.

How Devtron uses Argo CD and GitOps

The whole magic happens in the background. Once you create a configuration in the Devtron dashboard, it automatically creates a Git repository with YAML manifests. The name of the repository corresponds to the name application created in Devtron.

Devtron uses Helm as a tool for manifests templates. Argo CD supports Helm.

Devtron also creates an Argo CD application for each of the defined environments in the devtroncd namespace. Here’s the YAML manifest with that CRD object:

apiVersion: argoproj.io/v1alpha1
kind: Application
name: spring-boot-on-kubernetes-test
namespace: devtroncd
spec:
  destination:
    namespace: test
    server: 'https://kubernetes.default.svc'
  project: default
  source:
    helm:
      valueFiles:
        - _2-values.yaml
    path: reference-chart_4-11-0/4.11.1
    repoURL: >-
      https://github.com/piomin-devtron-test/devtron-spring-boot-on-kubernetes.git
    targetRevision: HEAD
  syncPolicy:
    automated:
      prune: true
    retry:
      backoff:
        duration: 5s
        factor: 2
        maxDuration: 5s
      limit: 1

Of course, we can access the Argo CD Dashboard to see the list of applications. Since we defined three environments in Devtron, there are three Argo CD applications. Each of them is for a particular environment.

If you use Devtron, in fact, you don’t have to know anything about the Argo CD instance. You can do all the required steps to deploy the application just by clicking everything in the Devtron dashboard. The goal of that section was to show you how Devtron automatically integrates with Argo CD to manage deployment according to the GitOps pattern.

Release a new version of the application

Let’s release and deploy a new version of our Spring Boot application on Kubernetes. To do that I will just change the number of the version in Maven pom.xml and push it to the remote repository. This change is made in the application repository. The current version is 1.3-SNAPSHOT.

A build in Devtron starts automatically after detecting a new push in the application repository. Then a new version of the app is automatically deployed on the test environment.

No matter which type of a deployment strategy (e.g. ROLLING or CANARY) we choose, Devtron creates the Argo CD Rollout object to run the app on Kubernetes. We can see the whole object by running the following command:

$ kubectl get rollout spring-boot-on-kubernetes-test -n test -o yaml

According to the configuration, there is also a Service ClusterIP created. To perform a simple check, let’s first enable port forwarding for our service:

$ kubectl port-forward service/spring-boot-on-kubernetes-test-service 8081:81 -n test

Then we can call the GET /actuator/info endpoint to display the current version of our sample application:

$ curl http://localhost:8081/actuator/info | json_pp
{
   "build" : {
      "group" : "pl.piomin.samples",
      "version" : "1.3-SNAPSHOT",
      "time" : "2022-04-25T14:41:42.473Z",
      "name" : "sample-spring-boot-on-kubernetes",
      "artifact" : "sample-spring-boot-on-kubernetes"
   }
}

Coming back to our pipeline, deploy the latest version of our application to the stage environment. Since we set a manual approval, we need to select the image to deploy. Devtron allows you to choose between previous images deployed to the current environment and the image deployed to the test environment. We will use the latest version that has already been deployed in the test namespace.

Finally, we can repeat the same step for the prod environment. It will take some time since we have 4 replicas on production and CANARY release enabled. Devtron tries to run a pod with a new version in 2-minute intervals as shown below.

Here’s the current view of our pipeline.

A canary release is possible thanks to the Argo Rollouts. Let’s take a look at the Rollout object created for the prod environment.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: spring-boot-on-kubernetes-prod
  namespace: prod
spec:
  minReadySeconds: 60
  replicas: 4
  revisionHistoryLimit: 3
  selector:
    matchLabels:
      app: spring-boot-on-kubernetes
      release: spring-boot-on-kubernetes-prod
  strategy:
    canary:
      maxSurge: 25%
      maxUnavailable: 1
      stableService: spring-boot-on-kubernetes-prod-service
      steps:
        - setWeight: 25
        - pause:
            duration: 15
        - setWeight: 50
        - pause:
            duration: 15
        - setWeight: 75
        - pause:
            duration: 15

Configuration per environment

And the last thing in our GitOps exercise with Devtron. Since we need more replicas in production than in other environments, we had to create a different deployment template. With Devtron we can easily override deployment templates for any environment. In the app configuration tab, you need to access the Environment Overrides section. Then just choose a particular environment and create a specific template there.

Final Thoughts

Devtron greatly simplifies the CI/CD process on Kubernetes. It provides a UI for building and managing pipelines. Thanks to that you may have GitOps and pipelines working together smoothly, where you can e.g. deploy only a version already deployed on the previous environment. It automates a lot of things, like creating a Git repository or Helm-based templates. For more information about Devtron, you can access the project’s Git repository here.

The post Getting Started with GitOps on Kubernetes with Devtron appeared first on Piotr's TechBlog.