Today, we are going to design the CI/CD process for building and deploying the app created in the previous article on Azure. In order to configure required services like Azure Spring Apps or Cosmos DB automatically we will use Terraform. We will use Azure DevOps and Azure Pipelines to build and deploy the app.

Preparation

For the purpose of that exercise, we need to provision an account on Azure and another one on the Azure DevOps platform. Once you install the az CLI and log in to Azure you can execute the following command for verification:

az account show

In the next step, you should set up the account on the Azure DevOps platform. In order to do it, go to the following site and click the “Start free” button or “Sign in” if you already have an account there. After that, you should see the Azure DevOps main page. Before we start, we need to create the organization and project. I’m using the “pminkows” name in both cases.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. The Spring Boot app used in the article is located in the microservices/account-service directory. You will also find the Terraform manifest inside the microservice/terraform directory and the azure-pipelines.yml file in the repository root directory. After you go to that directory you should just follow my further instructions.

Create Azure Resources with Terraform

Terraform is a great tool for defining resources according to the “Infrastructure as a code” approach. An official Terraform provider is allowing to configure infrastructure on Azure with the Azure Resource Manager API’s. In order to use it, we need to include the azurerm provider in the Terraform manifest. We will put all the required objects into the spring-group resource group.

terraform {
  required_version = ">= 1.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.3.0"
    }
  }
}

provider "azurerm" {
  features {}
}

resource "azurerm_resource_group" "spring-group" {
  location = "eastus"
  name     = "spring-apps"
}

Configure the Azure Cosmos DB Service

In the first step, we are going to configure the Cosmos DB instance required by our sample Spring Boot app. It requires a new database account (1). The name of my account is sample-pminkows-cosmosdb. It is placed inside our sample-spring-cloud resource group. We also need to define a default consistency level (consistency_policy) and a replication policy (geo_location). Once we enable a database account we can create a database instance (2). The name of our database is sampledb. Of course, it has to be placed in the previously created sample-pminkows-cosmosdb Cosmos DB account. Finally, we need to create a container inside our database (3). The name of the container should be the same as the value of the containerName field declared in the model class. We also have to set the partition key path. It corresponds to the name of the field inside the model class annotated with @PartitionKey.

# (1)
resource "azurerm_cosmosdb_account" "sample-db-account" {
  name                = "sample-pminkows-cosmosdb"
  location            = azurerm_resource_group.spring-group.location
  resource_group_name = azurerm_resource_group.spring-group.name
  offer_type          = "Standard"

  consistency_policy {
    consistency_level = "Session"
  }

  geo_location {
    failover_priority = 0
    location          = "eastus"
  }
}

# (2)
resource "azurerm_cosmosdb_sql_database" "sample-db" {
  name                = "sampledb"
  resource_group_name = azurerm_cosmosdb_account.sample-db-account.resource_group_name
  account_name        = azurerm_cosmosdb_account.sample-db-account.name
}

# (3)
resource "azurerm_cosmosdb_sql_container" "sample-db-container" {
  name                  = "accounts"
  resource_group_name   = azurerm_cosmosdb_account.sample-db-account.resource_group_name
  account_name          = azurerm_cosmosdb_account.sample-db-account.name
  database_name         = azurerm_cosmosdb_sql_database.sample-db.name
  partition_key_paths   = ["/customerId"]
  partition_key_version = 1
  throughput            = 400
}

Just for the record, here’s a model Java class inside our sample Spring Boot app:

@Container(containerName = "accounts")
public class Account {
   @Id
   @GeneratedValue
   private String id;
   private String number;
   @PartitionKey
   private String customerId;

   // GETTERS AND SETTERS ...
}

Install the Azure App Configuration Service

In the next step, we need to enable the Azure App Configuration service and put some properties into the store (1). The name of that instance is sample-spring-cloud-config. Our sample Spring Boot app uses the Spring Cloud Azure project to interact with the App Configuration service. In order to take advantage of that integration, we need to give proper names to all the configuration keys. They should be prefixed with the /application. They should also contain the name of the property automatically recognized by Spring Cloud. In our case, these are the properties used for establishing a connection with the Cosmos DB instance. We need to define three Spring Cloud properties: spring.cloud.azure.cosmos.key (2), spring.cloud.azure.cosmos.database (3), and spring.cloud.azure.cosmos.endpoint (4). We can retrieve the value of the Comso DB instance primary key or endpoint URL from the previously created azurerm_cosmosdb_account resource.

# (1)
resource "azurerm_app_configuration" "sample-config" {
  name                = "sample-spring-cloud-config"
  resource_group_name = azurerm_resource_group.spring-group.name
  location            = azurerm_resource_group.spring-group.location
}

# (2)
resource "azurerm_app_configuration_key" "cosmosdb-key" {
  configuration_store_id = azurerm_app_configuration.sample-config.id
  key                    = "/application/spring.cloud.azure.cosmos.key"
  value                  = azurerm_cosmosdb_account.sample-db-account.primary_key
}

# (3)
resource "azurerm_app_configuration_key" "cosmosdb-key" {
  configuration_store_id = azurerm_app_configuration.sample-config.id
  key                    = "/application/spring.cloud.azure.cosmos.database"
  value                  = "sampledb"
}

# (4)
resource "azurerm_app_configuration_key" "cosmosdb-key" {
  configuration_store_id = azurerm_app_configuration.sample-config.id
  key                    = "/application/spring.cloud.azure.cosmos.endpoint"
  value                  = azurerm_cosmosdb_account.sample-db-account.endpoint
}

Create the Azure Spring Apps Instance

In the last step, we need to configure the Azure Spring Apps service instance used for running Spring Boot apps. The name of our instance is sample-spring-cloud-apps (1). We will enable tracing for the Spring Azure Apps instance with the Application Insights service (2). After that, we will create a single app inside sample-spring-cloud-apps with the account-service name (3). This app requires a basic configuration containing an amount of requested resources, a version of Java runtime, or some environment variables including the address of the Azure App Configuration service instance. All those things should be set inside the deployment object represented by the azurerm_spring_cloud_java_deployment resource (4).

resource "azurerm_application_insights" "spring-insights" {
  name                = "spring-insights"
  location            = azurerm_resource_group.spring-group.location
  resource_group_name = azurerm_resource_group.spring-group.name
  application_type    = "web"
}

# (1)
resource "azurerm_spring_cloud_service" "spring-cloud-apps" {
  name                = "sample-spring-cloud-apps"
  location            = azurerm_resource_group.spring-group.location
  resource_group_name = azurerm_resource_group.spring-group.name
  sku_name            = "S0"

  # (2)
  trace {
    connection_string = azurerm_application_insights.spring-insights.connection_string
    sample_rate       = 10.0
  }

  tags = {
    Env = "Staging"
  }
}

# (3)
resource "azurerm_spring_cloud_app" "account-service" {
  name                = "account-service"
  resource_group_name = azurerm_resource_group.spring-group.name
  service_name        = azurerm_spring_cloud_service.spring-cloud-apps.name

  identity {
    type = "SystemAssigned"
  }
}

# (4)
resource "azurerm_spring_cloud_java_deployment" "slot-staging" {
  name                = "dep1"
  spring_cloud_app_id = azurerm_spring_cloud_app.account-service.id
  instance_count      = 1
  jvm_options         = "-XX:+PrintGC"
  runtime_version     = "Java_17"

  quota {
    cpu    = "500m"
    memory = "1Gi"
  }

  environment_variables = {
    "Env" : "Staging",
    "APP_CONFIGURATION_CONNECTION_STRING": azurerm_app_configuration.sample-config.primary_read_key[0].connection_string
  }
}

resource "azurerm_spring_cloud_active_deployment" "dep-staging" {
  spring_cloud_app_id = azurerm_spring_cloud_app.account-service.id
  deployment_name     = azurerm_spring_cloud_java_deployment.slot-staging.name
}

Apply the Terraform Manifest to Azure

Our Terraform configuration is ready. Finally, we can apply it to the target Azure account. Go to the microservices/terraform directory and then run the following commands:

$ terraform init
$ terraform apply -auto-approve

It can take several minutes until the command finishes. In the end, you should have a similar result. Terraform created 15 resources on Azure successfully.

We can switch to the Azure Portal for a moment. Let’s take a look at a list of resources inside our spring-apps resource group. As you see, all the required resources including Cosmos DB, App Configuration, and Azure Spring Apps are ready.

Build And Deploy the App with Azure Pipelines

After preparing the required infrastructure on Azure, we may proceed to the creation of a CI/CD pipeline for the app. Assuming you have already logged in to the Azure DevOps portal, you should find the “Pipelines” item on the left-side menu. Once you expand it, you should see several options.

Create Environment

Let’s start with the “Environments”. We will prepare just a single staging environment as shown below. We don’t need to choose any resources now (the “None” option).

Thanks to environments we can add approval checks for our pipelines. In order to do it, you should go to your environment details and switch to the “Approvals and checks” tab. There are several different available options. Let’s choose a simple approval, which requires someone to manually approve running the particular stage of the pipeline.

After clicking the “Next” button, you will be redirected to the next page containing a list of approvers. We can set a single person responsible for it or a whole group. For me, it doesn’t matter since I have only one user in the project. After defining a list of approvers click the “Create” button.

Define the Azure Pipeline

Now, let’s switch to the “Pipelines” view. We can create a pipeline manually with a GUI editor or just provide the azure-pipelines.yml file in the repository root directory. Of course, a GUI editor is also creating and committing the YAML manifest with a pipeline definition to the Git repository.

Let’s analyze our pipeline step by step. It is triggered by the commit into the master branch in the Git repository (1). We choose a standard agent pool (2). Our pipeline consists of two stages: Build_Test and Deploy_Stage (3). In the Build_Test stage we are building the app with Maven (4) and publishing the JAR file to the Azure Artifacts Feeds (5). Thanks to that we will be able to use that artifact in the next stage.

The next Deploy_Stage stage (6) waits until the previous stage is finished successfully (7). However, it won’t continue until we do a review and approve the pipeline. In order to do that, the job must refer to the previously defined staging environment (8) that contains the approval check. Once we approve the pipeline it proceeds to the step responsible for downloading artifacts from the Azure Artifacts Feeds (9). After that, it starts a deployment process (10). We need to use the AzureSpringCloud task responsible for deploying to the Azure Spring Apps service.

The deployment task requires several inputs. We need to set the Azure subscription ID (11), the ID of the Azure Spring Apps instance (12), the name of the app inside the Azure Spring Apps (13), and the name of a target deployment slot (14). Finally, we are setting the path to the JAR file downloaded in the previous step of the whole job (15). The pipeline reads the values of the Azure subscription ID and Azure Spring Apps instance ID from the input variables: subscription and serviceName.

# (1)
trigger:
- master

# (2)
pool:
  vmImage: ubuntu-latest

# (3)
stages:
- stage: Build_Test
  jobs:
  - job: Maven_Package
    steps:
    - task: MavenAuthenticate@0
      inputs:
        artifactsFeeds: 'pminkows'
        mavenServiceConnections: 'pminkows'
      displayName: 'Maven Authenticate'
    # (4)
    - task: Maven@3
      inputs:
        mavenPomFile: 'microservices/account-service/pom.xml'
        mavenOptions: '-Xmx3072m'
        javaHomeOption: 'JDKVersion'
        jdkVersionOption: '1.17'
        jdkArchitectureOption: 'x64'
        publishJUnitResults: true
        testResultsFiles: '**/surefire-reports/TEST-*.xml'
        goals: 'deploy'
        mavenAuthenticateFeed: true # (5)
      displayName: 'Build'

# (6)
- stage: Deploy_Stage
  dependsOn: Build_Test
  condition: succeeded() # (7)
  jobs:
    - deployment: Deployment_Staging
      environment:
        name: staging # (8) 
      strategy:
        runOnce:
          deploy:
            steps:
            # (9)
            - task: DownloadPackage@1
              inputs:
                packageType: 'maven'
                feed: 'pminkows'
                view: 'Local'
                definition: 'pl.piomin:account-service'
                version: '1.0'
                downloadPath: '$(System.ArtifactsDirectory)'
            - script: 'ls -la $(System.ArtifactsDirectory)' 
            # (10)
            - task: AzureSpringCloud@0
              inputs:
                azureSubscription: $(subscription) # (11)
                Action: 'Deploy'
                AzureSpringCloud: $(serviceName) # (12)
                AppName: 'account-service' # (13)
                DeploymentName: dep1 # (14)
                Package: '$(System.ArtifactsDirectory)/account-service-1.0.jar' # (15)

Run the Azure Pipeline

Let’s import our pipeline into the Azure DevOps platform. Azure DevOps provides a simple wizard for that. We need to choose the Git repository containing the pipeline definition.

After selecting the repository we will see the review page. We can change the definition of our pipeline taken from the azure-pipelines.yml file. If there is no need for any changes, we may add some variables or run (and save) the pipeline.

However, before running the pipeline we should define the required variables. The serviceName variable needs to contain the fully qualified ID of the Azure Spring Apps resource, e.g. /subscriptions/d4cde383-3611-4557-b2b1-b64b50378c9d/resourceGroups/spring-apps/providers/Microsoft.AppPlatform/Spring/sample-spring-cloud-apps.

We also need to create the Azure Artifact Feed. The pipeline uses it to cache and store artifacts during the Maven build. We should go to the “Artifacts” section. Then click the “Create Feed” button. The name of my feed is pminkows.

Once we run the pipeline, it will publish the app artifact to the target feed. The Maven group ID and its name determine the artifact’s name. The current version number is 1.0.

Let’s run the pipeline. It is starting from the build phase.

After finishing the build phase successfully, it proceeds to the deployment phase. However, the pipeline requires us to perform a review and approve the movement to the next step.

We need to click the Deploy_Stage tile. After that, you should see a similar approval screen as shown below. You can approve or reject the changes.

After approval, the pipeline starts the deployment phase. After around one minute it should deploy our app into the target Azure Spring Apps instance. Here’s the successfully finished run of the pipeline.

We can switch to the Azure Portal once again. Go to the sample-spring-cloud-apps Azure Spring Apps instance, then choose “Apps” and “account-service”. Finally, go to the “Deployments” section and choose the dep1. It is the deployment slot used by our pipeline. As you see, our app is running in the staging environment.

Final Thoughts

This article shows the holistic approach to app deployment on Azure. We can use Terraform to define all the resources and services required by the app. After that, we can define the CI/CD pipeline with Azure DevOps. As a result, we have a fully automated way of managing all the aspects related to our Spring Boot app running in the Azure cloud.

The post Azure DevOps and Terraform for Spring Boot appeared first on Piotr's TechBlog.

Let me clarify some things before we begin. In this article, I’m not promoting or recommending Terraform as the best tool for managing OpenShift or Kubernetes clusters at scale. Usually, I prefer the GitOps approach for that. If you are interested in how to leverage such tools like ACM (Advanced Cluster Management for Kubernetes) and Argo CD for managing multiple clusters with the GitOps approach read that article. It describes the idea of a cluster continuous management. From my perspective, Terraform fits better for one-time actions, like for example, creating and configuring OpenShift for the demo or PoC and then removing it. We can also use Terraform to install Argo CD and then delegate all the next steps there.

Anyway, let’s focus on our scenario. We will widely use those two Terraform providers: Azure and Kubernetes. So, it is worth at least taking a look at the documentation to familiarize yourself with the basics.

Prerequisites

Of course, you don’t have to perform that exercise on Azure with ARO. If you already have OpenShift running you can skip the part related to the cluster creation and just run the Terraform script responsible for installing operators and apps. For the whole exercise, you need to install:

1. Azure CLI (instructions) – once you install the login to your Azure account and create the subscription. To check if all works run the following command: az account show
2. Terraform CLI (instructions) – once you install the Terraform CLI you can verify it with the following command: terraform version

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then you should follow my instructions

Terraform Providers

The Terraform scripts for cluster creation are available inside the aro directory, while the script for cluster configuration inside the servicemesh directory. Here’s the structure of our repository:

Firstly, let’s take a look at the list of Terraform providers used in our exercise. In general, we need providers to interact with Azure and OpenShift through the Kubernetes API. In most cases, the official Hashicorp Azure Provider for Azure Resource Manager will be enough (1). However, in a few cases, we will have to interact directly with Azure REST API (for example to create an OpenShift cluster object) through the azapi provider (2). The Hashicorp Random Provider will be used to generate a random domain name for our cluster (3). The rest of the providers allow us to interact with OpenShift. Once again, the official Hashicorp Kubernetes Provider is valid in most cases (4). We will also use the kubectl provider (5) and Helm for installing the Postgres database (6) used by the sample apps.

terraform {
  required_version = ">= 1.0"
  required_providers {
    // (1)
    azurerm = {
      source  = "hashicorp/azurerm"
      version = ">=3.3.0"
    }
    // (2)
    azapi = {
      source  = "Azure/azapi"
      version = ">=1.0.0"
    }
    // (3)
    random = {
      source = "hashicorp/random"
      version = "3.5.1"
    }
    local = {
      source = "hashicorp/local"
      version = "2.4.0"
    }
  }
}

provider "azurerm" {
  features {}
}

provider "azapi" {
}

provider "random" {}
provider "local" {}

Here’s the list of providers used in the Re Hat Service Mesh installation:

terraform {
  required_version = ">= 1.0"
  required_providers {
    // (4)
    kubernetes = {
      source = "hashicorp/kubernetes"
      version = "2.23.0"
    }
    // (5)
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.13.0"
    }
    // (6)
    helm = {
      source = "hashicorp/helm"
      version = "2.11.0"
    }
  }
}

provider "kubernetes" {
  config_path = "aro/kubeconfig"
  config_context = var.cluster-context
}

provider "kubectl" {
  config_path = "aro/kubeconfig"
  config_context = var.cluster-context
}

provider "helm" {
  kubernetes {
    config_path = "aro/kubeconfig"
    config_context = var.cluster-context
  }
}

In order to install providers, we need to run the following command (you don’t have to do it now):

$ terraform init

Create Azure Red Hat OpenShift Cluster with Terraform

Unfortunately, there is no dedicated, official Terraform provider for creating OpenShift clusters on Azure ARO. There are some discussions about such a feature (you can find it here), but still without a final effect. Maybe it will change in the future. However, creating an ARO cluster is not such a complicated thing since we may use existing providers listed in the previous section. You can find an interesting guide in the Microsoft docs here. It was also a starting point for my work. I improved several things there, for example, to avoid using the az CLI in the scripts and have the full configuration in Terraform HCL.

Let’s analyze our Terraform manifest step by step. Here’s a list of the most important elements we need to place in the HCL file:

1. We have to read some configuration data from the Azure client
2. I have an existing resource group with the openenv prefix, but you can put there any name you want. That’s our main resource group
3. ARO requires a different resource group than a main resource group
4. We need to create a virtual network for Openshift. There is a dedicated subnet for master nodes and another one for worker nodes. All the parameters visible there are required. You can change the IP address range as long as it doesn’t allow for conflicts between the master and worker nodes
5. ARO requires the dedicated service principal to create a cluster. Let’s create the Azure application, and then the service principal with the password. The password is auto-generated by Azure.
6. The newly created service principal requires some privileges. Let’s assign the “User Access Administrator” and network “Contributor”. Then, we need to search the service principal created by Azure under the “Azure Red Hat OpenShift RP” name and also assign a network “Contributor” there.
7. All the required objects have already been created. There is no dedicated resource for the ARO cluster. In order to define the cluster resource we need to leverage the azapi provider.
8. The definition of the OpenShift cluster is available inside the body section. All the fields you see there are required to successfully create the cluster.

// (1)
data "azurerm_client_config" "current" {}
data "azuread_client_config" "current" {}

// (2)
data "azurerm_resource_group" "my_group" {
  name = "openenv-${var.guid}"
}

resource "random_string" "random" {
  length           = 10
  numeric          = false
  special          = false
  upper            = false
}

// (3)
locals {
  resource_group_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/resourceGroups/aro-${random_string.random.result}-${data.azurerm_resource_group.my_group.location}"
  domain            = random_string.random.result
}

// (4)
resource "azurerm_virtual_network" "virtual_network" {
  name                = "aro-vnet-${var.guid}"
  address_space       = ["10.0.0.0/22"]
  location            = data.azurerm_resource_group.my_group.location
  resource_group_name = data.azurerm_resource_group.my_group.name
}
resource "azurerm_subnet" "master_subnet" {
  name                 = "master_subnet"
  resource_group_name  = data.azurerm_resource_group.my_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes     = ["10.0.0.0/23"]
  service_endpoints    = ["Microsoft.ContainerRegistry"]
  private_link_service_network_policies_enabled  = false
  depends_on = [azurerm_virtual_network.virtual_network]
}
resource "azurerm_subnet" "worker_subnet" {
  name                 = "worker_subnet"
  resource_group_name  = data.azurerm_resource_group.my_group.name
  virtual_network_name = azurerm_virtual_network.virtual_network.name
  address_prefixes     = ["10.0.2.0/23"]
  service_endpoints    = ["Microsoft.ContainerRegistry"]
  depends_on = [azurerm_virtual_network.virtual_network]
}

// (5)
resource "azuread_application" "aro_app" {
  display_name = "aro_app"
  owners       = [data.azuread_client_config.current.object_id]
}
resource "azuread_service_principal" "aro_app" {
  application_id               = azuread_application.aro_app.application_id
  app_role_assignment_required = false
  owners                       = [data.azuread_client_config.current.object_id]
}
resource "azuread_service_principal_password" "aro_app" {
  service_principal_id = azuread_service_principal.aro_app.object_id
}

// (6)
resource "azurerm_role_assignment" "aro_cluster_service_principal_uaa" {
  scope                = data.azurerm_resource_group.my_group.id
  role_definition_name = "User Access Administrator"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
resource "azurerm_role_assignment" "aro_cluster_service_principal_network_contributor_pre" {
  scope                = data.azurerm_resource_group.my_group.id
  role_definition_name = "Contributor"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
resource "azurerm_role_assignment" "aro_cluster_service_principal_network_contributor" {
  scope                = azurerm_virtual_network.virtual_network.id
  role_definition_name = "Contributor"
  principal_id         = azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}
data "azuread_service_principal" "aro_app" {
  display_name = "Azure Red Hat OpenShift RP"
  depends_on = [azuread_service_principal.aro_app]
}
resource "azurerm_role_assignment" "aro_resource_provider_service_principal_network_contributor" {
  scope                = azurerm_virtual_network.virtual_network.id
  role_definition_name = "Contributor"
  principal_id         = data.azuread_service_principal.aro_app.id
  skip_service_principal_aad_check = true
}

// (7)
resource "azapi_resource" "aro_cluster" {
  name      = "aro-cluster-${var.guid}"
  parent_id = data.azurerm_resource_group.my_group.id
  type      = "Microsoft.RedHatOpenShift/openShiftClusters@2023-07-01-preview"
  location  = data.azurerm_resource_group.my_group.location
  timeouts {
    create = "75m"
  }
  // (8)
  body = jsonencode({
    properties = {
      clusterProfile = {
        resourceGroupId      = local.resource_group_id
        pullSecret           = file("~/Downloads/pull-secret-latest.txt")
        domain               = local.domain
        fipsValidatedModules = "Disabled"
        version              = "4.12.25"
      }
      networkProfile = {
        podCidr              = "10.128.0.0/14"
        serviceCidr          = "172.30.0.0/16"
      }
      servicePrincipalProfile = {
        clientId             = azuread_service_principal.aro_app.application_id
        clientSecret         = azuread_service_principal_password.aro_app.value
      }
      masterProfile = {
        vmSize               = "Standard_D8s_v3"
        subnetId             = azurerm_subnet.master_subnet.id
        encryptionAtHost     = "Disabled"
      }
      workerProfiles = [
        {
          name               = "worker"
          vmSize             = "Standard_D8s_v3"
          diskSizeGB         = 128
          subnetId           = azurerm_subnet.worker_subnet.id
          count              = 3
          encryptionAtHost   = "Disabled"
        }
      ]
      apiserverProfile = {
        visibility           = "Public"
      }
      ingressProfiles = [
        {
          name               = "default"
          visibility         = "Public"
        }
      ]
    }
  })
  depends_on = [
    azurerm_subnet.worker_subnet,
    azurerm_subnet.master_subnet,
    azuread_service_principal_password.aro_app,
    azurerm_role_assignment.aro_resource_provider_service_principal_network_contributor
  ]
}

output "domain" {
  value = local.domain
}

Save Kubeconfig

Once we successfully create the OpenShift cluster, we need to obtain and save the kubeconfig file. It will allow Terraform to interact with the cluster through the master API. In order to get the kubeconfig content we need to call the Azure listAdminCredentials REST endpoint. It is the same as calling the az aro get-admin-kubeconfig command using CLI. It will return JSON with base64-encoded content. After decoding from JSON and Base64 we save the content inside the kubeconfig file in the current directory.

resource "azapi_resource_action" "test" {
  type        = "Microsoft.RedHatOpenShift/openShiftClusters@2023-07-01-preview"
  resource_id = "/subscriptions/${data.azurerm_client_config.current.subscription_id}/resourceGroups/openenv-${var.guid}/providers/Microsoft.RedHatOpenShift/openShiftClusters/aro-cluster-${var.guid}"
  action      = "listAdminCredentials"
  method      = "POST"
  response_export_values = ["*"]
}

output "kubeconfig" {
  value = base64decode(jsondecode(azapi_resource_action.test.output).kubeconfig)
}

resource "local_file" "kubeconfig" {
  content  =  base64decode(jsondecode(azapi_resource_action.test.output).kubeconfig)
  filename = "kubeconfig"
  depends_on = [azapi_resource_action.test]
}

Install OpenShift Operators with Terraform

Finally, we can interact with the existing OpenShift cluster via the kubeconfig file. In the first step, we will deploy some operators. In OpenShift operators are a preferred way of installing more advanced apps (for example consisting of several Deployments). Red Hat comes with a set of supported operators that allows us to extend OpenShift functionalities. It can be, for example, a service mesh, a clustered database, or a message broker.

Let’s imagine we want to install a service mesh on OpenShift. There are some dedicated operators for that. The OpenShift Service Mesh operator is built on top of the open-source project Istio. We will also install the OpenShift Distributed Tracing (Jaeger) and Kiali operators. In order to do that we need to define the Subscription CRD object. Also, if we install an operator in a different namespace than openshift-operators we have to create the OperatorGroup CRD object. Here’s the Terraform HCL script that installs our operators.

// (1)
resource "kubernetes_namespace" "openshift-distributed-tracing" {
  metadata {
    name = "openshift-distributed-tracing"
  }
}
resource "kubernetes_manifest" "tracing-group" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1"
    "kind"       = "OperatorGroup"
    "metadata"   = {
      "name"      = "openshift-distributed-tracing"
      "namespace" = "openshift-distributed-tracing"
    }
    "spec" = {
      "upgradeStrategy" = "Default"
    }
  }
}
resource "kubernetes_manifest" "tracing" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata" = {
      "name"      = "jaeger-product"
      "namespace" = "openshift-distributed-tracing"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "jaeger-product"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (2)
resource "kubernetes_manifest" "kiali" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata" = {
      "name"      = "kiali-ossm"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "kiali-ossm"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (3)
resource "kubernetes_manifest" "ossm" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata"   = {
      "name"      = "servicemeshoperator"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "stable"
      "installPlanApproval" = "Automatic"
      "name"                = "servicemeshoperator"
      "source"              = "redhat-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

// (4)
resource "kubernetes_manifest" "ossmconsole" {
  manifest = {
    "apiVersion" = "operators.coreos.com/v1alpha1"
    "kind"       = "Subscription"
    "metadata"   = {
      "name"      = "ossmconsole"
      "namespace" = "openshift-operators"
    }
    "spec" = {
      "channel"             = "candidate"
      "installPlanApproval" = "Automatic"
      "name"                = "ossmconsole"
      "source"              = "community-operators"
      "sourceNamespace"     = "openshift-marketplace"
    }
  }
}

After installing the operators we may proceed to the service mesh configuration (1). We need to use CRD objects installed by the operators. Kubernetes Terraform provider won’t be a perfect choice for that since it verifies the existence of an object before applying the whole script. Therefore we will switch to the kubectl provider that just applies the object without any initial verification. We need to create an Istio control plane using the ServiceMeshControlPlane object (2). As you see, it also enables distributed tracing with Jaeger and a dashboard with Kiali. Once a control plane is ready we may proceed to the next steps. We will create all the objects responsible for Istio configuration including VirtualService, DestinatioRule, and Gateway (3).

resource "kubernetes_namespace" "istio" {
  metadata {
    name = "istio"
  }
}

// (1)
resource "time_sleep" "wait_120_seconds" {
  depends_on = [kubernetes_manifest.ossm]

  create_duration = "120s"
}

// (2)
resource "kubectl_manifest" "basic" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio]
  yaml_body = <<YAML
kind: ServiceMeshControlPlane
apiVersion: maistra.io/v2
metadata:
  name: basic
  namespace: istio
spec:
  version: v2.4
  tracing:
    type: Jaeger
    sampling: 10000
  policy:
    type: Istiod
  telemetry:
    type: Istiod
  addons:
    jaeger:
      install:
        storage:
          type: Memory
    prometheus:
      enabled: true
    kiali:
      enabled: true
    grafana:
      enabled: true
YAML
}

resource "kubectl_manifest" "console" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio]
  yaml_body = <<YAML
kind: OSSMConsole
apiVersion: kiali.io/v1alpha1
metadata:
  name: ossmconsole
  namespace: istio
spec:
  kiali:
    serviceName: ''
    serviceNamespace: ''
    servicePort: 0
    url: ''
YAML
}

resource "time_sleep" "wait_60_seconds_2" {
  depends_on = [kubectl_manifest.basic]

  create_duration = "60s"
}

// (3)
resource "kubectl_manifest" "access" {
  depends_on = [time_sleep.wait_120_seconds, kubernetes_namespace.istio, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: maistra.io/v1
kind: ServiceMeshMemberRoll
metadata:
  name: default
  namespace: istio
spec:
  members:
    - demo-apps
YAML
}

resource "kubectl_manifest" "gateway" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
  namespace: demo-apps
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - quarkus-insurance-app.apps.${var.domain}
        - quarkus-person-app.apps.${var.domain}
YAML
}

resource "kubectl_manifest" "quarkus-insurance-app-vs" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-insurance-app-vs
  namespace: demo-apps
spec:
  hosts:
    - quarkus-insurance-app.apps.${var.domain}
  gateways:
    - microservices-gateway
  http:
    - match:
        - uri:
            prefix: "/insurance"
      rewrite:
        uri: " "
      route:
        - destination:
            host: quarkus-insurance-app
          weight: 100
YAML
}

resource "kubectl_manifest" "quarkus-person-app-dr" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: quarkus-person-app-dr
  namespace: demo-apps
spec:
  host: quarkus-person-app
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
YAML
}

resource "kubectl_manifest" "quarkus-person-app-vs-via-gw" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-person-app-vs-via-gw
  namespace: demo-apps
spec:
  hosts:
    - quarkus-person-app.apps.${var.domain}
  gateways:
    - microservices-gateway
  http:
    - match:
      - uri:
          prefix: "/person"
      rewrite:
        uri: " "
      route:
        - destination:
            host: quarkus-person-app
            subset: v1
          weight: 100
        - destination:
            host: quarkus-person-app
            subset: v2
          weight: 0
YAML
}

resource "kubectl_manifest" "quarkus-person-app-vs" {
  depends_on = [time_sleep.wait_60_seconds_2, kubernetes_namespace.demo-apps]
  yaml_body  = <<YAML
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-person-app-vs
  namespace: demo-apps
spec:
  hosts:
    - quarkus-person-app
  http:
    - route:
        - destination:
            host: quarkus-person-app
            subset: v1
          weight: 100
        - destination:
            host: quarkus-person-app
            subset: v2
          weight: 0
YAML
}

Finally, we will run our sample Quarkus apps that communicate through the Istio mesh and connect to the Postgres database. The script is quite large. All the apps are running in the demo-apps namespace (1). They are connecting with the Postgres database installed using the Terraform Helm provider from Bitnami chart (2). Finally, we are creating the Deployments for two apps: person-service and insurance-service (3). There are two versions per microservice. Don’t focus on the features of the apps. They are here just to show the subsequent layers of the installation process. We are starting with the operators and CRDs, then moving to Istio configuration, and finally installing our custom apps.

// (1)
resource "kubernetes_namespace" "demo-apps" {
  metadata {
    name = "demo-apps"
  }
}

resource "kubernetes_secret" "person-db-secret" {
  depends_on = [kubernetes_namespace.demo-apps]
  metadata {
    name      = "person-db"
    namespace = "demo-apps"
  }
  data = {
    postgres-password = "123456"
    password          = "123456"
    database-user     = "person-db"
    database-name     = "person-db"
  }
}

resource "kubernetes_secret" "insurance-db-secret" {
  depends_on = [kubernetes_namespace.demo-apps]
  metadata {
    name      = "insurance-db"
    namespace = "demo-apps"
  }
  data = {
    postgres-password = "123456"
    password          = "123456"
    database-user     = "insurance-db"
    database-name     = "insurance-db"
  }
}

// (2)
resource "helm_release" "person-db" {
  depends_on = [kubernetes_namespace.demo-apps]
  chart            = "postgresql"
  name             = "person-db"
  namespace        = "demo-apps"
  repository       = "https://charts.bitnami.com/bitnami"

  values = [
    file("manifests/person-db-values.yaml")
  ]
}
resource "helm_release" "insurance-db" {
  depends_on = [kubernetes_namespace.demo-apps]
  chart            = "postgresql"
  name             = "insurance-db"
  namespace        = "demo-apps"
  repository       = "https://charts.bitnami.com/bitnami"

  values = [
    file("manifests/insurance-db-values.yaml")
  ]
}

// (3)
resource "kubernetes_deployment" "quarkus-insurance-app" {
  depends_on = [helm_release.insurance-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-insurance-app"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-insurance-app"
        version = "v1"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-insurance-app"
          version = "v1"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-insurance-app"
          image = "piomin/quarkus-insurance-app:v1"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "insurance-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "insurance-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "insurance-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_service" "quarkus-insurance-app" {
  depends_on = [helm_release.insurance-db, time_sleep.wait_60_seconds_2]
  metadata {
    name = "quarkus-insurance-app"
    namespace = "demo-apps"
    labels = {
      app = "quarkus-insurance-app"
    }
  }
  spec {
    type = "ClusterIP"
    selector = {
      app = "quarkus-insurance-app"
    }
    port {
      port = 8080
      name = "http"
    }
  }
}

resource "kubernetes_deployment" "quarkus-person-app-v1" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-person-app-v1"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-person-app"
        version = "v1"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-person-app"
          version = "v1"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-person-app"
          image = "piomin/quarkus-person-app:v1"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "person-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_deployment" "quarkus-person-app-v2" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name      = "quarkus-person-app-v2"
    namespace = "demo-apps"
    annotations = {
      "sidecar.istio.io/inject": "true"
    }
  }
  spec {
    selector {
      match_labels = {
        app = "quarkus-person-app"
        version = "v2"
      }
    }
    template {
      metadata {
        labels = {
          app = "quarkus-person-app"
          version = "v2"
        }
        annotations = {
          "sidecar.istio.io/inject": "true"
        }
      }
      spec {
        container {
          name = "quarkus-person-app"
          image = "piomin/quarkus-person-app:v2"
          port {
            container_port = 8080
          }
          env {
            name = "POSTGRES_USER"
            value_from {
              secret_key_ref {
                key = "database-user"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_PASSWORD"
            value_from {
              secret_key_ref {
                key = "password"
                name = "person-db"
              }
            }
          }
          env {
            name = "POSTGRES_DB"
            value_from {
              secret_key_ref {
                key = "database-name"
                name = "person-db"
              }
            }
          }
        }
      }
    }
  }
}

resource "kubernetes_service" "quarkus-person-app" {
  depends_on = [helm_release.person-db, time_sleep.wait_60_seconds_2]
  metadata {
    name = "quarkus-person-app"
    namespace = "demo-apps"
    labels = {
      app = "quarkus-person-app"
    }
  }
  spec {
    type = "ClusterIP"
    selector = {
      app = "quarkus-person-app"
    }
    port {
      port = 8080
      name = "http"
    }
  }
}

Applying Terraform Scripts

Finally, we can apply the whole Terraform configuration described in the article. Here’s the aro-with-servicemesh.sh script responsible for running required Terraform commands. It is placed in the repository root directory. In the first step, we go to the aro directory to apply the script responsible for creating the Openshift cluster. The domain name is automatically generated by Terraform, so we will export it using the terraform output command. After that, we may apply the scripts with operators and Istio configuration. In order to do everything automatically we pass the location of the kubeconfig file and the generated domain name as variables.

#! /bin/bash

cd aro
terraform init
terraform apply -auto-approve
domain="apps.$(terraform output -raw domain).eastus.aroapp.io"

cd ../servicemesh
terraform init
terraform apply -auto-approve -var kubeconfig=../aro/kubeconfig -var domain=$domain

Let’s run the aro-with-service-mesh.sh script. Once you will do it you should have a similar output as visible below. In the beginning, Terraform creates several objects required by the ARO cluster like a virtual network or service principal. Once those resources are ready, it starts the main part – ARO installation.

Let’s switch to Azure Portal. As you see the installation is in progress. There are several other newly created resources. Of course, there is also the resource representing the OpenShift cluster.

Now, arm yourself with patience. You can easily go get a coffee…

You can verify the progress, e.g. by displaying a list of virtual machines. If you see all the 3 master and 3 worker VMs running it means that we are slowly approaching the end.

It may take even more than 40 minutes. That’s why I overridden a default timeout for azapi resource to 75 minutes. Once the cluster is ready, Terraform will connect to the instance of OpenShift to install operators there. In the meantime, we can switch to Azure Portal and see the details about the ARO cluster. It displays, among others, the OpenShift Console URL. Let’s log in to the console.

In order to obtain the admin password we need to run the following command (for my cluster and resource group name):

$ az aro list-credentials -n aro-cluster-p2pvg -g openenv-p2pvg

Here’s our OpenShift console:

Let’s back to the installation process. The first part has been just finished. Now, the script executes terraform commands in the servicemesh directory. As you see, it installed our operators.

Let’s check out how it looks in the OpenShift Console. Go to the Operators -> Installed Operators menu item.

Of course, the installation is continued in the background. After installing the operators, it created the Istio Control Plane using the CRD object.

Let’s switch to the OpenShift Console once again. Go to the istio project. In the list of installed operators find Red Hat OpenShift Service Mesh and then go to the Istio Service Mesh Control Plane tab. You should see the basic object. As you see all 9 required components, including Istio, Kiali, and Jaeger instances, are successfully installed.

And finally the last part of our exercise. Installation is finished. Terraform applied deployment with our Postgres databases and some Quarkus apps.

In order to see the list of apps we can go to the Topology view in the Developer perspective. All the pods are running. As you see there is also a Kiali console available. We can click that link.

In the Kiali dashboard, we can see a detailed view of our service mesh. For example, there is a diagram showing a graphical visualization of traffic between the services.

Final Thoughts

If you use Terraform for managing your cloud infrastructure this article is for you. Did you already have doubts is it possible to easily create and configure the OpenShift cluster with Terraform? This article should dispel your doubts. You can also easily create your ARO cluster just by cloning this repository and running a single script on your cloud account. Enjoy

The post Manage OpenShift with Terraform appeared first on Piotr's TechBlog.

The solution presented in this article is not the only way how we can deal with HashiCorp Vault on Kubernetes. If you are interested in other approaches you may refer to some of my previous articles. In that article, you can find a guide on how to integrate Vault secrets with Argo CD through the plugin. If you are running Spring Boot apps on Kubernetes you can also be interested in Spring Cloud Vault support. In that case please refer to the following article.

How it works

I guess you may not be very familiar with the Container Storage Interface (CSI) pattern. At the high-level CSI is a standard for exposing block or file storage to the containers. It is implemented by different storage providers.

The Secrets Store CSI Driver is running on Kubernetes as a DeamonSet. It is interacting with every instance of Kubelet on the Kubernetes nodes. Once the pod is starting, the Secrets Store CSI Driver communicates with the external secrets provider to retrieve the secret content. The following diagram illustrates how Secrets Store CSI Driver works on Kubernetes.

It provides the SecretProviderClass CRD to manage that process. In this provider class, we need to set the secure vault address and the location of the secret keys. Here’s the SecretProviderClass for our scenario. We will use HashiCorp Vault running on Kubernetes.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: default
spec:
  parameters:
    objects: |-
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"
    roleName: webapp
    vaultAddress: 'http://vault.vault.svc:8200'
  provider: vault

Here’s the location of our secret in the HashiCorp Vault. As you see the current value of the password entry is test1.

Source Code

As usual – if you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my two GitHub repositories. The first of them contains Terraform scripts for installing Vault and Secrets Store CSI. After cloning it you go to the vault-ocp directory. The second repository contains a simple Spring Boot app for a test scenario. Once you clone it, go to the spring-util-app directory. Then you should just follow my instructions

Install Vault and Secrets Store CSI Driver with Terraform

As I mentioned before, we will use Terraform to set up almost the whole test scenario today. We will just leverage Skaffold, in the last step, to deploy the Spring Boot app on the Kubernetes cluster.

In order to install both Vault and Secrets Store CSI Driver we will use Helm charts. To do that part of the exercise, we need kubernetes (1) and helm (2) as the Terraform providers. The third step (3) is required only if you run the scenario on OpenShift. It changes the default service account access restrictions and security context constraints (SCCs) to ensure that a pod has sufficient permissions to start on OpenShift. Then we may proceed to the Helm charts installation. For the Secrets Store CSI Driver chart (4), it is important to enable secrets rotation since we will test this feature at the end of the article. Finally, we can install the HashiCorp Vault chart (5).

# (1)
provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

# (2)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

# (3)
resource "kubernetes_cluster_role_binding" "privileged" {
  metadata {
    name = "system:openshift:scc:privileged"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "system:openshift:scc:privileged"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "secrets-store-csi-driver"
    namespace = "k8s-secrets-store-csi"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "vault-csi-provider"
    namespace = "vault"
  }
}

resource "kubernetes_namespace" "vault" {
  metadata {
    name = "vault"
  }
}

resource "kubernetes_service_account" "vault-sa" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name      = "vault"
    namespace = "vault"
  }
}

resource "kubernetes_secret_v1" "vault-secret" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name = "vault-token"
    namespace = "vault"
    annotations = {
      "kubernetes.io/service-account.name" = "vault"
    }
  }

  type = "kubernetes.io/service-account-token"
}

# (4)
resource "helm_release" "secrets-store-csi-driver" {
  chart            = "secrets-store-csi-driver"
  name             = "csi-secrets-store"
  namespace        = "k8s-secrets-store-csi"
  create_namespace = true
  repository       = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"

  set {
    name  = "linux.providersDir"
    value = "/var/run/secrets-store-csi-providers"
  }

  set {
    name  = "syncSecret.enabled"
    value = "true"
  }

  set {
    name  = "enableSecretRotation"
    value = "true"
  }
}

# (5)
resource "helm_release" "vault" {
  chart            = "vault"
  name             = "vault"
  namespace        = "vault"
  create_namespace = true
  repository       = "https://helm.releases.hashicorp.com"

  values = [
    file("values.yaml")
  ]
}

The Helm values.yaml file used by the Vault chart is visible below. In order to simplify deployment, we will enable the development mode (1). It generates a root token automatically and runs a single instance of Vault. We can enable Route for OpenShift (2) and use the image supported by Red Hat (3). Of course, we also need to enable global OpenShift configuration (4). You can omit all three steps (2) (3) (4) when running the scenario on vanilla Kubernetes. Finally, we need to enable CSI support (5) and disable the Vault sidecar injector which is not needed in that exercise (7). The path in the csi.deamonSet.providersDir property should be the same as the linux.providersDir in the Halm chart params (6).

server:
  dev:
    enabled: true # (1)
  route: # (2)
    enabled: true
    host: ""
    tls: null
  image: # (3)
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.12.4-ubi"
  serviceAccount:
    name: vault
    create: false
global: # (4)
  openshift: true
csi: 
  debug: true
  enabled: true # (5)
  daemonSet:
    providersDir: /var/run/secrets-store-csi-providers # (6)
    securityContext:
      container:
        privileged: true
injector: # (7)
  enabled: false

Finally, let’s apply the configuration to the target cluster.

$ terraform apply -auto-approve -compact-warnings

Here’s the result of the terraform apply command for my cluster:

In order to verify if everything has been installed successfully we can display the details of the vault-csi-provider DaemonSet in the vault namespace.

$ kubectl describe ds vault-csi-provider -n vault

Then, we can do a very similar thing for the Secrets Store CSI driver. We need to display the details of the csi-secrets-store-secrets-store-csi-driver DeamonSet.

$ kubectl describe ds csi-secrets-store-secrets-store-csi-driver \
  -n k8s-secrets-store-csi

Configure Vault with Terraform

The big advantage of using Terraform in our scenario is its integration with Vault. There is a dedicated Terraform provider for interacting with HashiCorp Vault. In order to set up the provider, we need to pass the Vault token (root in dev mode) and address. We will still need the kubernetes provider in that part of the exercise.

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "vault" {
  token = "root"
  address = var.vault-addr
}

We need to set the Kubernetes context path name and Vault API address in the variables.tf file. Here’s my Terraform variables.tf file:

variable "cluster-context" {
  type    = string
  default = "default/api-cluster-6sccr-6sccr-sandbox1544-opentlc-com:6443/opentlc-mgr"
}

variable "vault-addr" {
  type = string
  default = "http://vault-vault.apps.cluster-6sccr.6sccr.sandbox1544.opentlc.com"
}

The Terraform script responsible for configuring Vault is visible below. There are several things we need to do before deploying a sample Spring Boot app. Here’s a list of the required steps:

1. We need to enable Kubernetes authentication in Vault. The Secrets Store CSI Driver will use it to authenticate against the instance of Vault
2. In the second step, we will create a test secret. Its name is password and the value is test1. It is stored in Vault under the /secret/data/db-pass path.
3. Then, we have to Configure Kubernetes authenticate method.
4. In the fourth step, we are creating the policy for our app. It has read access to the secret created in step 2
5. We are creating the ServicerAccount for our sample Spring Boot app in the default namespace. The name of the ServiceAccount object is webapp-sa.
6. Finally, we can proceed to the last step in the Vault configuration – the authentication role required to access the secret. The name of the role is webapp and is then used by the Secrets Store CSI SecretProviderClass CR. The authentication role refers to the already created policy and ServiceAccount webapp-sa in the default namespace.
7. Once the Vault backend is configured properly we create the Secrets Store CSI SecretProviderClass CR.

# (1)
resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

# (2)
resource "vault_kv_secret_v2" "secret" {
  mount = "secret"
  name = "db-pass"
  data_json = jsonencode(
    {
      password = "test1"
    }
  )
}

data "kubernetes_secret" "vault-token" {
  metadata {
    name      = "vault-token"
    namespace = "vault"
  }
}

# (3)
resource "vault_kubernetes_auth_backend_config" "example" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://172.30.0.1:443"
  kubernetes_ca_cert     = data.kubernetes_secret.vault-token.data["ca.crt"]
  token_reviewer_jwt     = data.kubernetes_secret.vault-token.data.token
}

# (4)
resource "vault_policy" "internal-app" {
  name = "internal-app"

  policy = <<EOT
path "secret/data/db-pass" {
  capabilities = ["read"]
}
EOT
}

# (5)
resource "kubernetes_service_account" "webapp-sa" {
  metadata {
    name      = "webapp-sa"
    namespace = "default"
  }
}

# (6)
resource "vault_kubernetes_auth_backend_role" "internal-role" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "webapp"
  bound_service_account_names      = ["webapp-sa"]
  bound_service_account_namespaces = ["default"]
  token_ttl                        = 3600
  token_policies                   = ["internal-app"]
}

# (7)
resource "kubernetes_manifest" "vault-database" {
  manifest = {
    "apiVersion" = "secrets-store.csi.x-k8s.io/v1alpha1"
    "kind"       = "SecretProviderClass"
    "metadata" = {
      "name"      = "vault-database"
      "namespace" = "default"
    }
    "spec" = {
      "provider"   = "vault"
      "parameters" = {
        "vaultAddress" = "http://vault.vault.svc:8200"
        "roleName"     = "webapp"
        "objects"      = "- objectName: \"db-password\"\n  secretPath: \"secret/data/db-pass\"\n  secretKey: \"password\""
      }
    }
  }
}

Once again, to apply the configuration we need to execute the terraform apply command.

Of course, we could apply the whole configuration visible above using Vault CLI or UI. However, we can verify it with Vault UI. In order to log in there we must use the root token. After login, we need to go to the Access tab and then to the Auth Methods menu. As you see, there is a webapp method defined in Terraform scripts.

Let’s switch to the Policies tab. Then we can check out if the internal-app policy exists.

Run the App with Mounted Secrets

Once we applied the whole configuration with Terraform we may proceed to the sample Spring Boot app. The idea is pretty simple. Our app just reads the data from the file and exposes it as the REST endpoint. Here’s the REST @Controller implementation:

@RestController
@RequestMapping("/api")
class SampleUtilController {

    @GetMapping("/db-password")
    fun resourceString(): String {
        val file = File("/mnt/secrets-store/db-password")
        return if(file.exists()) file.readText()
        else "none"
    }
}

Here’s the app Deployment manifest. As you see we are using the secrets-store.csi.k8s.io implementation of the CSI driver for mounted volume. It refers the vault-database SecretProviderClass object created with the Terraform script. The volume is containing the file with the value of our secret. We are mounting it under the path /mnt/secrets-store, which is accessed by the Spring Boot application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-util-app
spec:
  selector:
    matchLabels:
      app: sample-util-app
  template:
    metadata:
      labels:
        app: sample-util-app
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: sample-util-app
          image: piomin/sample-util-app
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /mnt/secrets-store
              name: secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-database"

Here’s the Kubernetes Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: sample-util-app
spec:
  type: ClusterIP
  selector:
    app: sample-util-app
  ports:
    - port: 8080
      targetPort: 8080

We can easily build and deploy the app with Skaffold. It also allows exposing a port outside the cluster as a local port with the port-forward option.

$ skaffold dev --port-forward

Finally, we can our test endpoint GET /api/db-password. It returns the value we have already set in Vault for the db-pass/password secret.

$ curl http://localhost:8080/api/db-password
test2

Now, let’s test the secret rotation feature. In order to do that we need to change the value of the db-pass/password secret. We can do it using Vault UI. We can set the test2 value:

Secrets Store CSI Driver periodically queries managed secrets to detect changes. So, after the change, we would probably wait a moment until our app refreshes that value. The default poll interval is 2 minutes. We can override it using the rotation-poll-interval parameter (e.g. on the Helm chart). However, the most important thing is, that everything happens without restarting the pod. The only trace of change is in the events:

Now, let’s query for the latest value of the key used by the app. As you see the value has been refreshed.

Final Thoughts

If you are looking for a solution that injects Vault secrets into your app without creating Kubernetes Secret, Secrets Store CSI Driver is the solution for you. It is able to refresh the value of the secret in your app without restarting the container. In this article, I’m presenting how to install and configure it with Terraform to simplify the installation and configuration process.

The post Vault with Secrets Store CSI Driver on Kubernetes appeared first on Piotr's TechBlog.

How it works

Let’s discuss our architecture in greater detail. In the first step, we are installing ArgoCD and Sealed Secrets on Kubernetes with Terraform. In order to install both these tools, we will leverage Terraform support for Helm charts. During ArgoCD installation we will also create the Application that refers to the Git repository with configuration (1). This repository will contain YAML manifests including an encrypted version of our Kubernetes Secret. When Terraform installs Sealed Secrets it sets the private key for secrets decryption and the public key for encryption (2).

Once we successfully install Sealed Secrets we can interact with its controller running on the cluster with kubeseal CLI. With the kubeseal command we can get an encrypted version of the input Kubernetes Secret (3). Then we place an encrypted secret inside the repository with the app deployment manifests (4). Argo CD will automatically apply the latest configuration to the Kubernetes cluster (5). Once the new encrypted secret appears Sealed Secrets detects it and tries to decrypt using a previously set private key (6). As a result, a new Secret object is created and then injected into our sample app (7). That’s the last step of our exercise. We will test the result using the HTTP endpoint exposed by the app.

Prerequisites

To proceed with the exercise you need to have a running instance of Kubernetes. It can be a local or a cloud instance – it doesn’t matter. Additionally, you also need to install two CLI tools on your laptop:

1. kubeseal – the client-side part of Sealed Secrets. You will installation instructions here.
2. terraform – to run Terraform HCL scripts you need to have CLI. You will installation instructions here.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that you need to clone my GitHub repository. This repository contains sample Terraform scripts for initializing Kubernetes. You should go to the sealedsecrets directory. Then just follow my instructions.

Install ArgoCD and Sealed Secrets with Terraform

Assuming you already have the terraform CLI installed the first thing you need to do is to define the Helm provider with the path to your Kube config and the name of the Kube context. Here’s our providers.tf file:

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

Since I’m using Kubernetes on the Docker Desktop the name of my context is docker-desktop. Here’s the variables.tf file:

variable "cluster-context" {
  type    = string
  default = "docker-desktop"
}

Here’s the Terraform script for installing ArgoCD and Sealed Secrets. For Sealed Secrets, we need to set keys for encryption and decryption. By default, the Sealed Secrets chart detects an existing TLS secret with the name sealed-secrets-key inside the target namespace. If it does not exist the chart creates a new one containing generated keys. In order to define the secret with the predefined TLS keys, we first need to create the namespace (1). Then we create the Secret sealed-secrets-key that contains our tls.crt and tls.key (2). After that we may install Sealed Secrets in the sealed-secrets namespace using Helm chart (3).

At the same time, we are installing ArgoCD in the argocd namespace also using Helm chart (4). The chart automatically creates the namespace thanks to the create_namespace parameter. Once we install ArgoCD we can create the Application object responsible for synchronization between the Git repository and Kubernetes cluster. We can also do it using the same Terraform script thanks to the argocd-apps Helm chart (5). It allows us to define a list of ArgoCD Applications inside the Helm values file (6).

# Sealed Secrets Installation

# (1)
resource "kubernetes_namespace" "sealed-secrets-ns" {
  metadata {
    name = "sealed-secrets"
  }
}

# (2)
resource "kubernetes_secret" "sealed-secrets-key" {
  depends_on = [kubernetes_namespace.sealed-secrets-ns]
  metadata {
    name      = "sealed-secrets-key"
    namespace = "sealed-secrets"
  }
  data = {
    "tls.crt" = file("keys/tls.crt")
    "tls.key" = file("keys/tls.key")
  }
  type = "kubernetes.io/tls"
}

# (3)
resource "helm_release" "sealed-secrets" {
  depends_on = [kubernetes_secret.sealed-secrets-key]
  chart      = "sealed-secrets"
  name       = "sealed-secrets"
  namespace  = "sealed-secrets"
  repository = "https://bitnami-labs.github.io/sealed-secrets"
}

# ArgoCD Installation

# (4)
resource "helm_release" "argocd" {
  chart            = "argo-cd"
  name             = "argocd"
  namespace        = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  create_namespace = true
}

# (5)
resource "helm_release" "argocd-apps" {
  depends_on = [helm_release.argocd]
  chart      = "argocd-apps"
  name       = "argocd-apps"
  namespace  = "argocd"
  repository = "https://argoproj.github.io/argo-helm"

  # (6)
  values = [
    file("argocd/applications.yaml")
  ]
}

We store Helm values inside the argocd/applications.yml file. In fact, we are going to apply the same set of YAML manifests into two different namespaces: demo-1 and demo-2. The namespace is automatically created during the synchronization.

applications:
 - name: sample-app-1
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-1
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
      - CreateNamespace=true
 - name: sample-app-2
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-2
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
       - CreateNamespace=true

Now, the only thing is to apply the configuration to the Kubernetes cluster. Before we do that we need to initialize Terraform working directory with the following command:

$ cd sealed-secrets
$ terraform init

Finally, we can apply the configuration:

$ terraform apply

Here’s the output of the terraform apply command:

Encrypt Secret with Kubeseal

Assuming you have already installed Sealed Secrets with Terraform on your Kubernetes cluster and kubeseal CLI on your laptop you can encrypt your secret for the first time. Here’s our Kubernetes Secret. It contains just the single field password with the base64-encoded value 123456. We are going to create the SealedSecret object from that Secret using the kubeseal command.

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
data:
  password: MTIzNDU2

By default, kubeseal tries to find the Sealed Secrets controller under the sealed-secrets-controller name inside the kube-system namespace. As you see we have already installed it in the sealed-secrets namespace under the sealed-secrets name.

We need to override both the controller name and namespace in the kubeseal command with the --controller-name and --controller-namespace parameters. Here’s our command:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets

The result may be quite surprising. Sealed Secrets doesn’t allow encrypting secrets without a namespace set in the YAML manifest. That’s because, by default, it uses a strict scope. With that scope, the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data. For me, it adds the default namespace as shown below.

Therefore it won’t be possible to decrypt the secret in a different namespace than the namespace set for the input Kubernetes Secret. On the other hand, we want to apply the same configuration in two different namespaces demo-1 and demo-2. In that case, we have to change the default scope to cluster-wide. With that kubeseal parameter the secret can be unsealed in any namespace and can be given any name. Here’s the command we should use to generate the SealedSecret object:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets \
   --scope cluster-wide

The output file of the command visible above contains the encrypted secret inside the SealedSecret object. Now, we should just add that YAML manifest to our Git repository.

Apply Sealed Secret with ArgoCD

Our sample Git repository with configuration for ArgoCD is available here. You should go to the apps/simple-with-secret directory. You will find there a Deployment, Service and SealedSecret objects. What’s important they don’t have any namespace set. Here’s our SealedSecret object:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  creationTimestamp: null
  name: sample-secret
spec:
  encryptedData:
    password: AgCW2Nf1gZzn42QQai/zr0VAtb5ZFyOjxMC8ghYcp5bu4EiYmJupX726zTx4XHQThrPgi/jHvzJoymToYJMIYuMegfKmZGcyMMZxJavYFTtlF9CIegPCkD3kjrJMCWcOadyDkBNIIfFAO6ljPwMD+stpsoBZ6WT8fGokxSwE/poKPpWFozC5RImf7HsYjGYVd8onxCySmcJZFYERi2G0qSWBlFDUsJ/ao5vyxIeiS25DBV1Bn475Lgyv6uTfvY6mesvrxw7OWjJmve2xRD/hS87Wp7cdBE264M/NMk1z24VysQr/ezSSI6S14NgzbcWo/hsKwWLmy6u259o8Xot5nVYpo2EhKFm/r62rko0eC2XMkjXhntMLKLpML3mTdadIFK50OauJvyVZS21sgeTlIMeSq6A6trekYyZvBtQaVixIthGHa/ymJXlIBZVJRL7/SJXquaX+J75AXUzPD3Hag8Kt5R5F6TVY2ox8RkMCpAVVAsiztMbyfRgzel6cAfDyj6l5f8GWI2T7gu5uHXgZFwVeyESn3aTO8qqws6NpLlwrtnjLwoCiXXC1Qo39wXaSJoH7fdJwihvOyiwbfaHkjhQwavNHpBoMEbKYQTV6DXSOTN8eeT1ZPoTN8AM+DtMdS2IpvMxZRsgaanh3O7gf5L02nGEq2WyP75s5sLoa7F8dQ27ZUeznqxIrNzrLqNM4dJuqZTbL4AM=
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/cluster-wide: "true"
      creationTimestamp: null
      name: sample-secret
    type: Opaque

Once it will be applied to the cluster, the Sealed Secrets controller will decrypt it and create the Kubernetes Secret object. Our sample app just takes the Kubernetes Secret and prints the value of the password key.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: quay.io/pminkows/sample-kotlin-spring:1.4.30
        ports:
        - containerPort: 8080
          name: http
        env:
          - name: PASS
            valueFrom:
              secretKeyRef:
                key: password
                name: sample-secret

We will test the app’s HTTP endpoint through the Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  type: ClusterIP
  selector:
    app: sample-app
  ports:
  - port: 8080
    name: http

Let’s verify what happened. Go to the ArgoCD dashboard. As you see there are two applications created by the Terraform script during installation. Both of them automatically applied the configuration from the Git repository to the cluster.

Let’s display details of the selected ArgoCD Application. As you see the sample-secret Secret object has already been created from the sample-secret SealedSecret object.

Now, let’s enable port-forward for the Kubernetes Service on port 8080:

$ kubectl port-forward svc/sample-app 8080:8080 -n demo-1

The app is able to display a list of environment variables. We can also display just a selected variable by calling the following endpoint:

$ curl http://localhost:8080/actuator/env/PASS

Final Thoughts

In general, there are two popular approaches to managing secrets on Kubernetes in GitOps style. In the first of them, we store the encrypted value of the secret in the Git repository. Then the software running on the cluster decrypts the value and creates Kubernetes Secret. That solution is represented by the Sealed Secrets and has been described today. In the second of them, we store just a reference to the secret, not the value of the secret in the Git repository. The value of the secret is stored in the third-party tool. Based on the key software running on the cluster retrieves the value.

The most popular example of such a third-party tool is HashiCorp Vault. You can read more about managing secrets with Vault and ArgoCD in the following article. There is also another very promising project in that area – External Secrets. You can expect my article about it soon

The post Sealed Secrets on Kubernetes with ArgoCD and Terraform appeared first on Piotr's TechBlog.

For a basic introduction to using Argo CD on Kubernetes, you may refer to this article.

Introduction

First of all, I would like to define the whole cluster and store its configuration in Git. I can’t use only Argo CD to achieve it, because Argo CD must run on the existing Kubernetes cluster. That’s why I need a tool that is able to create a cluster and then install Argo CD there. In that case, Terraform seems to be a natural choice. On the other hand, I don’t want to use Terraform to manage apps running on Kubernetes. It is perfect for a one-time activity like creating a cluster, but not for continuous tasks like app delivery and configuration management.

Here’s the list of things we are going to do:

1. In the first step, we will create a local Kubernetes cluster using Terraform
2. Then we will install OLM (Operator Lifecycle Manager) on the cluster. We need it to install Kafka with Strimzi (Step 5)
3. We will use Terraform to install Argo CD from the Helm chart and create a single Argo CD Application responsible for the whole cluster configuration based on Git
4. After that, Argo CD Application installs Strimzi Operator, creates Argo CD Project dedicated to Kafka installation and Argo CD Application that runs Kafka on Kubernetes
5. Finally, the Argo CD Application automatically creates all the CRD objects required for running Kafka

The most important thing here is that everything should happen after running the terraform apply command. Terraform installs Argo CD, and then Argo CD installs Kafka, which is our sample app in that scenario. Let’s see how it works.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. After that, you should just follow my instructions. Let’s begin.

1. Create Kubernetes Cluster with Terraform

In order to easily create a Kubernetes cluster, we will use Kind. There is a dedicated Terraform provider for Kind available here. Of course, you can run Kubernetes on any cloud, and you will also find Terraform providers for that.

Our cluster consists of three worker nodes and a single master node. We need three nodes because finally, we will install a Kafka cluster running in three instances. Each of them will be deployed on a different node. Here’s our Terraform main.tf file for that step. We need to define the latest version of the tehcyx/kind provider (which is 0.0.12) in the required_providers section. The name of our cluster is cluster1. We will also enable the wait_for_ready option, to proceed to the next steps after the cluster is ready.

terraform {
  required_providers {
    kind = {
      source = "tehcyx/kind"
      version = "0.0.12"
    }
  }
}

provider "kind" {}

resource "kind_cluster" "default" {
  name = "cluster-1"
  wait_for_ready = true
  kind_config {
    kind = "Cluster"
    api_version = "kind.x-k8s.io/v1alpha4"

    node {
      role = "control-plane"
    }

    node {
      role = "worker"
      image = "kindest/node:v1.23.4"
    }

    node {
      role = "worker"
      image = "kindest/node:v1.23.4"
    }

    node {
      role = "worker"
      image = "kindest/node:v1.23.4"
    }
  }
}

Just to verify a configuration you can run the command terraform init, and then terraform plan. After that, you could apply the configuration using terraform apply, but as you probably remember we will do it after the last all the configuration is ready to apply everything in one command.

2. Install OLM on Kubernetes

As I mentioned before, Operator Lifecycle Manager (OLM) is a prerequisite for installing the Strimzi Kafka operator. You can find the latest release of OLM here. In fact, it comes down to applying two YAML manifests on Kubernetes. The first of them crds.yaml contains CRD definitions. The second of them olm.yaml provides all required Kubernetes objects to install OLM. Let’s just copy both these files into the local directory inside our Terraform repository. In order to apply them to Kubernetes, we first need to enable the Terraform kubectl provider.

terraform {
  ...

  required_providers {
    kubectl = {
      source  = "gavinbunney/kubectl"
      version = ">= 1.7.0"
    }
  }
}

Why do we use the kubectl provider instead of the official Terraform Kubernetes provider? The crds.yaml contains pretty large CRDs that go over size limits. We can easily solve that problem by enabling the server-side apply on the kubectl provider. The next case is that there are multiple Kubernetes objects defined inside both the YAML files. The kubectl provider supports it via the for_each parameter.

data "kubectl_file_documents" "crds" {
  content = file("olm/crds.yaml")
}

resource "kubectl_manifest" "crds_apply" {
  for_each  = data.kubectl_file_documents.crds.manifests
  yaml_body = each.value
  wait = true
  server_side_apply = true
}

data "kubectl_file_documents" "olm" {
  content = file("olm/olm.yaml")
}

resource "kubectl_manifest" "olm_apply" {
  depends_on = [data.kubectl_file_documents.crds]
  for_each  = data.kubectl_file_documents.olm.manifests
  yaml_body = each.value
}

Le’s consider the last case in this section. Before applying any YAML we are creating a new Kubernetes cluster in the previous step. Therefore, we cannot use the existing context. Fortunately, we can use the output arguments from the kubectl provider with the Kubernetes address and auth credentials.

provider "kubectl" {
  host = "${kind_cluster.default.endpoint}"
  cluster_ca_certificate = "${kind_cluster.default.cluster_ca_certificate}"
  client_certificate = "${kind_cluster.default.client_certificate}"
  client_key = "${kind_cluster.default.client_key}"
}

3. Install Argo CD with Helm

This is the last step on the Terraform side. We are going to install Argo CD using its Helm chart. We also need to create a single Argo CD Application responsible for the cluster management. This Application will install the Kafka Strimzi operator and create another Argo CD Application‘s used e.g. for running the Kafka cluster. In the first step, we need to do the same thing as before: define a provider and set the Kubernetes cluster address. Here’s our definition in Terraform:

provider "helm" {
  kubernetes {
    host = "${kind_cluster.default.endpoint}"
    cluster_ca_certificate = "${kind_cluster.default.cluster_ca_certificate}"
    client_certificate = "${kind_cluster.default.client_certificate}"
    client_key = "${kind_cluster.default.client_key}"
  }
}

The tricky thing here is that we need to create the Application just after Argo CD installation. By default, Terraform verifies if there are required CRD objects on Kubernetes. In that case, it requires the Application CRD from argoproj.io/v1alpha1. Fortunately, we can use the Helm chart parameter allowing us to pass the declaration of additional Applications. In order to do that, we have to set a custom values.yaml file. Here’s the Terraform declaration for the Argo CD installation:

resource "helm_release" "argocd" {
  name  = "argocd"

  repository       = "https://argoproj.github.io/argo-helm"
  chart            = "argo-cd"
  namespace        = "argocd"
  version          = "4.9.7"
  create_namespace = true

  values = [
    file("argocd/application.yaml")
  ]
}

In order to create an initial Application, we need to use the Helm chart server.additionalApplications parameter as shown. Here’s the whole argocd/application.yaml file. To simplify, the configuration used by Argo CD is located in the repository as Terraform configuration. You can find all the required YAMLs in the argocd/manifests directory.

server:
  additionalApplications:
   - name: cluster-config
     namespace: argocd
     project: default
     source:
       repoURL: https://github.com/piomin/sample-terraform-kubernetes-argocd.git
       targetRevision: HEAD
       path: argocd/manifests/cluster
       directory:
         recurse: true
     destination:
       server: https://kubernetes.default.svc
     syncPolicy:
       automated:
         prune: false
         selfHeal: false

4. Configure Kubernetes cluster with Argo CD

The last two steps are managed by Argo CD. We have successfully completed the Kubernetes cluster installation process. Now, it’s time to install our first application there. Our example app is Kafka. So, firstly we need to install the Kafka Strimzi operator. To do that, we just need to define a Subscription object managed by the previously installed OLM. The definition is available in the repository as the strimzi.yaml file.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-strimzi-kafka-operator
  namespace: operators
spec:
  channel: stable
  name: strimzi-kafka-operator
  source: operatorhubio-catalog
  sourceNamespace: olm

We could configure a lot of aspects related to the whole cluster here. However, we just need to create a dedicated Argo CD Project and Application for Kafka configuration. Here’s our Project definition:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: kafka
  namespace: argocd
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - name: '*'
      namespace: '*'
      server: '*'
  sourceRepos:
    - '*'

Let’s place the kafka ArgoCD Application inside the newly created Project.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: kafka
  namespace: argocd
spec:
  destination:
    namespace: kafka
    server: https://kubernetes.default.svc
  project: kafka
  source:
    path: argocd/manifests/kafka
    repoURL: https://github.com/piomin/sample-terraform-kubernetes-argocd.git
    targetRevision: HEAD
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

5. Create Kafka Cluster using GitOps

Finally, the last part of our exercise. We will create and run a 3-node Kafka cluster on Kind. Here’s the Kafka object definition we store in Git. We are setting 3 replicas for both Kafka and Zookeeper (used by the Kafka cluster). This manifest is available in the repository under the path argocd/manifests/kafka/cluster.yaml. We are exposing the cluster on 9092 (plain) and 9093 (TLS) ports. The Kafka cluster has storage mounted as the PVC into the Deployment.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
spec:
  kafka:
    replicas: 3
    version: 3.2.0
    logging:
      type: inline
      loggers:
        kafka.root.logger.level: "INFO"
    config:
      auto.create.topics.enable: "false"
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: "3.2"
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    storage:
      type: jbod
      volumes:
        - id: 0
          type: persistent-claim
          size: 30Gi
          deleteClaim: true
  zookeeper:
    replicas: 3
    storage:
      type: persistent-claim
      size: 10Gi
      deleteClaim: true
  entityOperator:
    topicOperator: {}
    userOperator: {}

We will also define a single Kafka Topic inside the argocd/manifests/kafka/cluster.yaml manifest.

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaTopic
metadata:
  name: my-topic
  labels:
    strimzi.io/cluster: my-cluster
spec:
  partitions: 10
  replicas: 3

Execution on Kubernetes

Terraform

We have already prepared all the required scripts. Let’s proceed to the execution phase. If you still haven’t cloned the Git repository it’s time to do it:

$ git clone https://github.com/piomin/sample-terraform-kubernetes-argocd.git
$ cd sample-terraform-kubernetes-argocd

Firstly, let’s initialize our working directory containing Terraform configuration:

$ terraform init

Once we do it, we may preview a list of actions to perform:

$ terraform plan

You should receive a pretty large content as a response. Here’s the last part of my result:

If everything looks fine and there are no errors we may proceed to the next (final) step. Let’s begin the process:

$ terraform apply

All 24 objects should be successfully applied. Here’s the last part of the logs:

Now, you should have your cluster ready and running. Let’s display a list of Kind clusters:

$ kind get clusters
cluster-1

The name of our cluster is cluster-1. But the name of the Kubernetes context is kind-cluster-1:

Let’s display a list of applications deployed on the Kind cluster. You should have at least Argo CD and OLM installed. After some time Argo CD applies the configuration stored in the Git repository. Then, you should see the Kafka Strimzi operator installed in the operators namespace.

Argo CD

After that, we can go to the Argo CD web console. To access it easily on the local port let’s enable port-forward:

$ kubectl port-forward service/argocd-server 8443:443 -n argocd

Now, you can display the Argo CD web console on the https://localhost:8443. The default username is admin. The password is auto-generated by the Argo CD. You can find it inside the Kubernetes Secret argocd-initial-admin-secret.

$ kubectl get secret argocd-initial-admin-secret -n argocd --template={{.data.password}} | base64 -D

Here’s the list of our Argo CD Applications. The cluster-config has an auto-sync option enabled. It installs the Strimzi operator and creates kafka Argo CD Application. I could also enable auto-sync for kafka Application. But just for the demo purpose, I left there a manual approval. So, let’s run Kafka on our cluster. To do that click the Sync button on the kafka tile.

Once you do the Kafka installation is starting. Finally, you should have the whole cluster ready and running. Each Kafka and Zookeeper node are running on the different Kubernetes worker node:

That’s all. We created everything using a single Terraform command and one click on the Argo CD web console. Of course, we could enable auto-sync for the kafka application, so we even don’t need to log in to the Argo CD web console for the final effect.

The post Manage Kubernetes Cluster with Terraform and Argo CD appeared first on Piotr's TechBlog.