A platform team manages an Internal Developer Platform (IDP) to build golden paths and enable developer self-service in the organization. It may consist of many different tools and solutions. On the other hand, Internal Developer Portals serve as the GUI interface through which developers can discover and access internal developer platform capabilities. In the context of OpenShift, Red Hat Developer Hub simplifies the adoption of several cluster services for developers (e.g. Kubernates-native CI/CD tools). Today, you will learn how to integrate Developer Hub with OpenShift Pipelines (Tekton) and OpenShift GitOps (Argo CD). Let’s begin!

Source Code

If you would like to try it by yourself, you may always take a look at my source code. Our sample GitHub repository contains software templates written in the Backstage technology called Skaffolder. In this article, we will analyze a template dedicated to OpenShift available in the templates/spring-boot-basic-on-openshift directory. After cloning this repository, you should just follow my instructions.

Here’s the structure of our repository. It is pretty similar to the template for Spring Boot on Kubernetes described in my previous article about Backstage. Besides the template, it also contains the Argo CD and Tekton templates with YAML deployment manifests to apply on OpenShift.

.
├── README.md
├── backstage-templates.iml
├── skeletons
│   └── argocd
│       ├── argocd
│       │   └── app.yaml
│       └── manifests
│           ├── deployment.yaml
│           ├── pipeline.yaml
│           ├── service.yaml
│           ├── tasks.yaml
│           └── trigger.yaml
├── templates
│   └── spring-boot-basic-on-openshift
│       ├── skeleton
│       │   ├── README.md
│       │   ├── catalog-info.yaml
│       │   ├── devfile.yaml
│       │   ├── k8s
│       │   │   └── deployment.yaml
│       │   ├── pom.xml
│       │   ├── renovate.json
│       │   ├── skaffold.yaml
│       │   └── src
│       │       ├── main
│       │       │   ├── java
│       │       │   │   └── ${{values.javaPackage}}
│       │       │   │       ├── Application.java
│       │       │   │       ├── controller
│       │       │   │       │   └── ${{values.domainName}}Controller.java
│       │       │   │       └── domain
│       │       │   │           └── ${{values.domainName}}.java
│       │       │   └── resources
│       │       │       └── application.yml
│       │       └── test
│       │           ├── java
│       │           │   └── ${{values.javaPackage}}
│       │           │       └── ${{values.domainName}}ControllerTests.java
│       │           └── resources
│       │               └── k6
│       │                   └── load-tests-add.js
│       └── template.yaml
└── templates.yaml

Prerequisites

Before we start the exercise, we need to prepare our OpenShift cluster. We have to install three operators: OpenShift GitOps (Argo CD), OpenShift Pipelines (Tekton), and of course, Red Hat Developer Hub.

Once we install the OpenShift GitOps, it automatically creates an instance of Argo CD in the openshift-gitops namespace. That instance is managed by the openshift-gitops ArgoCD CRD object.

We need to override some default configuration settings there. Then, we will add a new user backstage with privileges for creating applications, and projects and generating API keys. Finally, we change the default TLS termination method for Argo CD Route to reencrypt. It is required to integrate with the Backstage Argo CD plugin. We also add the demo namespace as an additional namespace to place Argo CD applications in.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  sourceNamespaces:
    - demo
  server:
    ...
    route:
      enabled: true
      tls:
        termination: reencrypt
  ...
  rbac:
    defaultPolicy: ''
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
      p, backstage, applications, *, */*, allow
      p, backstage, projects, *, *, allow
    scopes: '[groups]'
  extraConfig:
    accounts.backstage: 'apiKey, login'

In order to generate the apiKey for the backstage user, we need to sign in to Argo CD with the argocd CLI as the admin user. Then, we should run the following command for the backstage account and export the generated token as the ARGOCD_TOKEN env variable:

$ export ARGOCD_TOKEN=$(argocd account generate-token --account backstage)

Finally, let’s obtain the long-lived API token for Kubernetes by creating a secret:

apiVersion: v1
kind: Secret
metadata:
  name: default-token
  namespace: backstage
  annotations:
    kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token

Then, we can copy and export it as the OPENSHIFT_TOKEN environment variable with the following command:

$ export OPENSHIFT_TOKEN=$(kubectl get secret default-token -o go-template='{{.data.token | base64decode}}')

Let’s add the ClusterRole view to the Backstage default ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-view-backstage
subjects:
- kind: ServiceAccount
  name: default
  namespace: backstage
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

Configure Red Hat Developer Hub on OpenShift

After installing the Red Hat Developer Hub operator on OpenShift, we can use the Backstage CRD to create and configure a new instance of the portal. Firstly, we will override some default settings using the app-config-rhdh ConfigMap (1). Then we will provide some additional secrets like tokens to the third-party party tools in the app-secrets-rhdh Secret (2). Finally, we will install and configure several useful plugins with the dynamic-plugins-rhdh ConfigMap (3). Here is the required configuration in the Backstage CRD.

apiVersion: rhdh.redhat.com/v1alpha1
kind: Backstage
metadata:
  name: developer-hub
  namespace: backstage
spec:
  application:
    appConfig:
      configMaps:
        # (1)
        - name: app-config-rhdh
      mountPath: /opt/app-root/src
    # (3)
    dynamicPluginsConfigMapName: dynamic-plugins-rhdh
    extraEnvs:
      secrets:
        # (2)
        - name: app-secrets-rhdh
    extraFiles:
      mountPath: /opt/app-root/src
    replicas: 1
    route:
      enabled: true
  database:
    enableLocalDb: true

Override Default Configuration Settings

The instance of Backstage will be deployed in the backstage namespace. Since OpenShift exposes it as a Route, the address of the portal on my cluster is https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io (1). Firstly, we need to override that address in the app settings. Then we need to enable authentication through the GitHub OAuth with the GitHub Red Hat Developer Hub application (2). Then, we should set the proxy endpoint to integrate with Sonarqube through the HTTP Request Action plugin (3). Our instance of Backstage should also read templates from the particular URL location (4) and should be able to create repositories in GitHub (5).

kind: ConfigMap
apiVersion: v1
metadata:
  name: app-config-rhdh
  namespace: backstage
data:
  app-config-rhdh.yaml: |

    # (1)
    app:
     baseUrl: https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io

    backend:
      baseUrl: https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io

    # (2)
    auth:
      environment: development
      providers:
        github:
          development:
            clientId: ${GITHUB_CLIENT_ID}
            clientSecret: ${GITHUB_CLIENT_SECRET}

    # (3)
    proxy:
      endpoints:
        /sonarqube:
          target: ${SONARQUBE_URL}/api
          allowedMethods: ['GET', 'POST']
          auth: "${SONARQUBE_TOKEN}:"

    # (4)
    catalog:
      rules:
        - allow: [Component, System, API, Resource, Location, Template]
      locations:
        - type: url
          target: https://github.com/piomin/backstage-templates/blob/master/templates.yaml

    # (5)
    integrations:
      github:
        - host: github.com
          token: ${GITHUB_TOKEN}
          
    sonarqube:
      baseUrl: https://sonarcloud.io
      apiKey: ${SONARQUBE_TOKEN}

Integrate with GitHub

In order to use GitHub auth we need to register a new app there. Go to the “Settings > Developer Settings > New GitHub App” in your GitHub account. Then, put the address of your Developer Hub instance in the “Homepage URL” field and the callback address in the “Callback URL” field (base URL + /api/auth/github/handler/frame)

Then, let’s edit our GitHub app to generate a new secret as shown below.

The client ID and secret should be saved as the environment variables for future use. Note, that we also need to generate a new personal access token (“Settings > Developer Settings > Personal Access Tokens”).

export GITHUB_CLIENT_ID=<YOUR_GITHUB_CLIENT_ID>
exporg GITHUB_CLIENT_SECRET=<YOUR_GITHUB_CLIENT_SECRET>
export GITHUB_TOKEN=<YOUR_GITHUB_TOKEN>

We already have a full set of required tokens and access keys, so we can create the app-secrets-rhdh Secret to store them on our OpenShift cluster.

$ oc create secret generic app-secrets-rhdh -n backstage \
  --from-literal=GITHUB_CLIENT_ID=${GITHUB_CLIENT_ID} \
  --from-literal=GITHUB_CLIENT_SECRET=${GITHUB_CLIENT_SECRET} \
  --from-literal=GITHUB_TOKEN=${GITHUB_TOKEN} \
  --from-literal=SONARQUBE_TOKEN=${SONARQUBE_TOKEN} \
  --from-literal=SONARQUBE_URL=https://sonarcloud.io \
  --from-literal=ARGOCD_TOKEN=${ARGOCD_TOKEN}

Install and Configure Plugins

Finally, we can proceed to the plugins installation. Do you remember how we can do it with the open-source Backstage on Kubernetes? I described it in my previous article. Red Hat Developer Hub drastically simplifies that process with the idea of dynamic plugins. This approach is based on the Janus IDP project. Developer Hub on OpenShift comes with ~60 preinstalled plugins that allow us to integrate various third-party tools including Sonarqube, Argo CD, Tekton, Kubernetes, or GitHub. Some of them are enabled by default, some others are installed but disabled. We can verify it after signing in to the Backstage UI. We can easily verify it in the “Administration” section:

Let’s take a look at the ConfigMap which contains a list of plugins to activate. It is pretty huge since we also provide configuration for the frontend plugins. Some plugins are optional. From the perspective of our exercise goal we need to activate at least the following list of plugins:

• janus-idp-backstage-plugin-argocd – to view the status of Argo CD synchronization in the UI
• janus-idp-backstage-plugin-tekton – to view the status of Tekton pipelines in the UI
• backstage-plugin-kubernetes-backend-dynamic – to integrate with the Kubernetes cluster
• backstage-plugin-kubernetes – to view the Kubernetes app pods in the UI
• backstage-plugin-sonarqube – to view the status of the Sonarqube scan in the UI
• roadiehq-backstage-plugin-argo-cd-backend-dynamic – to create the Argo CD Application from the template

kind: ConfigMap
apiVersion: v1
metadata:
  name: dynamic-plugins-rhdh
  namespace: backstage
data:
  dynamic-plugins.yaml: |
    includes:
      - dynamic-plugins.default.yaml
    plugins:
      - package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-github-pull-requests
        disabled: true
        pluginConfig:
          dynamicPlugins:
            frontend:
              roadiehq.backstage-plugin-github-pull-requests:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntityGithubPullRequestsOverviewCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 4"
                          md: "span 6"
                          xs: "span 12"
                      if:
                        allOf:
                          - isGithubPullRequestsAvailable
                  - mountPoint: entity.page.pull-requests/cards
                    importName: EntityGithubPullRequestsContent
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isGithubPullRequestsAvailable
      - package: './dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-argocd'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-argocd:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: ArgocdDeploymentSummary
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 8"
                          xs: "span 12"
                      if:
                        allOf:
                          - isArgocdConfigured
                  - mountPoint: entity.page.cd/cards
                    importName: ArgocdDeploymentLifecycle
                    config:
                      layout:
                        gridColumn: '1 / -1'
                      if:
                        allOf:
                          - isArgocdConfigured
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-tekton'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-tekton:
                mountPoints:
                  - mountPoint: entity.page.ci/cards
                    importName: TektonCI
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isTektonCIAvailable
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-topology'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-topology:
                mountPoints:
                  - mountPoint: entity.page.topology/cards
                    importName: TopologyPage
                    config:
                      layout:
                        gridColumn: "1 / -1"
                        height: 75vh
                      if:
                        anyOf:
                          - hasAnnotation: backstage.io/kubernetes-id
                          - hasAnnotation: backstage.io/kubernetes-namespace
      - package: './dynamic-plugins/dist/janus-idp-backstage-scaffolder-backend-module-sonarqube-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic'
        disabled: false
        pluginConfig:
          kubernetes:
            customResources:
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'pipelines'
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'pipelineruns'
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'taskruns'
            - group: 'route.openshift.io'
              apiVersion: 'v1'
              plural: 'routes'
            serviceLocatorMethod:
              type: 'multiTenant'
            clusterLocatorMethods:
              - type: 'config'
                clusters:
                  - name: ocp
                    url: https://api.piomin.eastus.aroapp.io:6443
                    authProvider: 'serviceAccount'
                    skipTLSVerify: true
                    skipMetricsLookup: true
                    serviceAccountToken: ${OPENSHIFT_TOKEN}
      - package: './dynamic-plugins/dist/backstage-plugin-kubernetes'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              backstage.plugin-kubernetes:
                mountPoints:
                  - mountPoint: entity.page.kubernetes/cards
                    importName: EntityKubernetesContent
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        anyOf:
                          - hasAnnotation: backstage.io/kubernetes-id
                          - hasAnnotation: backstage.io/kubernetes-namespace
      - package: './dynamic-plugins/dist/backstage-plugin-sonarqube'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              backstage.plugin-sonarqube:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntitySonarQubeCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 4"
                          md: "span 6"
                          xs: "span 12"
                      if:
                        allOf:
                          - isSonarQubeAvailable
      - package: './dynamic-plugins/dist/backstage-plugin-sonarqube-backend-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              roadiehq.backstage-plugin-argo-cd:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntityArgoCDOverviewCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 8"
                          xs: "span 12"
                      if:
                        allOf:
                          - isArgocdAvailable
                  - mountPoint: entity.page.cd/cards
                    importName: EntityArgoCDHistoryCard
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isArgocdAvailable
      - package: './dynamic-plugins/dist/roadiehq-scaffolder-backend-argocd-dynamic'
        disabled: false
        pluginConfig: {}
      - package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd-backend-dynamic
        disabled: false
        pluginConfig:
          argocd:
            appLocatorMethods:
              - type: 'config'
                instances:
                  - name: main
                    url: "https://openshift-gitops-server-openshift-gitops.apps.piomin.eastus.aroapp.io"
                    token: "${ARGOCD_TOKEN}"

Once we provide the whole configuration described above, we are ready to proceed with our Skaffolder template for the sample Spring Boot app.

Prepare Backstage Template for OpenShift

Our template consists of several steps. Firstly, we generate the app source code and push it to the app repository. Then, we register the component in the Backstage catalog and create a configuration repository for Argo CD. It contains app deployment manifests and the definition of the Tekton pipeline and trigger. Trigger is exposed as the Route, and can be called from the GitHub repository through the webhook. Finally, we are creating the project in Sonarcloud, the application in Argo CD, and registering the webhook in the GitHub app repository. Here’s our Skaffolder template.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: spring-boot-basic-on-openshift-template
  title: Create a Spring Boot app for OpenShift
  description: Create a Spring Boot app for OpenShift
  tags:
    - spring-boot
    - java
    - maven
    - tekton
    - renovate
    - sonarqube
    - openshift
    - argocd
spec:
  owner: piomin
  system: microservices
  type: service

  parameters:
    - title: Provide information about the new component
      required:
        - orgName
        - appName
        - domainName
        - repoBranchName
        - groupId
        - javaPackage
        - apiPath
        - namespace
        - description
        - registryUrl
        - clusterDomain
      properties:
        orgName:
          title: Organization name
          type: string
          default: piomin
        appName:
          title: App name
          type: string
          default: sample-spring-boot-app-openshift
        domainName:
          title: Name of the domain object
          type: string
          default: Person
        repoBranchName:
          title: Name of the branch in the Git repository
          type: string
          default: master
        groupId:
          title: Maven Group ID
          type: string
          default: pl.piomin.services
        javaPackage:
          title: Java package directory
          type: string
          default: pl/piomin/services
        apiPath:
          title: REST API path
          type: string
          default: /api/v1
        namespace:
          title: The target namespace on Kubernetes
          type: string
          default: demo
        description:
          title: Description
          type: string
          default: Spring Boot App Generated by Backstage
        registryUrl:
          title: Registry URL
          type: string
          default: image-registry.openshift-image-registry.svc:5000
        clusterDomain:
          title: OpenShift Cluster Domain
          type: string
          default: .apps.piomin.eastus.aroapp.io
  steps:
    - id: sourceCodeTemplate
      name: Generating the Source Code Component
      action: fetch:template
      input:
        url: ./skeleton
        values:
          orgName: ${{ parameters.orgName }}
          appName: ${{ parameters.appName }}
          domainName: ${{ parameters.domainName }}
          groupId: ${{ parameters.groupId }}
          javaPackage: ${{ parameters.javaPackage }}
          apiPath: ${{ parameters.apiPath }}
          namespace: ${{ parameters.namespace }}

    - id: publish
      name: Publishing to the Source Code Repository
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=${{ parameters.orgName }}&repo=${{ parameters.appName }}
        defaultBranch: ${{ parameters.repoBranchName }}
        repoVisibility: public

    - id: register
      name: Registering the Catalog Info Component
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

    - id: configCodeTemplate
      name: Generating the Config Code Component
      action: fetch:template
      input:
        url: ../../skeletons/argocd
        values:
          orgName: ${{ parameters.orgName }}
          appName: ${{ parameters.appName }}
          registryUrl: ${{ parameters.registryUrl }}
          namespace: ${{ parameters.namespace }}
          repoBranchName: ${{ parameters.repoBranchName }}
        targetPath: ./gitops

    - id: publish
      name: Publishing to the Config Code Repository
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=${{ parameters.orgName }}&repo=${{ parameters.appName }}-config
        defaultBranch: ${{ parameters.repoBranchName }}
        sourcePath: ./gitops
        repoVisibility: public

    - id: sonarqube
      name: Create a new project on Sonarcloud
      action: http:backstage:request
      input:
        method: 'POST'
        path: '/proxy/sonarqube/projects/create?name=${{ parameters.appName }}&organization=${{ parameters.orgName }}&project=${{ parameters.orgName }}_${{ parameters.appName }}'
        headers:
          content-type: 'application/json'

    - id: create-argocd-resources
      name: Create ArgoCD Resources
      action: argocd:create-resources
      input:
        appName: ${{ parameters.appName }}
        argoInstance: main
        namespace: ${{ parameters.namespace }}
        repoUrl: https://github.com/${{ parameters.orgName }}/${{ parameters.appName }}-config.git
        path: 'manifests'

    - id: create-webhook
      name: Create GitHub Webhook
      action: github:webhook
      input:
        repoUrl: github.com?repo=${{ parameters.appName }}&owner=${{ parameters.orgName }}
        webhookUrl: https://el-${{ parameters.appName }}-${{ parameters.namespace }}.${{ parameters.clusterDomain }}

  output:
    links:
      - title: Open the Source Code Repository
        url: ${{ steps.publish.output.remoteUrl }}
      - title: Open the Catalog Info Component
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}
      - title: SonarQube project URL
        url: ${{ steps['create-sonar-project'].output.projectUrl }}

Define Templates for OpenShift Pipelines

Compared to the article about Backstage on Kubernetes, we use Tekton instead of CircleCI as a build tool. Let’s take a look at the definition of our pipeline. It consists of four steps. In the final step, we use the OpenShift S2I mechanism to build the app image and push it to the local container registry.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: ${{ values.appName }}
  labels:
    backstage.io/kubernetes-id: ${{ values.appName }}
spec:
  params:
    - description: branch
      name: git-revision
      type: string
      default: master
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/${{ values.orgName }}/${{ values.appName }}.git'
        - name: revision
          value: $(params.git-revision)
        - name: sslVerify
          value: 'false'
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: maven
      params:
        - name: GOALS
          value:
            - test
        - name: PROXY_PROTOCOL
          value: http
        - name: CONTEXT_DIR
          value: .
      runAfter:
        - git-clone
      taskRef:
        kind: ClusterTask
        name: maven
      workspaces:
        - name: source
          workspace: source-dir
        - name: maven-settings
          workspace: maven-settings
    - name: sonarqube
      params:
        - name: SONAR_HOST_URL
          value: 'https://sonarcloud.io'
        - name: SONAR_PROJECT_KEY
          value: ${{ values.appName }}
      runAfter:
        - maven
      taskRef:
        kind: Task
        name: sonarqube-scanner
      workspaces:
        - name: source
          workspace: source-dir
        - name: sonar-settings
          workspace: sonar-settings
    - name: get-version
      params:
        - name: CONTEXT_DIR
          value: .
      runAfter:
        - sonarqube
      taskRef:
        kind: Task
        name: maven-get-project-version
      workspaces:
        - name: source
          workspace: source-dir
    - name: s2i-java
      params:
        - name: PATH_CONTEXT
          value: .
        - name: TLSVERIFY
          value: 'false'
        - name: MAVEN_CLEAR_REPO
          value: 'false'
        - name: IMAGE
          value: >-
            ${{ values.registryUrl }}/${{ values.namespace }}/${{ values.appName }}:$(tasks.get-version.results.version)
      runAfter:
        - get-version
      taskRef:
        kind: ClusterTask
        name: s2i-java
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir
    - name: maven-settings
    - name: sonar-settings

In order to run the pipeline after creating it, we need to apply the PipelineRun object.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: ${{ values.appName }}-init
spec:
  params:
    - name: git-revision
      value: master
  pipelineRef:
    name: ${{ values.appName }}
  serviceAccountName: pipeline
  workspaces:
    - name: source-dir
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - name: sonar-settings
      secret:
        secretName: sonarqube-secret-token
    - configMap:
        name: maven-settings
      name: maven-settings

In order to call the pipeline via the webhook from the app source repository, we also need to create the Tekton TriggerTemplate object. Once we push a change to the target repository, we trigger the run of the Tekton pipeline on the OpenShift cluster.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: ${{ values.appName }}
spec:
  params:
    - default: ${{ values.repoBranchName }}
      description: The git revision
      name: git-revision
    - description: The git repository url
      name: git-repo-url
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: ${{ values.appName }}-run-
      spec:
        params:
          - name: git-revision
            value: $(tt.params.git-revision)
        pipelineRef:
          name: ${{ values.appName }}
        serviceAccountName: pipeline
        workspaces:
          - name: source-dir
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 1Gi
          - name: sonar-settings
            secret:
              secretName: sonarqube-secret-token
          - configMap:
              name: maven-settings
            name: maven-settings

Deploy the app on OpenShift

Here’s the template for the app Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${{ values.appName }}
  labels:
    app: ${{ values.appName }}
    app.kubernetes.io/name: spring-boot
    backstage.io/kubernetes-id: ${{ values.appName }}
spec:
  selector:
    matchLabels:
      app: ${{ values.appName }}
  template:
    metadata:
      labels:
        app: ${{ values.appName }}
        backstage.io/kubernetes-id: ${{ values.appName }}
    spec:
      containers:
        - name: ${{ values.appName }}
          image: ${{ values.registryUrl }}/${{ values.namespace }}/${{ values.appName }}:1.0
          ports:
            - containerPort: 8080
              name: http
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/liveness
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits:
              memory: 1024Mi

Here’s the current version of the catalog-info.yaml file.

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.appName }}
  title: ${{ values.appName }}
  annotations:
    janus-idp.io/tekton: ${{ values.appName }}
    tektonci/build-namespace: ${{ values.namespace }}
    github.com/project-slug: ${{ values.orgName }}/${{ values.appName }}
    sonarqube.org/project-key: ${{ values.orgName }}_${{ values.appName }}
    backstage.io/kubernetes-id: ${{ values.appName }}
    argocd/app-name: ${{ values.appName }}
  tags:
    - spring-boot
    - java
    - maven
    - tekton
    - argocd
    - renovate
    - sonarqube
spec:
  type: service
  owner: piomin
  lifecycle: experimental

Now, let’s create a new component in Red Hat Developer Hub using our template. In the first step, you should choose the “Create a Spring Boot App for OpenShift” template as shown below.

Then, provide all the parameters in the form. Probably you will have to override the default organization name to your GitHub account name and the address of your OpenShift cluster. Once you make all the required changes click the “Review” button, and then the “Create” button on the next screen. After that, Red Hat Developer Hub creates all the things we need.

After confirmation, Developer Hub redirects to the page with the progress information. There are 8 action steps defined. All of them should be finished successfully. Then, we can just click the “Open the Catalog Info Component” link.

Viewing Component in Red Hat Developer Hub UI

Our app overview tab contains general information about the component registered in Backstage, the status of the Sonarqube scan, and the status of the Argo CD synchronization process. We can switch to the several other available tabs.

In the “CI” tab, we can see the history of the OpenShift Pipelines runs. We can switch to the logs of each pipeline step by clicking on it.

If you are familiar with OpenShift, you can recognize that view as a topology view from the OpenShift Console developer perspective. It visualizes all the deployments in the particular namespace.

In the “CD” tab, we can see the history of Argo CD synchronization operations.

Final Thoughts

Red Hat Developer Hub simplifies installation and configuration of Backstage in the Kubernetes-native environment. It introduces the idea of dynamic plugins, which can be easily customized in the configuration files. You can compare this approach with my previous article about Backstage on Kubernetes.

The post IDP on OpenShift with Red Hat Developer Hub appeared first on Piotr's TechBlog.

Today, we will consider performance issues related to running Tekton pipelines at scale. We will run several different pipelines at the same time or the same pipeline several times simultaneously. It results in maintaining a long history of previous runs. In order to handle it successfully, Tekton provides a special module configured with the TektonResults CRD. It can also clean up of the selected resources using the Kubernetes CronJob.

Source Code

This time we won’t work much with a source code. However, if you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. After that, you should follow my further instructions.

Install Tekton on Kubernetes

We can easily install Tekton on Kubernetes using the operator. We need to apply the following YAML manifest:

$ kubectl apply -f https://storage.googleapis.com/tekton-releases/operator/latest/release.yaml

After that, we can choose between some installation profiles: lite , all, basic. Let’s choose the all profile:

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/operator/main/config/crs/kubernetes/config/all/operator_v1alpha1_config_cr.yaml

On OpenShift, we can do it using the web UI. OpenShift Console provides the Operator Hub section, where we can find the “Red Hat OpenShift Pipelines” operator. This operator installs Tekton and integrates it with OpenShift. Once you install it, you can e.g. create, manage, and run pipelines in OpenShift Console.

OpenShift Console offers a dedicated section in the menu for Tekton pipelines as shown below.

We can also install the tkn CLI on the local machine to interact with Tekton Pipelines running on the Kubernetes cluster. For example, on macOS, we can do it using Homebrew:

$ brew install tektoncd-cli

How It Works

Create a Tekton Pipeline

Firstly, let’s discuss some basic concepts around Tekton. We can run the same pipeline several times simultaneously. We can trigger that process by creating the PipelineRun object directly, or indirectly e.g. via the tkn CLI command or graphical dashboard. However, each time the PipelineRun object must be created somehow. The Tekton pipeline consists of one or more tasks. Each task is executed by the separated pod. In order to share the data between those pods, we need to use a persistent volume. An example of such data is the app source code cloned from the git repository. We need to attach such a PVC (Persistent Volume Claim) as the pipeline workspace in the PipelineRun definition. The following diagram illustrates that scenario.

Let’s switch to the code. Here’s the YAML manifest with our sample pipeline. The pipeline consists of three tasks. It refers to the tasks from Tekton Hub: git-clone, s2i-java and openshift-client. With these three simple tasks we clone the git repository with the app source code, build the image using the source-to-image approach, and deploy it on the OpenShift cluster. As you see, the pipeline defines a workspace with the source-dir name. Both git-clone and s2i-java share the same workspace. It tags the image with a branch name. The name of the branch is set as the pipeline input parameter.

apiVersion: tekton.dev/v1
kind: Pipeline
metadata:
  name: sample-pipeline
spec:
  params:
    - description: Git branch name
      name: branch
      type: string
    - description: Target namespace
      name: namespace
      type: string
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-spring-kotlin-microservice.git'
        - name: revision
          value: $(params.branch)
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: s2i-java
      params:
        - name: IMAGE
          value: image-registry.openshift-image-registry.svc:5000/$(params.namespace)/sample-spring-kotlin-microservice:$(params.branch)
      runAfter:
        - git-clone
      taskRef:
        kind: ClusterTask
        name: s2i-java
      workspaces:
        - name: source
          workspace: source-dir
    - name: openshift-client
      params:
        - name: SCRIPT
          value: oc process -f openshift/app.yaml -p namespace=$(params.namespace) -p version=$(params.branch) | oc apply -f -
      runAfter:
        - s2i-java
      taskRef:
        kind: ClusterTask
        name: openshift-client
      workspaces:
        - name: manifest-dir
          workspace: source-dir
  workspaces:
    - name: source-dir

Run a Pipeline Several Times Simultaneously

Now, let’s consider the scenario where we run the pipeline several times with the code from different Git branches. Here’s the updated diagram illustrating it. As you see, we need to attach a dedicated volume to the pipeline run. We store there a code related to each of the source branches.

In order to start the pipeline, we can apply the PipelineRun object. The PipelineRun definition must satisfy the previous requirement for a dedicated volume per run. Therefore we need to define the volumeClaimTemplate, which automatically creates the volume and bounds it to the pods within the pipeline. Here’s a sample PipelineRun object for the master branch:

apiVersion: tekton.dev/v1
kind: PipelineRun
metadata:
  generateName: sample-pipeline-
  labels:
    tekton.dev/pipeline: sample-pipeline
spec:
  params:
    - name: branch
      value: master
    - name: namespace
      value: app-master
  pipelineRef:
    name: sample-pipeline
  taskRunTemplate:
    serviceAccountName: pipeline
  workspaces:
    - name: source-dir
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
          volumeMode: Filesystem

With this bash script, we can run our pipeline for every single branch existing in the source repository prefixed by the feature word. It uses the tkn CLI to interact with Tekton

#! /bin/bash

for OUTPUT in $(git branch -r)
do
  branch=$(echo $OUTPUT | sed -e "s/^origin\///")
  if [[ $branch == feature* ]]
  then
    echo "Running the pipeline: branch="$branch
    tkn pipeline start sample-pipeline -p branch=$branch -p namespace=app-$branch -w name=source-dir,volumeClaimTemplateFile=pvc.yaml
  fi
done

The script is available in the sample GitHub repository under the openshift directory. If you want to reproduce my action you need to clone the repository and then execute the run.sh script on your OpenShift cluster.

$ git clone https://github.com/piomin/sample-spring-kotlin-microservice.git
$ cd sample-spring-kotlin-microservice/openshift
$ ./run.sh

The PipelineRun object is responsible not only for starting a pipeline. We can also use it to see the history of runs with detailed logs generated by each task. However, there is also the other side of the coin. The more times we run the pipeline, the more objects we store on the Kubernetes cluster.

Tekton creates a dedicated PVC per each PipelineRun. Such a PVC exists on Kubernetes until we don’t delete the parent PipelineRun.

Pruning Old Pipeline Runs

I just ran the sample-pipeline six times using different feature-* branches. However, you can imagine that there are many more previous runs. It results in many existing PipelineRun and PersistenceVolumeClaim objects on Kubernetes. Fortunately, Tekton provides an automatic mechanism for removing objects from the previous runs. It installs the global CronJob responsible for pruning the PipelineRun objects. We can override the default CronJob configuration in the TektonConfig CRD. I’ll change the CronJob frequency execution from one day to 10 minutes for testing purposes.

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonConfig
metadata:
  name: config
spec:
  # other properties ...
  pruner:
    disabled: false
    keep: 100
    resources:
      - pipelinerun
    schedule: '*/10 * * * *'

We can customize the behavior of the Tekton pruner per each namespace. Thanks to that, it is possible to set the different configurations e.g. for the “production” and “development” pipelines. In order to do that, we need to annotate the namespace with some Tekton parameters. For example, instead of keeping the specific number of previous pipeline runs, we can set the time criterion. The operator.tekton.dev/prune.keep-since annotation allows us to retain resources based on their age. Let’s set it to 1 hour. The annotation requires setting that time in minutes, so the value is 60. We will also override the default pruning strategy to keep-since, which enables removing by time.

kind: Namespace
apiVersion: v1
metadata:
  name: tekton-demo
  annotations:
    operator.tekton.dev/prune.keep-since: "60"
    operator.tekton.dev/prune.strategy: "keep-since"
spec: {}

The CronJob exists in the Tekton operator installation namespace.

$ kubectl get cj -n openshift-pipelines
NAME                           SCHEDULE       SUSPEND   ACTIVE   LAST SCHEDULE   AGE
tekton-resource-pruner-ksdkj   */10 * * * *   False     0        9m44s           24m

As you see, the job runs every ten minutes.

$ kubectl get job -n openshift-pipelines
NAME                                    COMPLETIONS   DURATION   AGE
tekton-resource-pruner-ksdkj-28524850   1/1           5s         11m
tekton-resource-pruner-ksdkj-28524860   1/1           5s         75s

There are no PipelineRun objects older than 1 hour in the tekton-demo namespace.

$ kubectl get pipelinerun -n tekton-demo
NAME                        SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
sample-pipeline-run-2m4rq   True        Succeeded   55m         51m
sample-pipeline-run-4gjqw   True        Succeeded   55m         53m
sample-pipeline-run-5sxcf   True        Succeeded   55m         51m
sample-pipeline-run-667mb   True        Succeeded   34m         30m
sample-pipeline-run-6jqvl   True        Succeeded   34m         32m
sample-pipeline-run-8slfx   True        Succeeded   34m         31m
sample-pipeline-run-bvjq6   True        Succeeded   34m         30m
sample-pipeline-run-d87kn   True        Succeeded   55m         51m
sample-pipeline-run-lrvm2   True        Succeeded   34m         30m
sample-pipeline-run-tx4hl   True        Succeeded   55m         51m
sample-pipeline-run-w5cq8   True        Succeeded   55m         52m
sample-pipeline-run-wn2xx   True        Succeeded   34m         30m

This approach works fine. It minimizes the number of Kubernetes objects stored on the cluster. However, after removing the old objects, we cannot access the full history of pipeline runs. In some cases, it can be useful. Can we do it better? Yes! We can enable Tekton Results.

Using Tekton Results

Install and Configure Tekton Results

Tekton Results is a feature that allows us to archive the complete information for every pipeline run and task run. After pruning the old PipelineRun or TaskRun objects, we can still access the full history using Tekton Results API. It archives all the required information in the form of results and records stored in the database. Before we enable it, we need to prepare several things. In the first step, we need to generate the certificate for exposing Tekton Results REST API over HTTPS. Let’s generate public/private keys with the following openssl command:

$ openssl req -x509 \
    -newkey rsa:4096 \
    -keyout key.pem \
    -out cert.pem \
    -days 365 \
    -nodes \
    -subj "/CN=tekton-results-api-service.openshift-pipelines.svc.cluster.local" \
    -addext "subjectAltName = DNS:tekton-results-api-service.openshift-pipelines.svc.cluster.local"

Then, we can use the key.pem and cert.pem files to create the Kubernetes TLS Secret in the Tekton operator namespace.

$ kubectl create secret tls tekton-results-tls \
    -n openshift-pipelines \
    --cert=cert.pem \
    --key=key.pem

We also need to generate credentials for the Postgres database in Kubernetes Secret form. By default, Tekton Results uses a PostgreSQL database to store data. We can choose between the external instance of that database or the instance managed by the Tekton operator. We will use the internal Postgres installed on our cluster.

$ kubectl create secret generic tekton-results-postgres \
    -n openshift-pipelines \
    --from-literal=POSTGRES_USER=result \
    --from-literal=POSTGRES_PASSWORD=$(openssl rand -base64 20)

Tekton Results requires a persistence volume for storing the logs from pipeline runs.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: tekton-logs
  namespace: openshift-pipelines 
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi

Finally, we can proceed to the main step. We need to create the TektonResults object. I won’t get into the details of that object. You can just create it “as is” on your cluster.

apiVersion: operator.tekton.dev/v1alpha1
kind: TektonResult
metadata:
  name: result
spec:
  targetNamespace: openshift-pipelines
  logs_api: true
  log_level: debug
  db_port: 5432
  db_host: tekton-results-postgres-service.openshift-pipelines.svc.cluster.local
  logs_path: /logs
  logs_type: File
  logs_buffer_size: 32768
  auth_disable: true
  tls_hostname_override: tekton-results-api-service.openshift-pipelines.svc.cluster.local
  db_enable_auto_migration: true
  server_port: 8080
  prometheus_port: 9090
  logging_pvc_name: tekton-logs

Archive Pipeline Runs with Tekton Results

After applying the TektonResult object into the cluster Tekton runs three additional pods in the openshift-pipelines namespace. There are pods with a Postgres database, with Tekton Results API, and a watcher responsible for monitoring and archiving existing PipelineRun objects.

If you run Tekton on OpenShift you will also see the additional “Overview” menu in the “Pipelines” section. It displays the summary of pipeline runs for the selected namespace.

However, the best thing in this mechanism is that we can still access the old pipeline runs with Tekton Results although the PipelineRun objects have been deleted. Tekton Results integrates smoothly with OpenShift Console. The archived pipeline run is marked with the special icon as shown below. We can still access the logs or the results of running every single task in that pipeline.

If we switch to the tkn CLI it doesn’t return any PipelineRun. That’s because all the runs were older than one hour, and thus they were removed by the pruner.

$ kubectl get pipelinerun
NAME                     SUCCEEDED   REASON      STARTTIME   COMPLETIONTIME
sample-pipeline-yiuqhf   Unknown     Running     30s

Consequently, there is also a single PersistentVolumeClaim object.

$ kubectl get pvc
NAME             STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                           AGE
pvc-0f16a64031   Bound    pvc-ba6ea9ef-4281-4a39-983b-0379419076b0   1Gi        RWO            ocs-external-storagecluster-ceph-rbd   41s

Of course, we can still access access details and logs of archived pipeline runs via the OpenShift Console.

Final Thoughts

Tekton is a Kubernetes-native tool for CI/CD pipelines. This approach involves many advantages, but may also lead to some challenges. One of them is running pipelines at scale. In this article, I focused on showing you new Tekton features that address some concerns around the intensive usage of pipelines. Features like pipeline run pruning or Tekton Results archives work fine and smoothly integrate with e.g. the OpenShift Console. Tekton gradually adds new useful features. It is becoming a really interesting alternative to more popular CI/CD tools like Jenkins, GitLab CI, or Circle CI.

The post Running Tekton Pipelines on Kubernetes at Scale appeared first on Piotr's TechBlog.

You can find several posts on my blog about ArgoCD and continuous delivery on Kubernetes. For a quick intro to CI/CD process with Tekton and ArgoCD, you can refer to the following article. For a more advanced approach dedicated to database management in the CD process see the following post.

Prerequisites

In order to do the exercise, you need to have a Kubernetes cluster. Then you need to install the tools we will use today – ArgoCD and Tekton. Here are the installation instructions for Tekton Pipelines and Tekton Triggers. Tekton is optional in our exercise. We will just use it to build the application image after pushing the commit to the repository.

ArgoCD is the key tool today. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

$ helm install my-argo-cd argo/argo-cd -n argocd

Once we install ArgoCD, we can display a list of running pods. You should have a similar result to mine:

$ kubectl get pod
openshift-gitops-application-controller-0                     1/1     Running     0          1m
openshift-gitops-applicationset-controller-654f99c9b4-pwnc2   1/1     Running     0          1m
openshift-gitops-dex-server-5dc77fcb7d-6tkg5                  1/1     Running     0          1m
openshift-gitops-redis-87698688c-r59zf                        1/1     Running     0          1m
openshift-gitops-repo-server-5f6f7f4996-rfdg8                 1/1     Running     0          1m
openshift-gitops-server-dcf746865-tlmlp                       1/1     Running     0          1m

Finally, you also need to have an account on GitHub. In our scenario, ArgoCD will require access to the repository to obtain a list of opened pull requests. Therefore, we need to create a personal access token for authentication over GitHub. In your GitHub profile go to Settings > Developer Settings > Personal access tokens. Choose Tokens (classic) and then click the Generate new token button. Then you should enable the repo scope. Of course, you need to save the value of the generated token. We will create a secret on Kubernetes using that value.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. To do that you need to clone my two GitHub repositories. First of them, contains the source code of our sample app written in Kotlin. The second of them contains configuration managed by ArgoCD with YAML manifests for creating preview environments on Kubernetes. Finally, you should just follow my instructions.

How it works

Let’s describe our scenario. There are two repositories on GitHub. In the repository with app source code, we are creating branches for working on new features. Usually, when we are starting a new branch we are just at the beginning of our work. Therefore, we don’t want to deploy it anywhere. Once we make progress and we have a version for testing, we are creating a pull request. Pull request represents the relation between source and target branches. We may still push commits to the source branch. Once we merge a pull request all the commits from the source branch will also be merged.

After creating a new pull request we want ArgoCD to provision a new preview environment on Kubernetes. Once we merge a pull request we want ArgoCD to remove the preview environment automatically. Fortunately, ArgoCD can monitor pull requests with ApplicationSet generators. Our ApplicationSet will connect to the app source repository to detect new pull requests. However, it will use YAML manifests stored in a different, config repository. Those manifests contain a generic definition of our preview environments. They are written in Helm and may be shared across several different apps and scenarios. Here’s the diagram that illustrates our scenario. Let’s proceed to the technical details.

Using ArgoCD ApplicationSet and Helm Templates

ArgoCD requires access to the GitHub API to detect a current list of opened pull requests. Therefore we will create a Kubernetes Secret that contains our GitHub personal access token:

$ kubectl create secret generic github-token \
  --from-literal=token=<YOUR_GITHUB_PERSON_ACCESS_TOKEN>

In the config repository, we will define a template for our sample preview environment. It is available inside the preview directory. It contains the namespace declaration:

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Values.namespace }}

We are also defining Kubernetes Deployment for a sample app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.name }}
spec:
  replicas: 1
  selector:
    matchLabels:
      app: {{ .Values.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.name }}
    spec:
      containers:
      - name: {{ .Values.name }}
        image: quay.io/pminkows/{{ .Values.image }}:{{ .Values.version }}
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "1024Mi"
            cpu: "1000m"
        ports:
        - containerPort: 8080

Let’s also add the Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.name }}-service
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.name }}
  ports:
  - port: 8080
    name: http-port

Finally, we can create the ArgoCD ApplicationSet with the Pull Request Generator. We are monitoring the app source code repository (1). In order to authenticate over GitHub, we are injecting the Secret containing access token (2). While the ApplicationSet targets the source code repository, the generated ArgoCD Application refers to the config repository (3). It also sets several Helm parameters. The name of the preview namespace is the same as the name of the branch with the preview prefix (4). The app image is tagged with the commit hash (5). We are also setting the name of the app image (6). All the configuration settings are applied automatically by ArgoCD (7).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sample-spring-preview
spec:
  generators:
    - pullRequest:
        github:
          owner: piomin
          repo: sample-spring-kotlin-microservice # (1)
          tokenRef:
            key: token
            secretName: github-token # (2)
        requeueAfterSeconds: 60
  template:
    metadata:
      name: 'sample-spring-{{branch}}-{{number}}'
    spec:
      destination:
        namespace: 'preview-{{branch}}'
        server: 'https://kubernetes.default.svc'
      project: default
      source:
        # (3)
        path: preview/
        repoURL: 'https://github.com/piomin/openshift-cluster-config.git'
        targetRevision: HEAD
        helm:
          parameters:
            # (4)
            - name: namespace
              value: 'preview-{{branch}}'
            # (5)
            - name: version
              value: '{{head_sha}}'
            # (6)
            - name: image
              value: sample-kotlin-spring
            - name: name
              value: sample-spring-kotlin
      # (7)
      syncPolicy:
        automated:
          selfHeal: true

Build Image with Tekton

ArgoCD is responsible for creating a preview environment on Kubernetes and applying the Deployment manifest there. However, we still need to build the image after a push to the source branch. In order to do that, we will create a Tekton pipeline. It’s a very simple pipeline. It just clones the repository and builds the image with the commit hash as a tag.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: sample-kotlin-pipeline
spec:
  params:
    - description: branch
      name: git-revision
      type: string
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-spring-kotlin-microservice.git'
        - name: revision
          value: $(params.git-revision)
        - name: sslVerify
          value: 'false'
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: s2i-java-preview
      params:
        - name: PATH_CONTEXT
          value: .
        - name: TLSVERIFY
          value: 'false'
        - name: MAVEN_CLEAR_REPO
          value: 'false'
        - name: IMAGE
          value: >-
            quay.io/pminkows/sample-kotlin-spring:$(tasks.git-clone.results.commit)
      runAfter:
        - git-clone
      taskRef:
        kind: ClusterTask
        name: s2i-java
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir

This pipeline should be triggered by the push in the app repository. Therefore we have to create the TriggerTemplate and EventListener CRD objects.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: sample-kotlin-spring-trigger-template
  namespace: pminkows-cicd
spec:
  params:
    - default: master
      description: The git revision
      name: git-revision
    - description: The git repository url
      name: git-repo-url
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: sample-kotlin-spring-pipeline-run-
      spec:
        params:
          - name: git-revision
            value: $(tt.params.git-revision)
        pipelineRef:
          name: sample-kotlin-pipeline
        serviceAccountName: pipeline
        workspaces:
          - name: source-dir
            persistentVolumeClaim:
              claimName: kotlin-pipeline-pvc
---
apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: sample-kotlin-spring
spec:
  serviceAccountName: pipeline
  triggers:
    - bindings:
        - kind: ClusterTriggerBinding
          ref: github-push
      name: trigger-1
      template:
        ref: sample-kotlin-spring-trigger-template

After that Tekton automatically creates Kubernetes Service with the webhook for triggering the pipeline.

Finally, we need to set the webhook URL in our GitHub app repository. That’s all that we need to do. Let’s see how it works.

Kubernetes Preview Environments in Action

Creating environment

In the first step, we will create two branches in our sample GitHub repository: branch-a and branch-c. If we push some changes into each of those branches, our pipeline should be triggered by the webhook. It will build the image from the source branch and push it to the remote registry.

As you see, our pipeline was running two times.

Here’s the Quay registry with our sample app images. They are tagged using the commit hash.

Now, we can create pull requests for our branches. As you see I have two pull requests in the sample repository.

Let’s take a look at one of our pull requests. Firstly, pay attention to the pull request id (55) and a list of commits assigned to the pull request.

ArgoCD monitors a list of opened PRs via ApplicationSet. Each time it detects a new PR it creates a dedicated ArgoCD Application for synchronizing YAML manifests stored in the Git config repository with the target Kubernetes cluster. We have two opened PRs, so there are two applications in ArgoCD.

We can take a look at the ArgoCD Application details. As you see it creates the namespace containing Kubernetes Deployment and Service for our app.

Let’s display a list of running in one of our preview namespaces:

$ kubectl get po -n preview-branch-a
NAME                                    READY   STATUS    RESTARTS   AGE
sample-spring-kotlin-5c7cc45bc7-wck78   1/1     Running   0          22m

Let’s verify the tag of the image used in the pod:

Adding commits to the existing PR

What about making some changes in one of our preview branches? Our latest commit with the “Make some changes” title will be automatically included in the PR.

ArgoCD ApplicationSet will detect a commit in the pull request. Then it will update the ArgoCD Application with the latest commit hash (71d05d8). As a result, it will try to run a new pod containing the latest version of the app. In the meantime, our pipeline is building a new image staged by the commit hash. As you see, the image is not available yet.

Let’s display a list of running pods in the preview-branch-c namespace:

$ kubectl get po -n preview-branch-c
NAME                                    READY   STATUS             RESTARTS   AGE
sample-spring-kotlin-67f6947c89-xn2r8   1/1     Running            0          29m
sample-spring-kotlin-6d844c8c94-qjrr4   0/1     ImagePullBackOff   0          24s

Once the pipeline will finish the build, it pushes the image to the Quay registry:

And the latest version of the app from the branch-c is available on Kubernetes:

Now, you can close or merge the pull request. As a result, ArgoCD will automatically remove our preview namespace with the app.

Final Thoughts

In this article, I showed you how to create and manage preview environments on Kubernetes in the GitOps way with ArgoCD. In this concept, a preview environment exists on Kubernetes as long as the particular pull request lives in the GitHub repository. ArgoCD uses a global, generic template for creating such an environment. Thanks to that, we can have a single, shared process across the whole organization.

The post Preview Environments on Kubernetes with ArgoCD appeared first on Piotr's TechBlog.

If you want to compare Microcks with other testing tools, you may be interested in my article about contract testing with Quarkus and Pact available here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

Install Microcks on Kubernetes

We can install Microcks on Kubernetes with Helm chart or with the operator. Since I’m using OpenShift as a Kubernetes platform in this exercise, the simplest way is through the operator. Assuming we have already installed the OLM (Operator Lifecycle Manager) on Kubernetes we need to apply the following YAML manifest:

apiVersion: v1
kind: Namespace
metadata:
  name: microcks
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: operatorgroup
  namespace: microcks
spec:
  targetNamespaces:
  - microcks
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: my-microcks
  namespace: microcks
spec:
  channel: stable
  name: microcks
  source: operatorhubio-catalog
  sourceNamespace: olm

With OpenShift we can install the Microcks operator using the UI dashboard. Once we do it, we need to create the object responsible for Microcks installation. In the OpenShift console, we need to click the “MicrocksInstall” link as shown below.

Microcks requires some additional staff like Keycloak or MongoDB to be installed on the cluster. Here’s the YAML manifest responsible for installation on Kubernetes.

apiVersion: microcks.github.io/v1alpha1
kind: MicrocksInstall
metadata:
  name: my-microcksinstall
  namespace: microcks
spec:
  name: my-microcksinstall
  version: 1.7.0
  microcks:
    replicas: 1
  postman:
    replicas: 1
  keycloak:
    install: true
    persistent: true
    volumeSize: 1Gi
  mongodb:
    install: true
    persistent: true
    volumeSize: 2Gi

By applying the YAML manifest with the MicrocksInstall object, we will start the installation process. We should have the following list of running pods if it finished successfully:

$ kubectl get pod -n microcks
NAME                                                     READY   STATUS    RESTARTS   AGE
microcks-ansible-operator-56ddbdcccf-6lrdl               1/1     Running   0          4h14m
my-microcksinstall-76b67f4f77-jdcj7                      1/1     Running   0          4h13m
my-microcksinstall-keycloak-79c5f68f45-5nm69             1/1     Running   0          4h13m
my-microcksinstall-keycloak-postgresql-97f69c476-km6p2   1/1     Running   0          4h13m
my-microcksinstall-mongodb-846f7c7976-tl5jh              1/1     Running   0          4h13m
my-microcksinstall-postman-runtime-574d4bf7dc-xfct7      1/1     Running   0          4h13m

In order to prepare the contract testing process on Kubernetes with Microcks we need to access it dashboard. It is exposed by the my-microcksinstall service under the 8080 port. I’m accessing it through the OpenShift Route object. If you use Kubernetes you can enable port forwarding (the kubectl port-forward command) for that service or expose it through the Ingress object.

Here’s the Microcks UI dashboard. In the first step, we need to import our API documentation and samples by clicking the Importers button. Before we do it, we need to prepare the document in one of the supported specifications. In our case, it is the OpenAPI specification.

Create the Provider Side App

We create a simple app with Quarkus that exposes some REST endpoints. In order to access it go to the employee-service directory in our sample Git repository. Here’s the controller class responsible for the endpoints implementation.

@Path("/employees")
@Produces(MediaType.APPLICATION_JSON)
public class EmployeeController {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(EmployeeController.class);

   @Inject
   EmployeeRepository repository;

   @POST
   public Employee add(@Valid Employee employee) {
      LOGGER.info("Employee add: {}", employee);
      return repository.add(employee);
   }

   @Path("/{id}")
   @GET
   public Employee findById(@PathParam("id") Long id) {
      LOGGER.info("Employee find: id={}", id);
      return repository.findById(id);
   }

   @GET
   public Set<Employee> findAll() {
      LOGGER.info("Employee find");
      return repository.findAll();
   }

   @Path("/department/{departmentId}")
   @GET
   public Set<Employee> findByDepartment(@PathParam("departmentId") Long departmentId) {
      LOGGER.info("Employee find: departmentId={}", departmentId);
      return repository.findByDepartment(departmentId);
   }

   @Path("/organization/{organizationId}")
   @GET
   public Set<Employee> findByOrganization(@PathParam("organizationId") Long organizationId) {
      LOGGER.info("Employee find: organizationId={}", organizationId);
      return repository.findByOrganization(organizationId);
   }

}

If we add the module responsible for generating and exposing OpenAPI documentation we can easily access it under the /q/openapi path. In order to achieve to include the following Maven dependency:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-smallrye-openapi</artifactId>
</dependency>

Here’s the OpenAPI descriptor automatically generated by Quarkus for our employee-service app.

openapi: 3.0.3
info:
  title: employee-service API
  version: "1.2"
paths:
  /employees:
    get:
      tags:
      - Employee Controller
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
    post:
      tags:
      - Employee Controller
      requestBody:
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Employee'
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Employee'
  /employees/department/{departmentId}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: departmentId
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
  /employees/organization/{organizationId}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: organizationId
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                uniqueItems: true
                type: array
                items:
                  $ref: '#/components/schemas/Employee'
  /employees/{id}:
    get:
      tags:
      - Employee Controller
      parameters:
      - name: id
        in: path
        required: true
        schema:
          format: int64
          type: integer
      responses:
        "200":
          description: OK
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Employee'
components:
  schemas:
    Employee:
      required:
      - organizationId
      - departmentId
      - name
      - position
      type: object
      properties:
        id:
          format: int64
          type: integer
        organizationId:
          format: int64
          type: integer
        departmentId:
          format: int64
          type: integer
        name:
          pattern: \S
          type: string
        age:
          format: int32
          maximum: 100
          minimum: 1
          type: integer
        position:
          pattern: \S
          type: string

We need to add the examples section to each responses and parameters section. Let’s begin with the GET /employees endpoint. It returns all existing employees. The name of the example is not important. You can set any name you want. As the returned value we set the JSON array with three Employee objects.

/employees:
  get:
    tags:
      - Employee Controller
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              uniqueItems: true
              type: array
              items:
                $ref: '#/components/schemas/Employee'
            examples:
              all_persons:
                value: |-
                  [
                     {"id": 1, "name": "Test User 1", "age": 20, "organizationId": 1, "departmentId": 1, "position": "developer"},
                     {"id": 2, "name": "Test User 2", "age": 30, "organizationId": 1, "departmentId": 2, "position": "architect"},
                     {"id": 3, "name": "Test User 3", "age": 40, "organizationId": 2, "departmentId": 3, "position": "developer"},
                  ]

For comparison, let’s take a look at the OpenAPI docs for the POST /employees endpoint. It returns a single JSON object as a response. We also had to add examples in the requestBody section.

/employees:
  post:
    tags:
      - Employee Controller
    requestBody:
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Employee'
          examples:
            add_employee:
              summary: Hire a new employee
              description: Should return 200
              value: '{"name": "Test User 4", "age": 50, "organizationId": 2, "departmentId": 3, "position": "tester"}'
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/Employee'
            examples:
              add_employee:
                value: |-
                  {"id": 4, "name": "Test User 4", "age": 50, "organizationId": 2, "departmentId": 3, "position": "tester"}

Finally, we can proceed to the endpoint GET /employees/department/{departmentId}, which returns all the employees assigned to the particular department. This endpoint is called by another app – department-service. We need to add the example value of the path variable referring to the id of a department. The same as before, we also have to include an example JSON in the responses section.

/employees/department/{departmentId}:
  get:
    tags:
      - Employee Controller
    parameters:
      - name: departmentId
        in: path
        required: true
        schema:
          format: int64
          type: integer
        examples:
          find_by_dep_1:
            summary: Main id of department
            value: 1
    responses:
      "200":
        description: OK
        content:
          application/json:
            schema:
              uniqueItems: true
              type: array
              items:
                $ref: '#/components/schemas/Employee'
            examples:
              find_by_dep_1:
                value: |-
                  [
                    { "id": 1, "name": "Test User 1", "age": 20, "organizationId": 1, "departmentId": 1, "position": "developer" }
                  ]

Create the Consumer Side App

As I mentioned before, we have another app – department-service. It consumer endpoint exposed by the employee-service. Here’s the implementation of the Quarkus declarative REST client responsible for calling the GET /employees/department/{departmentId} endpoint:

@ApplicationScoped
@Path("/employees")
@RegisterRestClient(configKey = "employee")
public interface EmployeeClient {

    @GET
    @Path("/department/{departmentId}")
    @Produces(MediaType.APPLICATION_JSON)
    List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId);

}

That client is then used by the department-service REST controller to return all the employees in the particular organization with the division into departments.

@Path("/departments")
@Produces(MediaType.APPLICATION_JSON)
public class DepartmentController {

    private static final Logger LOGGER = LoggerFactory
       .getLogger(DepartmentController.class);

    @Inject
    DepartmentRepository repository;
    @Inject
    @RestClient
    EmployeeClient employeeClient;

    // ... implementation of other endpoints

    @Path("/organization/{organizationId}/with-employees")
    @GET
    public Set<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
        LOGGER.info("Department find: organizationId={}", organizationId);
        Set<Department> departments = repository.findByOrganization(organizationId);
        departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
        return departments;
    }

}

Finally, we need to configure the base URI for the client in the Quarkus application.properties file. We can also set that address in the corresponding environment variable QUARKUS_REST_CLIENT_EMPLOYEE_URL.

quarkus.rest-client.employee.url = http://employee:8080

Test API with Microcks

Once we implemented our apps, we can back to the Microcks dashboard. In the Importers section import your openapi.yml file.

Once you will import the OpenAPI definition you can go to the APIs | Services section. You should already have the record with your API description as shown below. The current version of our is 1.2. The employee-service exposes five REST endpoints.

We can display the details of the service by clicking on the record or on the Details button. It displays a list of available endpoints with a number of examples declared in the OpenAPI manifest.

Let’s display the details of the GET /employees/department/{departmentId} endpoint. This endpoint is called by the department-service. Microcks creates the mock on Kubernetes for the purposes of contract testing. The mock is available inside Kubernetes and as well as outside it through the OpenShift Route. Now, our goal is to verify the compliance of our mock with the real service.

In order to verify the Microcks mock with the real service, we need to deploy our employee-service app on Kubernetes. We can easily do it using the Quarkus Kubernetes extension. Thanks to that it is possible to build the app, build the image and deploy it on Kubernetes or OpenShift using a single Maven command. Here’s the dedicated Maven profile for that. For OpenShift, we need to include the single dependency quarkus-openshift, and set the property quarkus.kubernetes.deploy to true.

<profile>
  <id>openshift</id>
  <activation>
    <property>
      <name>openshift</name>
    </property>
  </activation>
  <dependencies>
    <dependency>
      <groupId>io.quarkus</groupId>
      <artifactId>quarkus-openshift</artifactId>
    </dependency>
  </dependencies>
  <properties>
    <quarkus.kubernetes.deploy>true</quarkus.kubernetes.deploy>
    <quarkus.profile>openshift</quarkus.profile>
    <quarkus.container-image.builder>openshift</quarkus.container-image.builder>
  </properties>
</profile>

Then, we just need to activate the openshift profile during the Maven build. Go to the employee-service directory and execute the following command:

$ mvn clean package -Popenshift

The internal address of our app is defined by the name of the Kubernetes service:

$ oc get svc
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
employee-service   ClusterIP   172.30.141.85   <none>        80/TCP    27h

Finally, we can our samples against real endpoints. We need to set the address of our service in the Test Endpoint field. Then, we choose OPEN_API_SCHEMA as the Runner. Finally, we can execute the test by clicking the Launch test button.

Viewing and Analysing Test Results

Once we run the test we can display the results with the Microcks dashboard. By default, it calls all the endpoints defined in the OpenAPI manifest. We can override this behavior and choose a list of endpoints that should be tested.

We can verify the details of each test. For example, we can display a response from our employee-service running on the cluster for the GET /employees/department/{departmentId} endpoint.

We can automate the testing process with Microcks using its CLI. You can read more details about it in the docs. In order to execute the test in a similar way as we did before via the Microcks dashboard, we need to use the test command. It requires several input arguments. We need to pass the name and version of our API, the URI of the test endpoint, and the type of runner. By default, Microcks creates a client in Keycloak. The client id is microcks-serviceaccount. You need to log in to your Keycloak instance and copy the value of the client secret. As the microcksURL argument, we need to pass the external address of Microcks with the /api path. If you would run it inside the Kubernetes cluster, you could run the internal address http://my-microcksinstall.microcks.svc.cluster.local:8080/api.

$ microcks-cli test 'employee-service API:1.2' \
  http://employee-service.demo-microcks.svc.cluster.local \
  OPEN_API_SCHEMA \
  --microcksURL=https://my-microcksinstall-microcks.apps.pvxvtsz4.eastus.aroapp.io/api \
  --keycloakClientId microcks-serviceaccount \
  --keycloakClientSecret ab54d329-e435-41ae-a900-ec6b3fe15c54

Once we run a test, Microcks calculates the conformance score after finishing. It is a kind of grade that estimates how your API contract is actually covered by the samples you’ve attached to it. It is computed based on the number of samples you’ve got on each operation, and the complexity of dispatching rules of this operation. As you there is still a place for improvement in my case

Verify Contract on the Consumer Side

Let’s create a simple test that verifies the contract with employee-service in the department-service. It will call the department-service GET /departments/organization/{organizationId}/with-employees endpoint that interacts with the employee-service. We won’t mock the REST client.

@QuarkusTest
public class DepartmentExternalContractTests {

    @Test
    void findByOrganizationWithEmployees() {
        when().get("/departments/organization/{organizationId}/with-employees", 1L).then()
                .statusCode(200)
                .body("size()", notNullValue());
    }

}

Instead of mocking the client directly in the test, we use the mock endpoint exposed by Microcks. We will use the internal of this mock for the employee-service is https://my-microcksinstall.microcks.svc.cluster.local/rest/employee-service+API/1.2/employees. We can leverage the Quarkus environment variable to set the address for the client. As I mentioned before we can use the QUARKUS_REST_CLIENT_EMPLOYEE_URL environment variable.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: maven-contract-tests
spec:
  params:
    - default: >-
        image-registry.openshift-image-registry.svc:5000/openshift/java:latest
      description: Maven base image
      name: MAVEN_IMAGE
      type: string
    - default: .
      description: >-
        The context directory within the repository for sources on which we want
        to execute maven goals.
      name: CONTEXT_DIR
      type: string
    - name: EMPLOYEE_URL
      default: http://my-microcksinstall.microcks.svc.cluster.local:8080/rest/employee-service+API/1.2
  steps:
    - image: $(params.MAVEN_IMAGE)
      name: mvn-command
      env:
        - name: QUARKUS_REST_CLIENT_EMPLOYEE_URL
          value: $(params.EMPLOYEE_URL)
      resources: {}
      script: >
        #!/usr/bin/env bash

        /usr/bin/mvn clean package -Pmicrocks

      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
  workspaces:
    - name: source

As you see in the code above, we are activating the microcks profile during the Maven build. That’s because we want to run the test defined in DepartmentExternalContractTests class only if the build is performed on Kubernetes.

<profile>
  <id>microcks</id>
  <activation>
    <property>
      <name>microcks</name>
    </property>
  </activation>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <version>${surefire-plugin.version}</version>
        <configuration>
          <includes>
            <include>**/DepartmentExternalContractTests.java</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>

Finally, we can define and start a pipeline that clones our Git repository and runs the test against the endpoint mocked by the Microcks. Here’s the definition of our pipeline.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: microcks-pipeline
spec:
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-quarkus-microservices.git'
        - name: revision
          value: master
        - name: submodules
          value: 'true'
        - name: depth
          value: '1'
        - name: sslVerify
          value: 'false'
        - name: crtFileName
          value: ca-bundle.crt
        - name: deleteExisting
          value: 'true'
        - name: verbose
          value: 'true'
        - name: gitInitImage
          value: >-
            registry.redhat.io/openshift-pipelines/pipelines-git-init-rhel8@sha256:a538c423e7a11aae6ae582a411fdb090936458075f99af4ce5add038bb6983e8
        - name: userHome
          value: /tekton/home
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: maven-contract-tests
      params:
        - name: MAVEN_IMAGE
          value: >-
            image-registry.openshift-image-registry.svc:5000/openshift/java:latest
        - name: CONTEXT_DIR
          value: department-service
        - name: EMPLOYEE_URL
          value: >-
            http://my-microcksinstall.microcks.svc.cluster.local:8080/rest/employee-service+API/1.2
      runAfter:
        - git-clone
      taskRef:
        kind: Task
        name: maven-contract-tests
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir

Final Thoughts

In this article, I described just one of several possible scenarios for contract testing on Kubernetes with Microcks. You can as well use it to test interactions, for example with Kafka or with GraphQL endpoints. Microcks can be a central point for testing contracts of apps running on Kubernetes. It is able to verify contracts by calling real endpoints exposed by the apps. It provides UI for visualizing and running tests, as well as CLI for automation.

The post Contract Testing on Kubernetes with Microcks appeared first on Piotr's TechBlog.

If you are interested in more details about Spring Boot native compilation please refer to my article Microservices on Knative with Spring Boot and GraalVM.

Prerequisites

Native compilation for Java is a memory-intensive process. Therefore, we need to reserve at least 8GB for our Kubernetes cluster. We also have to install Tekton and Knative there, so it is worth having even more memory.

1. Install Knative Serving – we will use the latest version of Knative (1.3). Go to the following site for the installation manual. Once you did that, you can just verify if it works with the following command:

$ kubectl get pods -n knative-serving

2. Install Tekton – you can go to that site for more details. However, there is just a single command to install it:

$ kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then go to the callme-service directory. After that, you should just follow my instructions.

Spring Boot for Native GraalVM

In order to expose the REST endpoints, we need to include Spring Boot Starter Web. Our service also stored data in the H2 database, so we include Spring Boot Starter Data JPA. The last dependency is for native compilation support. The current version of Spring Native is 0.11.3.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.experimental</groupId>
  <artifactId>spring-native</artifactId>
  <version>0.11.3</version>
</dependency>

Let’s switch to the code. It is our model class exposed by the REST endpoint. It contains the current date, the name of the Kubernetes pod, and the version number.

@Entity
public class Callme {

   @Id
   @GeneratedValue
   private Integer id;
   @Temporal(TemporalType.TIMESTAMP)
   private Date addDate;
   private String podName;
   private String version;

   // getters, setters, constructor ...

}

There is a single endpoint that creates an event, stores it in the database and returns it as a result. The name of the pod and the name of the namespace are taken directly from Kubernetes Deployment. We use the version number from Maven pom.xml as the application version.

@RestController
@RequestMapping("/callme")
public class CallmeController {

   @Value("${spring.application.name}")
   private String appName;
   @Value("${POD_NAME}")
   private String podName;
   @Value("${POD_NAMESPACE}")
   private String podNamespace;
   @Autowired
   private CallmeRepository repository;
   @Autowired(required = false)
   BuildProperties buildProperties;

   @GetMapping("/ping")
   public String ping() {
      Callme c = repository.save(new Callme(new Date(), podName,
            buildProperties != null ? buildProperties.getVersion() : null));
      return appName +
            " v" + c.getVersion() +
            " (id=" + c.getId() + "): " +
            podName +
            " in " + podNamespace;
   }

}

In order to use the Maven version, we need to generate the build-info.properties file during the build. Therefore we should add the build-info goal in the spring-boot-maven-plugin execution properties. If you would like to build a native image locally just set the configuration environment property BP_NATIVE_IMAGE to true. Then you can just run the command mvn spring-boot:build-image.

<plugin>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-maven-plugin</artifactId>
  <executions>
    <execution>
      <goals>
        <goal>build-info</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <image>
      <builder>paketobuildpacks/builder:tiny</builder>
      <env>
        <BP_NATIVE_IMAGE>true</BP_NATIVE_IMAGE>
      </env>
    </image>
  </configuration>
</plugin>

Create Tekton Pipelines

Our pipeline consists of three tasks. In the first of them, we are cloning the source code repository with the Spring Boot application. In the second step, we are building the image natively on Kubernetes using the Cloud Native Builpacks task. After building the image we are pushing to the remote registry. Finally, we are running the image on Kubernetes as the Knative service.

Firstly, we need to install the Tekton git-clone task:

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/git-clone/0.4/git-clone.yaml

We also need the buildpacks task that allows us to run Cloud Native Buildpacks on Kubernetes:

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/master/task/buildpacks/0.3/buildpacks.yaml

Finally, we are installing the kubernetes-actions task to deploy Knative Service using the YAML manifest.

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/kubernetes-actions/0.2/kubernetes-actions.yaml

All our tasks are ready. Let’s just verify it:

$ kubectl get task                     
NAME                 AGE
buildpacks           1m
git-clone            1m
kubernetes-actions   33s

Finally, we are going to create a Tekton pipeline. In the buildpacks task reference, we need to set several parameters. Since we have two Maven modules in the repository, we first need to set the working directory to the callme-service (1). Also, we use Paketo Buildpacks, so we change the default builder image to the paketobuildpacks/builder:base (2). Finally, we need to enable native build for Cloud Native Buildpacks. In order to do that, we should set the environment variable BP_NATIVE_IMAGE to true (3). After building and pushing the image we can deploy it on Kubernetes (4).

apiVersion: tekton.dev/v1alpha1
kind: Pipeline
metadata:
  name: build-spring-boot-pipeline
spec:
  params:
    - description: image URL to push
      name: image
      type: string
  tasks:
    - name: fetch-repository
      params:
        - name: url
          value: 'https://github.com/piomin/sample-spring-boot-graalvm.git'
        - name: subdirectory
          value: ''
        - name: deleteExisting
          value: 'true'
      taskRef:
        kind: Task
        name: git-clone
      workspaces:
        - name: output
          workspace: source-workspace
    - name: buildpacks
      params:
        - name: APP_IMAGE
          value: $(params.image)
        - name: SOURCE_SUBPATH # (1)
          value: callme-service
        - name: BUILDER_IMAGE # (2)
          value: 'paketobuildpacks/builder:base'
        - name: ENV_VARS # (3)
          value:
            - BP_NATIVE_IMAGE=true
      runAfter:
        - fetch-repository
      taskRef:
        kind: Task
        name: buildpacks
      workspaces:
        - name: source
          workspace: source-workspace
        - name: cache
          workspace: cache-workspace
    - name: deploy
      params:
        - name: args # (4)
          value: 
            - apply -f callme-service/k8s/
      runAfter:
        - buildpacks
      taskRef:
        kind: Task
        name: kubernetes-actions
      workspaces:
        - name: manifest-dir
          workspace: source-workspace
  workspaces:
    - name: source-workspace
    - name: cache-workspace

In order to push the image into the remote secure registry, you need to create a Secret containing your username and password.

$ kubectl create secret docker-registry docker-user-pass \
    --docker-username=<USERNAME> \
    --docker-password=<PASSWORD> \
    --docker-server=https://index.docker.io/v1/

After that, you should create ServiceAccount that uses a newly created Secret. As you probably figured out our pipeline uses that ServiceAccount.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: buildpacks-service-account
secrets:
  - name: docker-user-pass

Deploy Knative Service with gradual rollouts

Here’s the YAML manifest with our Knative Service for the 1.0 version. It is a very simple definition. The only additional thing we need to do is to inject the name of the pod and namespace into the container.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: callme-service
spec:
  template:
    spec:
      containers:
      - name: callme
        image: piomin/callme-service:1.0
        ports:
          - containerPort: 8080
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

In order to inject the name of the pod and namespace into the container, we use Downward API. This Kubernetes feature is by default disabled on Knative. To enable it we need to add the property kubernetes.podspec-fieldref with value enabled in the config-features ConfigMap.

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-features
  namespace: knative-serving
data:
  kubernetes.podspec-fieldref: enabled

Now, let’s run our pipeline for the 1.0 version of our sample application. To run it, you should also create a PersistentVolumeClaim with the name tekton-workspace-pvc.

apiVersion: tekton.dev/v1alpha1
kind: PipelineRun
spec:
  params:
    - name: image
      value: 'piomin/callme-service:1.0'
  pipelineRef:
    name: build-spring-boot-pipeline
  serviceAccountName: buildpacks-service-account
  workspaces:
    - name: source-workspace
      persistentVolumeClaim:
        claimName: tekton-workspace-pvc
      subPath: source
    - name: cache-workspace
      persistentVolumeClaim:
        claimName: tekton-workspace-pvc
      subPath: cache

Finally, it is time to release a new version of our application – 1.1. Firstly, you should change the version number in Maven pom.xml.

We should also change the version number in the k8s/ksvc.yaml manifest. However, the most important thing is related to the annotation serving.knative.dev/rollout-duration. It enables gradual rollouts for Knative Service. The value 300s of this parameter means that our rollout to the latest revision will take exactly 300 seconds. Knative is going to roll out to 1% of traffic first, and then in equal incremental steps for the rest of the assigned traffic. In that case, it will increase the traffic to the latest service by 1% every 3 seconds.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: callme-service
  annotations:
    serving.knative.dev/rollout-duration: "300s"
spec:
  template:
    spec:
      containers:
      - name: callme
        image: piomin/callme-service:1.1
        ports:
          - containerPort: 8080
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

Verify Canary Release with Knative

Let’s verify our canary release process built with Knative and Tekton. Once you run the pipeline for the 1.0 and 1.1 version of our sample application, you can display a list of Knative Service. As you the latest revision is callme-service-00002, but the rollout is still in progress:

We can send some test requests to our Knative Service. For me, Knative is available on localhost:80, since I’m using Kubernetes on the Docker Desktop. The only thing I need to do is to set the name URL of the service in the Host header.

$ curl http://localhost:80/callme/ping \
  -H "Host:callme-service.default.example.com"

Here are some responses. As you see, the first two requests have been processed by the 1.0 version of our application. While the last request by the 1.1 version.

Let’s verify the current percentage traffic distribution between the two revisions. Currently, it is 52% to 1.1 and 48% to 1.0.

Finally, after the rollout procedure is finished you should get the same response as me.

Final Thoughts

As you see the process of canary release with Knative is very simple. You only need to set a single annotation on the Knative Service. You can compare it for example with Argo Rollouts which allows us to perform progressive traffic shifting for a standard Kubernetes Deployment.

The post Canary Release on Kubernetes with Knative and Tekton appeared first on Piotr's TechBlog.

Introduction

Our pipeline consists of three steps. In the first step, it clones the source code from the Git repository. Then it builds the application using Maven. Thanks to Quarkus Kubernetes features, we don’t need to create YAML manifests by ourselves. Quarkus will generate them based on the source code and configuration properties during the build. It can also build and publish images in the same step (but that’s just an option, disabled by default). Maybe this feature is not suitable for production, but we may use it here to simplify our pipeline. Anyway, once we generate the Kubernetes manifest we may proceed to the next step – validation with Datree. Here’s a picture to illustrate our process.

Now, some words about Datree. We can easily install it locally. After that, we may validate a single or several YAML manifests using the datree CLI. Of course, the main goal is to include Datree’s policy check as part of our CI/CD pipeline. Thanks to that, we hope to prevent Kubernetes misconfigurations from reaching production. Let’s begin!

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then go to the person-service directory. After that, you should just follow my instructions

Using Datree with Kubernetes Manifests

In order to do a quick intro to Datree, we will first try it locally. You need to install the datree CLI. I installed it on my macOS with Homebrew.

$ brew tap datreeio/datree
$ brew install datreeio/datree/datree

You also need to have JDK and Maven on your machine in order to build the application from the source code. Now, go to the person-service directory. Build the application with the following command:

$ mvn clean package

Since our application includes the quarkus-kubernetes Maven module we don’t need to create a YAML manifest manually. Quarkus generates it during the build. You can find the generated kubernetes.yml file in the target/kubernetes directory. Now, let’s just verify it using the datree test command as shown below.

$ datree test target/kubernetes/kubernetes.yml

Here’s the result. It doesn’t look very good for our current configuration What’s important for now, you will find the link to your Datree account at the bottom. Just click it and then sign up. You will see the list of your validation policies.

By default, there are 34 rules available in the Default policy. Not all of them are active. For example, we may enable the following rule that verifies if the owner label exists.

Moreover, we may create our own custom rule. To do that we first need to navigate to the account settings and enable the option Policy as Code. Then download the policy file.

Let’s say we would like to enable Prometheus metrics for our Deployment on Kubernetes. To enable scraping we should add the following annotation to the Deployment manifest:

metadata:
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/path: /g/metrics
    prometheus.io/port: "8080"

Now, let’s create the rule that verifies if those annotations have been added to the Deployment object. In order to do that, we need to add the following custom rule to the policies.yaml file. I think it is quite intuitive and doesn’t require a detailed explanation. If you are interested in more details please refer to the documentation.

customRules:
  - identifier: ENABLE_PROMETHEUS_LABELS
    name: Ensure the Prometheus labels are set
    defaultMessageOnFailure: Prometheus scraping not enabled!
    schema:
      properties:
        metadata:
          properties:
            annotations:
              properties:
                prometheus.io/scrape:
                  enum:
                    - "true"
              required:
                - prometheus.io/scrape
                - prometheus.io/path
                - prometheus.io/port
          required:
            - annotations

Finally, we need to add our custom rule to the Default policy.

policies:
  - name: Default
    isDefault: true
    rules:
      - identifier: ENABLE_PROMETHEUS_LABELS
        messageOnFailure: Prometheus scraping not enabled!
      - ...

A modified policy should be published to our Datree account using the following command:

$ datree publish policies.yaml

Create Datree Tekton Task

There are three tasks used in our example Tekton pipeline. The first two of them, git-clone and maven, are the standard tasks and we may easily get them from the Tekton Hub. You won’t find a task dedicated to Datree in the hub. However, we can easily create it by ourselves. I’ll use a very minimalistic version of that task only with options required in our case. If you need a more advanced definition you can find an example here.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: datree
spec:
  description: >-
    The Datree (datree.io) task
  workspaces:
    - name: source
  params:
    - name: yamlSrc
      description: Path for the yaml files relative to the workspace path
      type: string
      default: "./*.yaml"
    - name: policy
      description: Specify which policy to execute
      type: string
      default: "Default"
    - name: token
      description: The Datree token to the account on datree.io
      type: string
  steps:
    - name: datree-test
      image: datree/datree
      workingDir: $(workspaces.source.path)
      env:
        - name: DATREE_TOKEN
          value: $(params.token)
        - name: WORKSPACE_PATH
          value: $(workspaces.source.path)
        - name: PARAM_YAMLSRC
          value: $(params.yamlSrc)
        - name: PARAM_POLICY
          value: $(params.policy)
      script: |
        #!/usr/bin/env sh
        POLICY_FLAG=""

        if [ "${PARAM_POLICY}" != "" ] ; then
          POLICY_FLAG="--policy $PARAM_POLICY"
        fi

        /datree test $WORKSPACE_PATH/$PARAM_YAMLSRC $POLICY_FLAG

Now, we can apply the task to the Kubernetes cluster. You can find the file with the task definition in the repository here: .tekton/datree-task.yaml. First, let’s create a test namespace for our current example.

$ kubectl create ns datree
$ kubectl apply -f .tekton/datree-task.yaml -n datree

Build and Run Tekton Pipeline with Datree

Now, we have all tasks required to compose our pipeline.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: datree-pipeline
  namespace: datree
spec:
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/piomin/sample-quarkus-applications.git'
        - name: userHome
          value: /tekton/home
      taskRef:
        kind: Task
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: maven
      params:
        - name: GOALS
          value:
            - clean
            - package
        - name: CONTEXT_DIR
          value: /person-service
      runAfter:
        - git-clone
      taskRef:
        kind: Task
        name: maven
      workspaces:
        - name: source
          workspace: source-dir
        - name: maven-settings
          workspace: maven-settings
    - name: datree
      params:
        - name: yamlSrc
          value: /person-service/target/kubernetes/kubernetes.yml
        - name: policy
          value: Default
        - name: token
          value: ${YOUR_TOKEN} # put your Datree token here
      runAfter:
        - maven
      taskRef:
        kind: Task
        name: datree
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir
    - name: maven-settings

Before running the pipeline you need to obtain your token from the Datree account. Once again, go to your settings and copy the token.

Now, you need to set it as a pipeline parameter token. We also need to pass the location of the Kubernetes manifest inside a workspace. As I mentioned before, it is automatically generated by Quarkus under the path target/kubernetes/kubernetes.yml. In order to run the Tekton pipeline on Kubernetes, we should create the PipelineRun object.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: datree-pipeline-run-
  namespace: datree
  labels:
    tekton.dev/pipeline: datree-pipeline
spec:
  pipelineRef:
    name: datree-pipeline
  serviceAccountName: pipeline
  workspaces:
    - name: source-dir
      persistentVolumeClaim:
        claimName: tekton-workspace-pvc
    - configMap:
        name: maven-settings
      name: maven-settings

That’s all. Let’s just run our pipeline by applying the Tekton PipelineRun object.

$ kubectl apply -f .tekton/pipeline-run.yaml -n datree

Personally, I’m using OpenShift for running Tekton pipelines. That’s why I can easily verify the result with OpenShift Console.

Let’s just see how it looks. The pipeline has failed due to Kubernetes manifest validation errors detected by the Datree task. That’s exactly what we wanted to achieve. Now, let’s try to fix some errors to make the pipeline run finish successfully.

Customize Kubernetes Deployment with Quarkus

We will analyze a report generated by the Datree task issue by issue. But before we do that, let’s see the Deployment file generated by Quarkus during the build.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app.quarkus.io/commit-id: 130b6957fa6b21fac87fe65a40f4dee75e0e39c3
    app.quarkus.io/build-timestamp: 2022-02-17 - 12:12:10 +0000
  labels:
    app.kubernetes.io/name: person-service
    app.kubernetes.io/version: "1.0"
  name: person-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: person-service
      app.kubernetes.io/version: "1.0"
  template:
    metadata:
      annotations:
        app.quarkus.io/commit-id: 130b6957fa6b21fac87fe65a40f4dee75e0e39c3
        app.quarkus.io/build-timestamp: 2022-02-17 - 12:12:10 +0000
      labels:
        app.kubernetes.io/name: person-service
        app.kubernetes.io/version: "1.0"
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  key: database-user
                  name: person-db
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-password
                  name: person-db
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  key: database-name
                  name: person-db
          image: piomin/person-service:1.0
          imagePullPolicy: Always
          name: person-service
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP

Let’s take a look at a report generated by the Datree task after analyzing the manifest shown above.

(1) Ensure each container has a configured memory / CPU request – to fix that, we will add the following two properties in the application.properties file:

quarkus.kubernetes.resources.requests.memory=64Mi
quarkus.kubernetes.resources.requests.cpu=250m

(2) Ensure each container has a configured memory / CPU limit – that’s a very similar situation to the issue from point 1, but this time related to memory and CPU limits. We need to add the following properties in the application.properties file:

quarkus.kubernetes.resources.limits.memory=512Mi
quarkus.kubernetes.resources.limits.cpu=500m

(3) Ensure each container has a configured liveness and readiness probe – we just need to add a single dependency to the Maven pom.xml to enable health checks with Quarkus

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-smallrye-health</artifactId>
</dependency>

(4) Ensure Deployment has more than one replica configured – let’s set 2 replicas for our application

quarkus.kubernetes.replicas = 2

(5) Ensure workload has a configured owner label – we can also use the Quarkus Kubernetes module to add a label to the Deployment manifest

quarkus.kubernetes.labels.owner = piotr.minkowski

(6) Ensure the Prometheus labels are set – finally our custom rule for checking Prometheus annotations. Let’s add the module that automatically exposes the Prometheus endpoint for the Quarkus application and add annotations to the Deployment manifest

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-smallrye-metrics</artifactId>
</dependency>

If you run the pipeline once again, it should finish successfully.

Here’s the final version of the Kubernetes Deployment analyzed with Datree.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    app.quarkus.io/commit-id: 65b6ffded40ecac3297ae77c63c148920efedf0f
    app.quarkus.io/build-timestamp: 2022-02-18 - 07:57:34 +0000
    prometheus.io/scrape: "true"
    prometheus.io/path: /q/metrics
    prometheus.io/port: "8080"
    prometheus.io/scheme: http
  labels:
    owner: piotr.minkowski
    app.kubernetes.io/name: person-service
    app.kubernetes.io/version: "1.0"
  name: person-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app.kubernetes.io/name: person-service
      app.kubernetes.io/version: "1.0"
  template:
    metadata:
      annotations:
        app.quarkus.io/commit-id: 65b6ffded40ecac3297ae77c63c148920efedf0f
        app.quarkus.io/build-timestamp: 2022-02-18 - 07:57:34 +0000
        prometheus.io/scrape: "true"
        prometheus.io/path: /q/metrics
        prometheus.io/port: "8080"
        prometheus.io/scheme: http
      labels:
        owner: piotr.minkowski
        app.kubernetes.io/name: person-service
        app.kubernetes.io/version: "1.0"
    spec:
      containers:
        - env:
            - name: KUBERNETES_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
            - name: POSTGRES_USER
              valueFrom:
                secretKeyRef:
                  key: database-user
                  name: person-db
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: database-password
                  name: person-db
            - name: POSTGRES_DB
              valueFrom:
                secretKeyRef:
                  key: database-name
                  name: person-db
          image: pminkows/person-service:1.0
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/live
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
          name: person-service
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /q/health/ready
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 0
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 10
          resources:
            limits:
              cpu: 1000m
              memory: 512Mi
            requests:
              cpu: 250m
              memory: 64Mi

The post Validate Kubernetes Deployment in CI/CD with Tekton and Datree appeared first on Piotr's TechBlog.