• Argo CD
• Istio
• Apache Kafka
• Postgres
• HashiCorp Vault
• Prometheus
• Redis
• Cert Manager

Finally, we will migrate our sample Spring Boot app. I will also show you how to build such an app on Kubernetes and OpenShift in the same way using the Shipwright tool. However, before we start, let’s discuss some differences between “vanilla” Kubernetes and OpenShift.

Introduction

What are the key differences between Kubernetes and OpenShift? That’s probably the first question you will ask yourself when considering migration from Kubernetes. Today, I will focus only on those aspects that impact running the apps from our list. First of all, OpenShift is built on top of Kubernetes and is fully compatible with Kubernetes APIs and resources. If you can do something on Kubernetes, you can do it on OpenShift in the same way unless it doesn’t compromise security policy. OpenShift comes with additional security policies out of the box. For example, by default, it won’t allow you to run containers with the root user.

Apart from security reasons, only the fact that you can do something doesn’t mean that you should do it in that way. So, you can run images from Docker Hub, but Red Hat provides many supported container images built from Red Hat Enterprise Linux. You can find a full list of supported images here. Although you can install popular software on OpenShift using Helm charts, Red Hat provides various supported Kubernetes operators for that. With those operators, you can be sure that the installation will go without any problems and the solution might be integrated with OpenShift better. We will analyze all those things based on the examples from the tools list.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions.

Install Argo CD

Use Official Helm Chart

In the first step, we will install Argo CD on OpenShift. I’m assuming that on Kubernetes, you’re using the official Helm chart for that. In order to install that chart, we need to add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

Then, we can install the Argo CD in the argocd namespace on OpenShift with the following command. The Argo CD Helm chart provides some parameters dedicated to OpenShift. We need to enable arbitrary uid for the repo server by setting the openshift.enabled property to true. If we want to access the Argo CD dashboard outside of the cluster we should expose it as the Route. In order to do that, we need to enable the server.route.enabled property and set the hostname using the server.route.hostname parameter (piomin.eastus.aroapp.io is my OpenShift domain).

$ helm install argocd argo/argo-cd -n argocd --create-namespace \
    --set openshift.enabled=true \
    --set server.route.enabled=true \
    --set server.route.hostname=argocd.apps.piomin.eastus.aroapp.io

After that, we can access the Argo CD dashboard using the Route address as shown below. The admin user password may be taken from the argocd-initial-admin-secret Secret generated by the Helm chart.

Use the OpenShift GitOps Operator (Recommended Way)

The solution presented in the previous section works fine. However, it is not the optimal approach for OpenShift. In that case, the better idea is to use OpenShift GitOps Operator. Firstly, we should find the “Red Hat GitOps Operator” inside the “Operator Hub” section in the OpenShift Console. Then, we have to install the operator.

During the installation, the operator automatically creates the Argo CD instance in the openshift-gitops namespace.

OpenShift GitOps operator automatically exposes the Argo CD dashboard through the Route. It is also integrated with OpenShift auth, so we can use cluster credentials to sign in there.

Install Redis, Postgres and Apache Kafka

OpenShift Support in Bitnami Helm Charts

Firstly, let’s assume that we use Bitnami Helm charts to install all three tools from the chapter title (Redis, Postgres, Kafka) on Kubernetes. Fortunately, the latest versions of Bitnami Helm charts provide out-of-the-box compatibility with the OpenShift platform. Let’s analyze what it means.

Beginning from the 4.11 version OpenShift introduces new Security Context Constraints (SCC) called restricted-v2. In OpenShift, security context constraints allow us to control permissions assigned to the pods. The restricted-v2 SCC includes a minimal set of privileges usually required for a generic workload to run. It is the most restrictive policy that matches the current pod security standards. As I mentioned before, the latest version of the most popular Bitnami Helm charts supports the restricted-v2 SCC. We can check which of the charts support that feature by checking if they provide the global.compatibility.openshift.adaptSecurityContext parameter. The default value of that parameter is auto. It means that it is applied only if the detected running cluster is Openshift.

So, in short, we don’t have to change anything in the Helm chart configuration used on Kubernetes to make it work also on OpenShift. However, it doesn’t mean that we won’t change that configuration. Let’s analyze it tool after tool.

Install Redis on OpenShift with Helm Chart

In the first step, let’s add the Bitnami Helm repository with the following command:

$ helm repo add bitnami https://charts.bitnami.com/bitnami

Then, we can install and run a three-node Redis cluster with a single master node in the redis namespace using the following command:

$ helm install redis bitnami/redis -n redis --create-namespace

After installing the chart we can display a list of pods running the redis namespace:

$ oc get po
NAME               READY   STATUS    RESTARTS   AGE
redis-master-0     1/1     Running   0          5m31s
redis-replicas-0   1/1     Running   0          5m31s
redis-replicas-1   1/1     Running   0          4m44s
redis-replicas-2   1/1     Running   0          4m3s

Let’s take a look at the securityContext section inside one of the Redis cluster pods. It contains characteristic fields for the restricted-v2 SCC, which removes runAsUser, runAsGroup and fsGroup and let the platform use their allowed default IDs.

However, let’s stop for a moment to analyze the current situation. We installed Redis on OpenShift using the Bitnami Helm chart. By default, this chart is based on the Redis Debian image provided by Bitnami in the Docker Hub.

On the other hand, Red Hat provides its build of Redis image based on RHEL 9. Consequently, this image would be more suitable for running on OpenShift.

In order to use a different Redis image with the Bitnami Helm chart, we need to override the registry, repository, and tag fields in the image section. The full address of the current latest Red Hat Redis image is registry.redhat.io/rhel9/redis-7:1-16. In order to make the Bitnami chart work with that image, we need to override the default data path to /var/lib/redis/data and disable the container’s Security Context read-only root filesystem for the slave pods.

image:
  tag: 1-16
  registry: registry.redhat.io
  repository: rhel9/redis-7

master:
  persistence:
    path: /var/lib/redis/data

replica:
  persistence:
    path: /var/lib/redis/data
  containerSecurityContext:
    readOnlyRootFilesystem: false

Install Postgres on OpenShift with Helm Chart

With Postgres, we have every similar as before with Redis. The Bitnami Helm chart also supports OpenShift restricted-v2 SCC and Red Hat provide the Postgres image based on RHEL 9. Once again, we need to override some chart parameters to adapt to a different image than the default one provided by Bitnami.

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data

Of course, we can consider switching to one of the available Postgres operators. From the “Operator Hub” section we can install e.g. Postgres using Crunchy or EDB operators. However, these are not operators provided by Red Hat. Of course, you can use them on “vanilla” Kubernetes as well. In that case, the migration to OpenShift also won’t be complicated.

Install Kafka on OpenShift with the Strimzi Operator

The situation is slightly different in the case of Apache Kafka. Of course, we can use the Kafka Helm chart provided by Bitnami. However, Red Hat provides a supported version of Kafka through the Strimzi operator. This operator is a part of the Red Hat product ecosystem and is available commercially as the AMQ Streams. In order to install Kafka with AMQ Streams on OpenShift, we need to install the operator first.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-streams
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: amq-streams
  source: redhat-operators
  sourceNamespace: openshift-marketplace

Once we install the operator with the Strimzi CRDs we can provision the Kafka instance on OpenShift. In order to do that, we need to define the Kafka object. The name of the cluster is my-cluster. We should install it after a successful installation of the operator CRD, so we set the higher value of the Argo CD sync-wave parameter than for the amq-streams Subscription object. Argo CD should also ignore missing CRDs installed by the operator during sync thanks to the SkipDryRunOnMissingResource option.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.6'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    version: 3.6.0
    replicas: 3
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      deleteClaim: true
      size: 2Gi
    replicas: 3

GitOps Strategy for Kubernetes and OpenShift

In this section, we will focus on comparing differences in the GitOps manifest between Kubernetes and Openshift. We will use Kustomize to configure two overlays: openshift and kubernetes. Here’s the structure of our configuration repository:

.
├── base
│   ├── kustomization.yaml
│   └── namespaces.yaml
└── overlays
    ├── kubernetes
    │   ├── kustomization.yaml
    │   ├── namespaces.yaml
    │   ├── values-cert-manager.yaml
    │   └── values-vault.yaml
    └── openshift
        ├── cert-manager-operator.yaml
        ├── kafka-operator.yaml
        ├── kustomization.yaml
        ├── service-mesh-operator.yaml
        ├── values-postgres.yaml
        ├── values-redis.yaml
        └── values-vault.yaml

Configuration for Kubernetes

In addition to the previously discussed tools, we will also install “cert-manager”, Prometheus, and Vault using Helm charts. Kustomize allows us to define a list of managed charts using the helmCharts section. Here’s the kustomization.yaml file containing a full set of installed charts:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - namespaces.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
  - name: kafka
    repo: https://charts.bitnami.com/bitnami
    releaseName: kafka
    namespace: kafka
  - name: cert-manager
    repo: https://charts.jetstack.io
    releaseName: cert-manager
    namespace: cert-manager
    valuesFile: values-cert-manager.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml
  - name: prometheus
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: prometheus
    namespace: prometheus
  - name: istio
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: istio
    namespace: istio-system

For some of them, we need to override default Helm parameters. Here’s the values-vault.yaml file with the parameters for Vault. We enable development mode and UI dashboard:

server:
  dev:
    enabled: true
ui:
  enabled: true

Let’s also customize the default behavior of the “cert-manager” chart with the following values:

installCRDs: true
startupapicheck:
  enabled: false

Configuration for OpenShift

Then, we can switch to the configuration for Openshift. Vault has to be installed with the Helm chart, but for “cert-manager” we can use the operator provided by Red Hat. Since Openshift comes with built-in Prometheus, we don’t need to install it. We will also replace the Helm chart with Istio with the Red Hat-supported OpenShift Service Mesh operator. Here’s the kustomization.yaml for OpenShift:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - kafka-operator.yaml
  - cert-manager-operator.yaml
  - service-mesh-operator.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
    valuesFile: values-redis.yaml
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
    valuesFile: values-postgres.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml

For Vault we should enable integration with Openshift and support for the Route object. Red Hat provides a Vault image based on UBI in the registry.connect.redhat.com/hashicorp/vault registry. Here’s the values-vault.yaml file for OpenShift:

server:
  dev:
    enabled: true
  route:
    enabled: true
    host: ""
    tls: null
  image:
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.16.1-ubi"
global:
  openshift: true
injector:
  enabled: false

In order to install operators we need to define at least the Subscription object. Here’s the subscription for the OpenShift Service Mesh. After installing the operator we can create a control plane in the istio-system namespace using the ServiceMeshControlPlane CRD object. In order to apply the CRD after installing the operator, we need to use the Argo CD sync waves and define the SkipDryRunOnMissingResource parameter:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: servicemeshoperator
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: servicemeshoperator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
  name: basic
  namespace: istio-system
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  tracing:
    type: None
    sampling: 10000
  policy:
    type: Istiod
  addons:
    grafana:
      enabled: false
    jaeger:
      install:
        storage:
          type: Memory
    kiali:
      enabled: false
    prometheus:
      enabled: false
  telemetry:
    type: Istiod
  version: v2.5

Since the “cert-manager” operator is installed in a different namespace than openshift-operators, we also need to define the OperatorGroup object.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  targetNamespaces:
    - cert-manager

Finally, OpenShift comes with built-in Prometheus monitoring, so we don’t need to install it.

Apply the Configuration with Argo CD

Here’s the Argo CD Application responsible for installing our sample configuration on OpenShift. We should create it in the openshift-gitops namespace.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: install
  namespace: openshift-gitops
spec:
  destination:
    server: 'https://kubernetes.default.svc'
  project: default
  source:
    path: overlays/openshift
    repoURL: 'https://github.com/piomin/kubernetes-to-openshift-argocd.git'
    targetRevision: HEAD

Before that, we need to enable the use of the Helm chart inflator generator with Kustomize in Argo CD. In order to do that, we can add the kustomizeBuildOptions parameter in the openshift-gitops ArgoCD object as shown below.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  # ...
  kustomizeBuildOptions: '--enable-helm'

After creating the Argo CD Application and triggering the sync process, the installation starts on OpenShift.

Build App Images

We installed several software solutions including the most popular databases, message brokers, and security tools. However, now we want to build and run our own apps. How to migrate them from Kubernetes to OpenShift? Of course, we can run the app images exactly in the same way as in Kubernetes. On the other hand, we can build them on OpenShift using the Shipwright project. We can install it on OpenShift using the “Builds for Red Hat OpenShift Operator”.

After that, we need to create the ShiwrightBuild object. It needs to contain the name of the target namespace for running Shipwright in the targetNamespace field. In my case, the target namespace is builds-demo. For a detailed description of the Shipwright build, you can refer to that article on my blog.

apiVersion: operator.shipwright.io/v1alpha1
kind: ShipwrightBuild
metadata:
  name: openshift-builds
spec:
  targetNamespace: builds-demo

With Shipwright we can easily switch between multiple build strategies on Kubernetes, and on OpenShift as well. For example, on OpenShift we can use a built-in source-to-image (S2I) strategy, while on Kubernetes e.g. Kaniko or Cloud Native Buildpacks.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: sample-spring-kotlin-build
  namespace: builds-demo
spec:
  output:
    image: quay.io/pminkows/sample-kotlin-spring:1.0-shipwright
    pushSecret: pminkows-piomin-pull-secret
  source:
    git:
      url: https://github.com/piomin/sample-spring-kotlin-microservice.git
  strategy:
    name: source-to-image
    kind: ClusterBuildStrategy

Final Thoughts

Migration from Kubernetes to Openshift is not a painful process. Many popular Helm charts support OpenShift restricted-v2 SCC. Thanks to that, in some cases, you don’t need to change anything. However, sometimes it’s worth switching to the version of the particular tool supported by Red Hat.

The post Migrate from Kubernetes to OpenShift in the GitOps Way appeared first on Piotr's TechBlog.

We will do the same thing as described in that article, but fully with the GitOps approach applied by Argo CD. The main goal here is to integrate Postgres with the Vault database secrets engine to generate database credentials dynamically and initialize the DB schema for the sample Spring Boot app. In order to achieve these goals, we are going to install two Kubernetes operators: Atlas and Vault Config. Atlas is a tool for managing the database schema as code. Its Kubernetes Operator allows us to define the schema and apply it to our database using the CRD objects. The Vault Config Operator provided by the Red Hat Community of Practice does a very similar thing but for Hashicorp Vault.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions

How It Works

Before we start, let’s describe our sample scenario. Thanks to the database secrets engine Vault integrates with Postgres and generates its credentials dynamically based on configured roles. On the other hand, our sample Spring Boot app integrates with Vault and uses its database engine to authenticate against Postgres. All the aspects of that scenario are managed in the GitOps style. Argo CD installs Vault, Postgres, and additional operators on Kubernetes via their Helm charts. Then, it applies all the required CRD objects to configure both Vault and Postgres. We keep the whole configuration in a single Git repository in the form of YAML manifests.

Argo CD prepares the configuration on Vault and creates a table on Postgres for the sample Spring Boot app. Our app integrates with Vault through the Spring Cloud Vault project. It also uses Spring Data JPA to interact with the database. Here’s the illustration of our scenario.

Install Argo CD on Kubernetes

Traditionally, we need to start our GitOps exercise by installing Argo CD on the Kubernetes cluster. Of course, we can do it using the Helm chart. In the first step, we need to add the following repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

We will add one parameter to the argo-cm ConfigMap to ignore the MutatingWebhookConfiguration kind. This step is not necessary. It allows us to ignore the specific resource generated by one of the Helm charts used in the further steps. Thanks to that we will have everything in Argo CD in the “green” color Here’s the Helm values.yaml file with the required configuration:

configs:
  cm:
    resource.exclusions: |
      - apiGroups:
        - admissionregistration.k8s.io
        kinds:
        - MutatingWebhookConfiguration
        clusters:
        - "*"

Now, we can install the Argo CD in the argocd namespace using the configuration previously defined in the values.yml file:

$ helm install argo-cd argo/argo-cd \
    --version 6.7.8 \
    -n argo \
    --create-namespace

That’s not all. Since the Atlas operator is available in the OCI-type Helm repository, we need to apply the following Secret in the argocd namespace. By default, Argo CD doesn’t allow the OCI-type repo, so we need to include the enableOCI parameter in the definition.

apiVersion: v1
kind: Secret
metadata:
  name: ghcr-io-helm-oci
  namespace: argocd
  labels:
    argocd.argoproj.io/secret-type: repository
stringData:
  name: ariga
  url: ghcr.io/ariga
  enableOCI: "true"
  type: helm

Let’s take a look at the list of repositories in the Argo CD UI dashboard. You should see the “Successful” connection status.

Prepare Configuration Manifests for Argo CD

Config Repository Structure

Let me first explain the structure of our Git config repository. The additional configuration is stored in the apps directory. It includes the CRD objects required to initialize the database schema or Vault engines. In the bootstrap directory, we keep the values.yaml file for each Helm chart managed by Argo CD. It’s all that we need. The bootstrap-via-appset/bootstrap.yaml contains the definition of Argo CD ApplicationSet we need to apply to the Kubernetes cluster. This ApplicationSet will generate all required Argo CD applications responsible for installing the charts and creating CRD objects.

.
├── apps
│   ├── postgresql
│   │   ├── database.yaml
│   │   ├── policies.yaml
│   │   ├── roles.yaml
│   │   └── schema.yaml
│   └── vault
│       └── job.yaml
├── bootstrap
│   ├── values
│   │   ├── atlas
│   │   │   └── values.yaml
│   │   ├── cert-manager
│   │   │   └── values.yaml
│   │   ├── postgresql
│   │   │   └── values.yaml
│   │   ├── vault
│   │   │   └── values.yaml
│   │   └── vault-config-operator
│   │       └── values.yaml
└── bootstrap-via-appset
    └── bootstrap.yaml

Bootstrap with the Argo CD ApplicationSet

Let’s take a look at the ApplicationSet. It’s pretty interesting (I hope :)). I’m using here some relatively new Argo CD features like multiple sources (Argo CD 2.6) or application sets template patch (Argo CD 2.10). We need to generate an Argo CD Application per each tool we want to install on Kubernetes (1). In the generators section, we define parameters for Vault, PostgreSQL, Atlas Operator, Vault Config Operator, and Cert Manager (which is required by the Vault Config Operator). In the templatePatch section, we prepare a list of source repositories used by each Argo CD Application (2). There is always a Helm chart repo, which refers to our Git repository containing dedicated values.yaml files. For the Vault and PostgreSQL charts, we include another source containing CRDs or additional Kubernetes objects. We will discuss it later.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: bootstrap-config
  namespace: argocd
spec:
  goTemplate: true
  generators:
  - list:
      elements:
        - chart: vault
          name: vault
          repo: https://helm.releases.hashicorp.com
          revision: 0.27.0
          namespace: vault
          postInstall: true
        - chart: postgresql
          name: postgresql
          repo: https://charts.bitnami.com/bitnami
          revision: 12.12.10
          namespace: default
          postInstall: true
        - chart: cert-manager
          name: cert-manager
          repo: https://charts.jetstack.io
          revision: v1.14.4
          namespace: cert-manager
          postInstall: false
        - chart: vault-config-operator
          name: vault-config-operator
          repo: https://redhat-cop.github.io/vault-config-operator
          revision: v0.8.25
          namespace: vault-config-operator
          postInstall: false
        - chart: charts/atlas-operator
          name: atlas
          repo: ghcr.io/ariga
          revision: 0.4.2
          namespace: atlas
          postInstall: false
  template:
    metadata:
      name: '{{.name}}'
      annotations:
        argocd.argoproj.io/sync-wave: "1"
    spec:
      syncPolicy:
        automated: {}
        syncOptions:
          - CreateNamespace=true
      destination:
        namespace: '{{.namespace}}'
        server: https://kubernetes.default.svc
      project: default
  templatePatch: |
    spec:
      sources:
        - repoURL: '{{ .repo }}'
          chart: '{{ .chart }}'
          targetRevision: '{{ .revision }}'
          helm:
            valueFiles:
              - $values/bootstrap/values/{{ .name }}/values.yaml
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          ref: values
        {{- if .postInstall }}
        - repoURL: https://github.com/piomin/kubernetes-config-argocd.git
          targetRevision: HEAD
          path: apps/{{ .name }}
        {{- end }}

Once we apply the bootstrap-config ApplicationSet to the argocd namespace, all the magic just happens. You should see five applications in the Argo CD UI dashboard. All of them are automatically synchronized (Argo CD autoSync enabled) to the cluster. It does the whole job. Now, let’s analyze step-by-step what we have to put in that configuration.

The Argo CD ApplicationSet generates five applications for installing all required tools. Here’s the Application generated for installing Vault with Helm charts and applying an additional configuration stored in the apps/vault directory.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: vault
  namespace: argocd
spec:
  destination:
    namespace: vault
    server: https://kubernetes.default.svc
  project: default
  sources:
    - chart: vault
      helm:
        valueFiles:
          - $values/bootstrap/values/vault/values.yaml
      repoURL: https://helm.releases.hashicorp.com
      targetRevision: 0.27.0
    - ref: values
      repoURL: https://github.com/piomin/kubernetes-config-argocd.git
      targetRevision: HEAD
    - path: apps/vault
      repoURL: https://github.com/piomin/kubernetes-config-argocd.git
      targetRevision: HEAD
  syncPolicy:
    automated: {}
    syncOptions:
      - CreateNamespace=true

Configure Vault on Kubernetes

Customize Helm Charts

Let’s take a look at the Vault values.yaml file. We run it in the development mode (single, in-memory node, no unseal needed). We will also enable the UI dashboard.

server:
  dev:
    enabled: true
ui:
  enabled: true

With the parameters visible above Argo CD installs Vault in the vault namespace. Here’s a list of running pods:

$ kubectl get po -n vault
NAME                                    READY   STATUS      RESTARTS      AGE
vault-0                                 1/1     Running     0            1h
vault-agent-injector-7f7f68d457-fvsd2   1/1     Running     0            1h

It also exposes Vault API under the 8200 port in the vault Kubernetes Service.

$ kubectl get svc -n vault
NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE
vault                      ClusterIP   10.110.69.159    <none>        8200/TCP,8201/TCP   21h
vault-agent-injector-svc   ClusterIP   10.111.24.183    <none>        443/TCP             21h
vault-internal             ClusterIP   None             <none>        8200/TCP,8201/TCP   21h
vault-ui                   ClusterIP   10.110.160.239   <none>        8200/TCP            21h

For the Vault Config Operator, we need to override the default address of Vault API to vault.vault.svc:8200 (an a). In order to do that, we need to set the VAULT_ADDR env variable in the values.yaml file. We also disable Prometheus monitoring and enable integration with Cert Manager. Thanks to “cert-manager” we don’t need to generate any certificates or keys manually.

enableMonitoring: false
enableCertManager: true
env:
  - name: VAULT_ADDR
    value: http://vault.vault:8200

Enable Vault Config Operator

The Vault Config Operator needs to authenticate against Vault API using Kubernetes Authentication. So we need to configure a root Kubernetes Authentication mount point and role. Then we can create more roles or other Vault objects via the operator. Here’s the Kubernetes Job responsible for configuring Kubernetes mount point and role. It uses the Vault image and the vault CLI available inside that image. As you see, it creates the vault-admin role allowed in the default namespace.

apiVersion: batch/v1
kind: Job
metadata:
  name: vault-admin-initializer
  annotations:
    argocd.argoproj.io/sync-wave: "3"
spec:
  template:
    spec:
      containers:
        - name: vault-admin-initializer
          image: hashicorp/vault:1.15.2
          env:
            - name: VAULT_ADDR
              value: http://vault.vault.svc:8200
          command:
            - /bin/sh
            - -c
            - |
              export VAULT_TOKEN=root
              sleep 10
              vault auth enable kubernetes
              vault secrets enable database
              vault write auth/kubernetes/config kubernetes_host=https://kubernetes.default.svc:443 kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt
              vault write auth/kubernetes/role/vault-admin bound_service_account_names=default bound_service_account_namespaces=default policies=vault-admin ttl=1h
              vault policy write vault-admin - <<EOF
                path "/*" {
                  capabilities = ["create", "read", "update", "delete", "list","sudo"]
                }          
              EOF
      restartPolicy: Never

Argo CD applies such a Job after installing the Vault chart.

$ kubectl get job -n vault
NAME                      COMPLETIONS   DURATION   AGE
vault-admin-initializer   1/1           15s        1h

Configure Vault via CRDs

Once a root Kubernetes authentication is ready, we can proceed to the CRD object creation. In the first step, we create objects responsible for configuring a connection to the Postgres database. In the DatabaseSecretEngineConfig we set the connection URL, credentials, and the name of a Vault plugin used to interact with the database (postgresql-database-plugin). We also define a list of allowed roles (postgresql-default-role). In the next step, we define the postgresql-default-role DatabaseSecretEngineRole object. Of course, the name of the role should be the same as the name passed in the allowedRoles list in the previous step. The role defines a target database connection name in Vault and the SQL statement for creating new users with privileges.

kind: DatabaseSecretEngineConfig
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: postgresql-database-config
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  allowedRoles:
    - postgresql-default-role
  authentication:
    path: kubernetes
    role: vault-admin
  connectionURL: 'postgresql://{{username}}:{{password}}@postgresql.default:5432?sslmode=disable'
  path: database
  pluginName: postgresql-database-plugin
  rootCredentials:
    passwordKey: postgres-password
    secret:
      name: postgresql
  username: postgres
---
apiVersion: redhatcop.redhat.io/v1alpha1
kind: DatabaseSecretEngineRole
metadata:
  name: postgresql-default-role
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  creationStatements:
    - CREATE ROLE "{{name}}" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO "{{name}}"; GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO "{{name}}";
  maxTTL: 10m0s
  defaultTTL: 1m0s
  authentication:
    path: kubernetes
    role: vault-admin
  dBName: postgresql-database-config
  path: database

Once Argo CD applies both DatabaseSecretEngineConfig and DatabaseSecretEngineRole objects, we can verify it works fine by generating database credentials using the vault read command. We need to pass the name of the previously created role (postgresql-default-role). Our sample app will do the same thing but through the Spring Cloud Vault module.

Finally, we can create a policy and role for our sample Spring Boot. The policy requires only the privilege to generate new credentials:

kind: Policy
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: database-creds-view
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  authentication:
    path: kubernetes
    role: vault-admin
  policy: |
    path "database/creds/default" {
      capabilities = ["read"]
    }

Now, we have everything to proceed to the last step in this section. We need to create a Vault role with the Kubernetes authentication method dedicated to our sample app. In this role, we set the name and location of the Kubernetes ServiceAccount and the name of the Vault policy created in the previous step.

kind: KubernetesAuthEngineRole
apiVersion: redhatcop.redhat.io/v1alpha1
metadata:
  name: database-engine-creds-role
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  authentication:
    path: kubernetes
    role: vault-admin
  path: kubernetes
  policies:
    - database-creds-view
  targetServiceAccounts:
    - default
  targetNamespaces:
    targetNamespaces:
      - default

Managing Postgres Schema with Atlas Operator

Finally, we can proceed to the last step in the configuration part. We will use the AtlasSchema CRD object to configure the database schema for our sample app. The object contains two sections: credentials and schema. In the credentials section, we refer to the PostgreSQL Secret to obtain a password. In the schema section, we create the person table with the id primary key.

apiVersion: db.atlasgo.io/v1alpha1
kind: AtlasSchema
metadata:
  name: sample-spring-cloud-vault
  annotations:
    argocd.argoproj.io/sync-wave: "4"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  credentials:
    scheme: postgres
    host: postgresql.default
    user: postgres
    passwordFrom:
      secretKeyRef:
        key: postgres-password
        name: postgresql
    database: postgres
    port: 5432
    parameters:
      sslmode: disable
  schema:
    sql: |
      create table person (
        id serial primary key,
        name varchar(255),
        gender varchar(255),
        age int,
        external_id int
      );

Here’s the corresponding app @Entity model class in the sample Spring Boot app.

@Entity
public class Person {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String name;
    private int age;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    private Integer externalId;   
    
   // GETTERS AND SETTERS ...
   
}

Once Argo CD applies the AtlasSchema object, we can verify its status. As you see, it has been successfully executed on the target database.

We can log in to the database using psql CLI and verify that the person table exists in the postgres database:

Run Sample Spring Boot App

Dependencies

For this demo, I created a simple Spring Boot application. It exposes REST API and connects to the PostgreSQL database. It uses Spring Data JPA to interact with the database. Here are the most important dependencies of our app in the Maven pom.xml:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>
<dependency>
  <groupId>org.postgresql</groupId>
  <artifactId>postgresql</artifactId>
  <scope>runtime</scope>
</dependency>

The first of them enables bootstrap.yml processing on the application startup. The third one includes Spring Cloud Vault Database engine support.

Integrate with Vault using Spring Cloud Vault

The only thing we need to do is to provide the right configuration settings. Here’s the minimal set of the required dependencies to make it work without any errors. The following configuration is provided in the bootstrap.yml file:

spring:
  application:
    name: sample-db-vault
  datasource:
    url: jdbc:postgresql://postgresql:5432/postgres #(1)
  jpa:
    hibernate:
      ddl-auto: update
  cloud:
    vault:
      config.lifecycle: #(2)
        enabled: true
        min-renewal: 10s
        expiry-threshold: 30s
      kv.enabled: false #(3)
      uri: http://vault.vault:8200 #(4)
      authentication: KUBERNETES #(5)
      postgresql: #(6)
        enabled: true
        role: postgresql-default-role
        backend: database
      kubernetes: #(7)
        role: database-engine-creds-role

Let’s analyze the configuration visible above in the details:

(1) Firstly, we need to set the database connection URL without any credentials. Our application uses standard properties for authentication against the database (spring.datasource.username and spring.datasource.password). Thanks to that, we don’t need to do anything else

(2) As you probably remember, the maximum TTL for the database lease is 10 minutes. We enable lease renewal every 30 seconds. Just for the demo purpose. You will see that Spring Cloud Vault will create new credentials in PostgreSQL every 30 seconds, and the application still works without any errors

(3) Vault KV is not needed here, since I’m using only the database engine

(4) The application is going to be deployed in the default namespace, while Vault is running in the vault namespace. So, the address of Vault should include the namespace name

(5) (7) Our application uses the Kubernetes authentication method to access Vault. We just need to set the role name, which is database-engine-creds-role. All other settings should be left with the default values

(6) We also need to enable postgres database backend support. The name of the backend in Vault is database and the name of the Vault role used for that engine is postgresql-default-role.

Run the App on Kubernetes

Finally, we can run our sample app on Kubernetes by applying the following YAML manifest:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-deployment
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
        - name: sample-app
          image: piomin/sample-app:1.0-gitops
          ports:
            - containerPort: 8080
      serviceAccountName: default
---
apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  type: ClusterIP
  selector:
    app: sample-app
  ports:
  - port: 8080

Our app exposes REST API under the /persons path. We can easily test it with curl after enabling port forwarding as shown below:

$ kubectl port-forward svc/sample-app 8080:8080
$ curl http://localhost:8080/persons

Final Thoughts

This article proves that we can effectively configure and manage tools like Postgres database or Hashicorp Vault on Kubernetes with Argo CD. The database schema or Vault configuration can be stored in the Git repository in the form of YAML manifests thanks to Atlas and Vault Config Kubernetes operators. Argo CD applies all required CRDs automatically, which results in the integration between Vault, Postgres, and our sample Spring Boot app.

The post GitOps on Kubernetes for Postgres and Vault with Argo CD appeared first on Piotr's TechBlog.

The solution presented in this article is not the only way how we can deal with HashiCorp Vault on Kubernetes. If you are interested in other approaches you may refer to some of my previous articles. In that article, you can find a guide on how to integrate Vault secrets with Argo CD through the plugin. If you are running Spring Boot apps on Kubernetes you can also be interested in Spring Cloud Vault support. In that case please refer to the following article.

How it works

I guess you may not be very familiar with the Container Storage Interface (CSI) pattern. At the high-level CSI is a standard for exposing block or file storage to the containers. It is implemented by different storage providers.

The Secrets Store CSI Driver is running on Kubernetes as a DeamonSet. It is interacting with every instance of Kubelet on the Kubernetes nodes. Once the pod is starting, the Secrets Store CSI Driver communicates with the external secrets provider to retrieve the secret content. The following diagram illustrates how Secrets Store CSI Driver works on Kubernetes.

It provides the SecretProviderClass CRD to manage that process. In this provider class, we need to set the secure vault address and the location of the secret keys. Here’s the SecretProviderClass for our scenario. We will use HashiCorp Vault running on Kubernetes.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: default
spec:
  parameters:
    objects: |-
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"
    roleName: webapp
    vaultAddress: 'http://vault.vault.svc:8200'
  provider: vault

Here’s the location of our secret in the HashiCorp Vault. As you see the current value of the password entry is test1.

Source Code

As usual – if you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my two GitHub repositories. The first of them contains Terraform scripts for installing Vault and Secrets Store CSI. After cloning it you go to the vault-ocp directory. The second repository contains a simple Spring Boot app for a test scenario. Once you clone it, go to the spring-util-app directory. Then you should just follow my instructions

Install Vault and Secrets Store CSI Driver with Terraform

As I mentioned before, we will use Terraform to set up almost the whole test scenario today. We will just leverage Skaffold, in the last step, to deploy the Spring Boot app on the Kubernetes cluster.

In order to install both Vault and Secrets Store CSI Driver we will use Helm charts. To do that part of the exercise, we need kubernetes (1) and helm (2) as the Terraform providers. The third step (3) is required only if you run the scenario on OpenShift. It changes the default service account access restrictions and security context constraints (SCCs) to ensure that a pod has sufficient permissions to start on OpenShift. Then we may proceed to the Helm charts installation. For the Secrets Store CSI Driver chart (4), it is important to enable secrets rotation since we will test this feature at the end of the article. Finally, we can install the HashiCorp Vault chart (5).

# (1)
provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

# (2)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

# (3)
resource "kubernetes_cluster_role_binding" "privileged" {
  metadata {
    name = "system:openshift:scc:privileged"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "system:openshift:scc:privileged"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "secrets-store-csi-driver"
    namespace = "k8s-secrets-store-csi"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "vault-csi-provider"
    namespace = "vault"
  }
}

resource "kubernetes_namespace" "vault" {
  metadata {
    name = "vault"
  }
}

resource "kubernetes_service_account" "vault-sa" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name      = "vault"
    namespace = "vault"
  }
}

resource "kubernetes_secret_v1" "vault-secret" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name = "vault-token"
    namespace = "vault"
    annotations = {
      "kubernetes.io/service-account.name" = "vault"
    }
  }

  type = "kubernetes.io/service-account-token"
}

# (4)
resource "helm_release" "secrets-store-csi-driver" {
  chart            = "secrets-store-csi-driver"
  name             = "csi-secrets-store"
  namespace        = "k8s-secrets-store-csi"
  create_namespace = true
  repository       = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"

  set {
    name  = "linux.providersDir"
    value = "/var/run/secrets-store-csi-providers"
  }

  set {
    name  = "syncSecret.enabled"
    value = "true"
  }

  set {
    name  = "enableSecretRotation"
    value = "true"
  }
}

# (5)
resource "helm_release" "vault" {
  chart            = "vault"
  name             = "vault"
  namespace        = "vault"
  create_namespace = true
  repository       = "https://helm.releases.hashicorp.com"

  values = [
    file("values.yaml")
  ]
}

The Helm values.yaml file used by the Vault chart is visible below. In order to simplify deployment, we will enable the development mode (1). It generates a root token automatically and runs a single instance of Vault. We can enable Route for OpenShift (2) and use the image supported by Red Hat (3). Of course, we also need to enable global OpenShift configuration (4). You can omit all three steps (2) (3) (4) when running the scenario on vanilla Kubernetes. Finally, we need to enable CSI support (5) and disable the Vault sidecar injector which is not needed in that exercise (7). The path in the csi.deamonSet.providersDir property should be the same as the linux.providersDir in the Halm chart params (6).

server:
  dev:
    enabled: true # (1)
  route: # (2)
    enabled: true
    host: ""
    tls: null
  image: # (3)
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.12.4-ubi"
  serviceAccount:
    name: vault
    create: false
global: # (4)
  openshift: true
csi: 
  debug: true
  enabled: true # (5)
  daemonSet:
    providersDir: /var/run/secrets-store-csi-providers # (6)
    securityContext:
      container:
        privileged: true
injector: # (7)
  enabled: false

Finally, let’s apply the configuration to the target cluster.

$ terraform apply -auto-approve -compact-warnings

Here’s the result of the terraform apply command for my cluster:

In order to verify if everything has been installed successfully we can display the details of the vault-csi-provider DaemonSet in the vault namespace.

$ kubectl describe ds vault-csi-provider -n vault

Then, we can do a very similar thing for the Secrets Store CSI driver. We need to display the details of the csi-secrets-store-secrets-store-csi-driver DeamonSet.

$ kubectl describe ds csi-secrets-store-secrets-store-csi-driver \
  -n k8s-secrets-store-csi

Configure Vault with Terraform

The big advantage of using Terraform in our scenario is its integration with Vault. There is a dedicated Terraform provider for interacting with HashiCorp Vault. In order to set up the provider, we need to pass the Vault token (root in dev mode) and address. We will still need the kubernetes provider in that part of the exercise.

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "vault" {
  token = "root"
  address = var.vault-addr
}

We need to set the Kubernetes context path name and Vault API address in the variables.tf file. Here’s my Terraform variables.tf file:

variable "cluster-context" {
  type    = string
  default = "default/api-cluster-6sccr-6sccr-sandbox1544-opentlc-com:6443/opentlc-mgr"
}

variable "vault-addr" {
  type = string
  default = "http://vault-vault.apps.cluster-6sccr.6sccr.sandbox1544.opentlc.com"
}

The Terraform script responsible for configuring Vault is visible below. There are several things we need to do before deploying a sample Spring Boot app. Here’s a list of the required steps:

1. We need to enable Kubernetes authentication in Vault. The Secrets Store CSI Driver will use it to authenticate against the instance of Vault
2. In the second step, we will create a test secret. Its name is password and the value is test1. It is stored in Vault under the /secret/data/db-pass path.
3. Then, we have to Configure Kubernetes authenticate method.
4. In the fourth step, we are creating the policy for our app. It has read access to the secret created in step 2
5. We are creating the ServicerAccount for our sample Spring Boot app in the default namespace. The name of the ServiceAccount object is webapp-sa.
6. Finally, we can proceed to the last step in the Vault configuration – the authentication role required to access the secret. The name of the role is webapp and is then used by the Secrets Store CSI SecretProviderClass CR. The authentication role refers to the already created policy and ServiceAccount webapp-sa in the default namespace.
7. Once the Vault backend is configured properly we create the Secrets Store CSI SecretProviderClass CR.

# (1)
resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

# (2)
resource "vault_kv_secret_v2" "secret" {
  mount = "secret"
  name = "db-pass"
  data_json = jsonencode(
    {
      password = "test1"
    }
  )
}

data "kubernetes_secret" "vault-token" {
  metadata {
    name      = "vault-token"
    namespace = "vault"
  }
}

# (3)
resource "vault_kubernetes_auth_backend_config" "example" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://172.30.0.1:443"
  kubernetes_ca_cert     = data.kubernetes_secret.vault-token.data["ca.crt"]
  token_reviewer_jwt     = data.kubernetes_secret.vault-token.data.token
}

# (4)
resource "vault_policy" "internal-app" {
  name = "internal-app"

  policy = <<EOT
path "secret/data/db-pass" {
  capabilities = ["read"]
}
EOT
}

# (5)
resource "kubernetes_service_account" "webapp-sa" {
  metadata {
    name      = "webapp-sa"
    namespace = "default"
  }
}

# (6)
resource "vault_kubernetes_auth_backend_role" "internal-role" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "webapp"
  bound_service_account_names      = ["webapp-sa"]
  bound_service_account_namespaces = ["default"]
  token_ttl                        = 3600
  token_policies                   = ["internal-app"]
}

# (7)
resource "kubernetes_manifest" "vault-database" {
  manifest = {
    "apiVersion" = "secrets-store.csi.x-k8s.io/v1alpha1"
    "kind"       = "SecretProviderClass"
    "metadata" = {
      "name"      = "vault-database"
      "namespace" = "default"
    }
    "spec" = {
      "provider"   = "vault"
      "parameters" = {
        "vaultAddress" = "http://vault.vault.svc:8200"
        "roleName"     = "webapp"
        "objects"      = "- objectName: \"db-password\"\n  secretPath: \"secret/data/db-pass\"\n  secretKey: \"password\""
      }
    }
  }
}

Once again, to apply the configuration we need to execute the terraform apply command.

Of course, we could apply the whole configuration visible above using Vault CLI or UI. However, we can verify it with Vault UI. In order to log in there we must use the root token. After login, we need to go to the Access tab and then to the Auth Methods menu. As you see, there is a webapp method defined in Terraform scripts.

Let’s switch to the Policies tab. Then we can check out if the internal-app policy exists.

Run the App with Mounted Secrets

Once we applied the whole configuration with Terraform we may proceed to the sample Spring Boot app. The idea is pretty simple. Our app just reads the data from the file and exposes it as the REST endpoint. Here’s the REST @Controller implementation:

@RestController
@RequestMapping("/api")
class SampleUtilController {

    @GetMapping("/db-password")
    fun resourceString(): String {
        val file = File("/mnt/secrets-store/db-password")
        return if(file.exists()) file.readText()
        else "none"
    }
}

Here’s the app Deployment manifest. As you see we are using the secrets-store.csi.k8s.io implementation of the CSI driver for mounted volume. It refers the vault-database SecretProviderClass object created with the Terraform script. The volume is containing the file with the value of our secret. We are mounting it under the path /mnt/secrets-store, which is accessed by the Spring Boot application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-util-app
spec:
  selector:
    matchLabels:
      app: sample-util-app
  template:
    metadata:
      labels:
        app: sample-util-app
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: sample-util-app
          image: piomin/sample-util-app
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /mnt/secrets-store
              name: secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-database"

Here’s the Kubernetes Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: sample-util-app
spec:
  type: ClusterIP
  selector:
    app: sample-util-app
  ports:
    - port: 8080
      targetPort: 8080

We can easily build and deploy the app with Skaffold. It also allows exposing a port outside the cluster as a local port with the port-forward option.

$ skaffold dev --port-forward

Finally, we can our test endpoint GET /api/db-password. It returns the value we have already set in Vault for the db-pass/password secret.

$ curl http://localhost:8080/api/db-password
test2

Now, let’s test the secret rotation feature. In order to do that we need to change the value of the db-pass/password secret. We can do it using Vault UI. We can set the test2 value:

Secrets Store CSI Driver periodically queries managed secrets to detect changes. So, after the change, we would probably wait a moment until our app refreshes that value. The default poll interval is 2 minutes. We can override it using the rotation-poll-interval parameter (e.g. on the Helm chart). However, the most important thing is, that everything happens without restarting the pod. The only trace of change is in the events:

Now, let’s query for the latest value of the key used by the app. As you see the value has been refreshed.

Final Thoughts

If you are looking for a solution that injects Vault secrets into your app without creating Kubernetes Secret, Secrets Store CSI Driver is the solution for you. It is able to refresh the value of the secret in your app without restarting the container. In this article, I’m presenting how to install and configure it with Terraform to simplify the installation and configuration process.

The post Vault with Secrets Store CSI Driver on Kubernetes appeared first on Piotr's TechBlog.

You can use Vault in several different ways on Kubernetes. For example, you may integrate it directly with your Spring Boot app using the Spring Cloud Vault project. To read more about it please refer to that post on my blog.

Prerequisites

In this exercise, we are going to use Helm a lot. Our sample app Deployment is based on the Helm template. We also use Helm to install Vault and ArgoCD on Kubernetes. Finally, we need to customize the ArgoCD Helm chart parameters to enable and configure ArgoCD Vault Plugin. So, before proceeding please ensure you have basic knowledge about Helm. Of course, you should also install it on your laptop. For me, it is possible with homebrew:

$ brew install helm

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

Run and Configure Vault on Kubernetes

In the first step, we are going to install Vault on Kubernetes. We can easily do it using its official Helm chart. In order to simplify our exercise, we run it in development mode with a single server instance. Normally, you would configure it in HA mode. Let’s add the Hashicorp helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

In order to enable development mode the parameter server.dev.enabled should have the value true. We don’t need to override any other default values:

$ helm install vault hashicorp/vault \
    --set "server.dev.enabled=true"

To check if Vault is successfully installed on the Kubernetes cluster we can display a list of running pods:

$ kubectl get pod 
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          25s
vault-agent-injector-9456c6d55-hx2fd   1/1     Running   0          21s

We can configure Vault in several different ways. One of the options is through the UI. To login there we may use the root token generated only in development mode. Vault also exposes the HTTP API. The last available option is with the CLI. CLI is available inside the Vault pod, but as well we can install it locally. For me, it is possible using the brew install vault command. Then we need to enable port-forwarding and export Vault local address as the VAULT_ADDR environment variable:

$ kubectl port-forward vault-0 8200
$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault status

Then just login to the Vault server using the root token:

Enable Kubernetes Authentication on Vault

There are several authentication methods on Vault. However, since we run it on Kubernetes we should the method dedicated to that platform. What is important, this method is also supported by the ArgoCD Vault Plugin. Firstly, let’s enable the Kubernetes Auth method:

$ vault auth enable kubernetes

Then, we need to configure our authentication method. There are three required parameters: the URL of the Kubernetes API server, Kubernetes CA certificate, and a token reviewer JWT.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=<your Kubernetes API address> \
    kubernetes_ca_cert=@ca.crt

In order to easily obtain all those parameters, you can the following three commands. Then you can set them also e.g. using the Vault UI.

$ echo "https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"
$ kubectl exec vault-0 \
  -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
$ echo $(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Once we pass all the required parameters, we may proceed to the named role creation. ArgoCD Vault Plugin will use that role to authenticate against the Vault server. We need to provide the namespace with ArgoCD and the name of the Kubernetes service account used by the ArgoCD Repo Server. Our token expires after 24 hours.

$ vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-repo-server \
  bound_service_account_namespaces=argocd \
  policies=argocd \
  ttl=24h

That’s all for now. We will also need to create a test secret on Vault and configure a policy for the argocd role. Before that, let’s take a look at our sample Spring Boot app and its Helm template.

Helm Template for Spring Boot App

Our app is very simple. It just exposes a single HTTP endpoint that returns the value of the environment variable inside a container. Here’s the REST controller class written in Kotlin.

@RestController
@RequestMapping("/persons")
class PersonController {

    @Value("\${PASS:none}")
    lateinit var pass: String

    @GetMapping("/pass")
    fun printPass() = pass

}

We will use a generic Helm chart to deploy our app on Kubernetes. Our Deployment template contains a list of environment variables defined for the container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  replicas: {{ .Values.app.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
          ports:
          {{- range .Values.app.ports }}
            - containerPort: {{ .value }}
              name: {{ .name }}
          {{- end }}
          {{- if .Values.app.envs }}
          env:
          {{- range .Values.app.envs }}
            - name: {{ .name }}
              value: {{ .value }}
          {{- end }}
          {{- end }}

There is also another file in the templates directory. It contains a definition of the Kubernetes Service.

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.app.name }}
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.app.name }}
  ports:
  {{- range .Values.app.ports }}
  - port: {{ .value }}
    name: {{ .name }}
  {{- end }}

Finally, let’s take a look at the Chart.yaml file.

apiVersion: v2
name: sample-with-envs
description: A Helm chart for Kubernetes
type: application
version: 1.0
appVersion: "1.0"

Our goal is to use this Helm chart to deploy the sample Spring Boot app on Kubernetes with ArgoCD and Vault. Of course, before we do it we need to install ArgoCD.

Install ArgoCD with Vault Plugin on Kubernetes

Normally, it would be a very simple installation. But this time we need to customize the ArgoCD template to install it with the Vault plugin. Or more precisely, we have to customize the configuration of the ArgoCD Repository Server. It is one of the ArgoCD internal services. It maintains the local cache of the Git repository and generates Kubernetes manifests.

There are some different options for installing Vault plugin on ArgoCD. The full list of options is available here. Starting with the version 2.4.0 of ArgoCD it is possible to install it via a sidecar container. We will choose the option based on sidecar and initContainer. You may read more about it here. However, our case would be different a little since we Helm instead of Kustomize for installing ArgoCD. To clarify, we need to do three things to install the Vault plugin on ArgoCD. Let’s analyze those steps:

• define initContainer on the ArgoCD Repository Server Deployment to download argocd-vault-plugin binaries and mount them on the volume
• define the ConfigMap containing the ConfigManagementPlugin CRD overriding a default behavior of Helm on ArgoCD
• customize argocd-repo-server Deployment to mount the volume with argocd-vault-plugin and the ConfigMap created in the previous step

After those steps, we would have to integrate the plugin with the running instance of the Vault server. We will use a previously create Vault argocd role.

Firstly, let’s create the ConfigMap to customize the default behavior of Helm on ArgoCD. After running the helm template command we will also run the argocd-vault-plugin generate command to replace all inline placeholders with the secrets defined in Vault. The address and auth configuration of Vault are defined in the vault-configuration secret.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s vault-configuration -
      lockRepo: false

Here’s the vault-configuration Secret:

apiVersion: v1
kind: Secret
metadata:
  name: vault-configuration
  namespace: argocd 
data:
  AVP_AUTH_TYPE: azhz
  AVP_K8S_ROLE: YXJnb2Nk
  AVP_TYPE: dmF1bHQ=
  VAULT_ADDR: aHR0cDovL3ZhdWx0LmRlZmF1bHQ6ODIwMA==
type: Opaque

To see the values let’s display the Secret in Lens. Vault is running in the default namespace, so its address is http://vault.default:8200. The name of our role in Vault is argocd. We also need to set the auth type as k8s.

Finally, we need to customize the ArgoCD Helm installation. To achieve that let’s define the Helm values.yaml file. It contains the definition of initContainer and sidecar for argocd-repo-server. We also mount the cmp-plugin ConfigMap to the Deployment, and add additional privileges to the argocd-repo-service ServiceAccount to allow reading Secrets.

repoServer:
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.11.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}

In order to install ArgoCD on Kubernetes add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

Let’s install it in the argocd namespace using customized parameters in the values.yaml file:

$ kubectl create ns argocd
$ helm install argocd argo/argo-cd -n argocd -f values.yaml

Sync Vault Secrets with ArgoCD

Once we deployed Vault and ArgoCD on Kubernetes we may proceed to the next step. Now, we are going to create a secret on Vault. Firstly, let’s enable the KV engine:

$ vault secrets enable kv-v2

Then, we can create a sample secret with the argocd name and a single password key:

$ vault kv put kv-v2/argocd password="123456"

ArgoCD Vault Plugin uses the argocd policy to read secrets. So, in the next step, we need to create the following policy to enable reading the previously created secret:

$ vault policy write argocd - <<EOF
path "kv-v2/data/argocd" {
  capabilities = ["read"]
}
EOF

Then, we may define the ArgoCD Application for deploying our Spring Boot app on Kubernetes. The Helm template for Kubernetes manifests is available on the GitHub repository under the simple-with-envs directory (1). As a tool for creating manifests we choose plugin (2). However, we won’t set its name since we use a sidecar container with argocd-vault-plugin. ArgoCD Vault plugin allows passing inline values in the application manifest. It reads the content defined inside the HELM_VALUES environment variable (3) (depending on the environment variable name set inside cmp-plugin ConfigMap). And finally, the most important thing. ArgoCD Vault Plugin is looking for placeholders inside the <> brackets. For inline values, it should have the following structure: <path:path_to_the_vault_secret#name_of_the_key> (4). In our case, we define the environment variable PASS that uses the argocd secret and the password key stored inside the KV engine.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-helm
  namespace: argocd
spec:
  destination:
    name: ''
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: simple-with-envs # (1)
    repoURL: https://github.com/piomin/sample-generic-helm-charts.git 
    targetRevision: HEAD
    plugin: # (2)
      env:
        - name: HELM_VALUES # (3)
          value: |
            image:
              registry: quay.io
              repository: pminkows/sample-kotlin-spring
              tag: "1.4.30"

            app:
              name: sample-spring-boot-kotlin
              replicas: 1
              ports:
                - name: http
                  value: 8080
              envs:
                - name: PASS
                  value: <path:kv-v2/data/argocd#password> # (4)

Finally, we can create the ArgoCD Application. It should have the OutOfSync status:

Let’s synchronize its state with the Git repository. We can do it using e.g. ArgoCD UI. If everything works fine you should see the green tile with your application name.

Then, let’s just verify the structure of our app Deployment. You should see the value 123456 instead of the placeholder defined inside the ArgoCD Application.

It is just a formality, but in the end, you can test the endpoint GET /persons/pass exposed by our Spring Boot app. It prints the value of the PASS environment variable. To do that you should also enable port-forwarding for the app.

$ kubectl port-forward svc/simple-helm 8080:8080
$ curl http://localhost:8080/persons/pass

Final Thoughts

GitOps approach becomes very popular in a Kubernetes-based environment. As always, one of the greatest challenges with that approach is security. Hashicorp Vault is one of the best tools for managing and protecting sensitive data. It can be easily installed on Kubernetes and included in your GitOps process. In this article, I showed how to use it together with other very popular solutions for deploying apps: ArgoCD and Helm.

The post Manage Secrets on Kubernetes with ArgoCD and Vault appeared first on Piotr's TechBlog.

Why Spring Cloud Vault on Kubernetes?

First of all, let me explain why I decided to use Spring Cloud instead of Hashicorp’s Vault Agent. It is important to know that Vault Agent is always injected as a sidecar container into the application pod. So even if we have a single secret in Vault and we inject it once on startup there is always one additional container running. I’m not saying it’s wrong, since it is a standard approach on Kubernetes. However, I’m not very happy with it. I also had some problems in troubleshooting with Vault Agent. To be honest, it wasn’t easy to find my mistake in configuration based just on its logs. Anyway, Spring Cloud is an interesting alternative to the solution provided by Hashicorp. It allows you to easily integrate Spring Boot configuration properties with the Vault Database engine. In fact, you just need to include a single dependency to use it.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. To see the sample application go to the kubernetes/sample-db-vault directory. Then you should just follow my instructions

Prerequisites

Before we start, there are some required tools. Of course, we need to have a Kubernetes cluster locally or remotely. Personally, I use Docker Desktop, but you may use any other option you prefer. In order to run Vault on Kubernetes, we need to install Helm.

If you would like to build the application from the source code you need to have Skaffold, Java 17, and Maven. Alternatively, you may use a ready image from my Docker Hub account piomin/sample-app.

Install Vault on Kubernetes with Helm

The recommended way to run Vault on Kubernetes is via the Helm chart. Helm installs and configures all the necessary components to run Vault in several different modes. Firstly, let’s add the HashiCorp Helm repository.

$ helm repo add hashicorp https://helm.releases.hashicorp.com

Before proceeding it is worth updating all the repositories to ensure helm uses the latest versions of the components.

$ helm repo update

Since I will run Vault in the dedicated namespace, we first need to create it.

$ kubectl create ns vault

Finally, we can install the latest version of the Vault server and run it in development mode.

$ helm install vault hashicorp/vault \
    --set "server.dev.enabled=true" \
    -n vault

We can verify the installation by displaying a list of running pods in the vault namespace. As you see the Vault Agent is installed by the Helm Chart, so you can try using it as well. If you wish to just go to this tutorial prepared by HashiCorp.

$ kubectl get pod -n vault
NAME                                    READY   STATUS     RESTARTS   AGE
vault-0                                 1/1     Running    0          1h
vault-agent-injector-678dc584ff-wc2r7   1/1     Running    0          1h

Access Vault on Kubernetes

Before we run our application on Kubernetes, we need to configure several things on Vault. I’ll show you how to do it using the vault CLI. The simplest way to use CLI on Kubernetes is just by getting a shell of a running Vault container:

$ kubectl exec -it vault-0 -n vault -- /bin/sh

Alternatively, we can use Vault Web Console available at the 8200 port. To access it locally we should first enable port forwarding:

$ kubectl port-forward service/vault 8200:8200 -n vault

Now, you access it locally in your web browser at http://localhost:8200. In order to log in there use the Token method (a default token value is root). Then you may do the same as with the vault CLI but with the nice UI.

Configure Kubernetes authentication

Vault provides a Kubernetes authentication method that enables clients to authenticate with a Kubernetes service account token. This token is available to every single pod. Assuming you have already started an interactive shell session on the vault-0 pod just execute the following command:

$ vault auth enable kubernetes

In the next step, we are going to configure the Kubernetes authentication method. We need to set the location of the Kubernetes API, the service account token, its certificate, and the name of the Kubernetes service account issuer (required for Kubernetes 1.21+).

$ vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_PORT_443_TCP_ADDR:443" \
    token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
    kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt \
    issuer="https://kubernetes.default.svc.cluster.local"

Ok, now very important. You need to understand what happened here. We need to create a Vault policy that allows us to generate database credentials dynamically. We will enable the Vault database engine in the next section. For now, we are just creating a policy that will be assigned to the authentication role. The name of our Vault policy is internal-app:

$ vault policy write internal-app - <<EOF
path "database/creds/default" {
  capabilities = ["read"]
}
EOF

The next important thing is related to the Kubernetes RBAC. Although the Vault server is running in the vault namespace our sample application will be running in the default namespace. Therefore, the service account used by the application is also in the default namespace. Let’s create ServiceAccount for the application:

$ kubectl create sa internal-app

Now, we have everything to do the last step in this section. We need to create a Vault role for the Kubernetes authentication method. In this role, we set the name and location of the Kubernetes ServiceAccount and the Vault policy created in the previous step.

$ vault write auth/kubernetes/role/internal-app \
    bound_service_account_names=internal-app \
    bound_service_account_namespaces=default \
    policies=internal-app \
    ttl=24h

After that, we may proceed with the next steps. Let’s enable the Vault database engine.

Enable Vault Database Engine

Just to clarify, we are still inside the vault-0 pod. Let’s enable the Vault database engine.

$ vault secrets enable database

Of course, we need to run a database on Kubernetes. We will PostgreSQL since it is supported by Vault. The full deployment manifest is available on my GitHub repository in /kubernetes/k8s/postgresql-deployment.yaml. Here’s just the Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          env:
            - name: POSTGRES_PASSWORD
              valueFrom:
                secretKeyRef:
                  key: POSTGRES_PASSWORD
                  name: postgres-secret
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-claim

Let’s apply the whole manifest to deploy Postgres in the default namespace:

$ kubectl apply -f postgresql-deployment.yaml

Following Vault documentation, we first need to configure a plugin for the PostgreSQL database and then provide connection settings and credentials:

$ vault write database/config/postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="default" \
    connection_url="postgresql://{{username}}:{{password}}@postgres.default:5432?sslmode=disable" \
    username="postgres" \
    password="admin123"

I have disabled SSL for connection with Postgres by setting the property sslmode=disable. There is only one role allowed to use the Vault PostgresSQL plugin: default. The name of the role should be the same as the name passed in the field allowed_roles in the previous step. We also have to set a target database name and SQL statement that creates users with privileges. We set the max TTL of the lease to 10 minutes just to present revocation and renewal features of Spring Cloud Vault. It means that 10 minutes after your application has started it can no longer authenticate with the database.

$ vault write database/roles/default db_name=postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE,  SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl="1m" \
    max_ttl="10m"

And that’s all on the Vault server side. Now, we can test our configuration using a vault CLI as shown below. You can log in to the database using returned credentials. By default, they are valid for one minute (the default_ttl parameter in the previous command).

$ vault read database/creds/default

We can also verify a connection to the instance of PostgreSQL in Vault UI:

Now, we can generate new credentials just by renewing the Vault lease (vault lease renew LEASE_ID). Hopefully, Spring Cloud Vault does it automatically for our app. Let’s see how it works.

Use Spring Cloud Vault on Kubernetes

For the purpose of this demo, I created a simple Spring Boot application. It exposes REST API and connects to the PostgreSQL database. It uses Spring Data JPA to interact with the database. However, the most important thing here are the following two dependencies:

<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-bootstrap</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>

The first of them enables bootstrap.yml processing on the application startup. The second of them include Spring Cloud Vault Database engine support.

The only thing we need to do is to provide the right configuration settings Here’s the minimal set of the required dependencies to make it work without any errors. The following configuration is provided in the bootstrap.yml file:

spring:
  application:
    name: sample-db-vault
  datasource:
    url: jdbc:postgresql://postgres:5432/postgres #(1)
  jpa:
    hibernate:
      ddl-auto: update
  cloud:
    vault:
      config.lifecycle: #(2)
        enabled: true
        min-renewal: 10s
        expiry-threshold: 30s
      kv.enabled: false #(3)
      uri: http://vault.vault:8200 #(4)
      authentication: KUBERNETES #(5)
      postgresql: #(6)
        enabled: true
        role: default
        backend: database
      kubernetes: #(7)
        role: internal-app

Let’s analyze the configuration visible above in the details:

(1) We need to set the database connection URI, but WITHOUT any credentials. Assuming our application uses standard properties for authentication against the database (spring.datasource.username and spring.datasource.password) we don’t need to anything else

(2) As you probably remember, the max TTL for the database lease is 10 minutes. We enable lease renewal every 30 seconds. Just for the demo purpose. You will see that Spring Cloud Vault will create new credentials in PostgreSQL every 30 seconds, and the application still works without any errors

(3) Vault KV is not needed here, since I’m using only the database engine

(4) The application is going to be deployed in the default namespace, while Vault is running in the vault namespace. So, the address of Vault should include the namespace name

(5) (7) Our application uses the Kubernetes authentication method to access Vault. We just need to set the role name, which is internal-app. All other settings should be left with the default values

(6) We also need to enable postgres database backend support. The name of the backend in Vault is database and the name of Vault role used for that engine is default.

Run Spring Boot application on Kubernetes

The Deployment manifest is rather simple. But what is important here – we need to use the ServiceAccount internal-app used by the Vault Kubernetes authentication method.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app-deployment
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: piomin/sample-app
        ports:
        - containerPort: 8080
      serviceAccountName: internal-app

Our application requires Java 17. Since I’m using Jib Maven Plugin for building images I also have to override the default base image. Let’s use openjdk:17.0.1-slim-buster.

<plugin>
  <groupId>com.google.cloud.tools</groupId>
  <artifactId>jib-maven-plugin</artifactId>
  <version>3.1.4</version>
  <configuration>
    <from>
      <image>openjdk:17.0.1-slim-buster</image>
    </from>
  </configuration>
</plugin>

The repository is configured to easily deploy the application with Skaffold. Just go to the /kubernetes/sample-db-vault directory and run the following command in order to build and deploy our sample application on Kubernetes:

$ skaffold dev --port-forward

After that, you can call one of the REST endpoints to test if the application works properly:

$ curl http://localhost:8080/persons

Everything works fine? In the background, Spring Cloud Vault creates new credentials every 30 seconds. You can easily verify it inside the PostgreSQL container. Just connect to the postgres pod and run the psql process:

$ kubectl exec svc/postgres -i -t -- psql -U postgres

Now you can list users with the \du command. Repeat the command several times to see if the credentials have been regenerated. Of course, the application is able to renew the lease until the max TTL (10 minutes) is not exceeded.

The post Vault on Kubernetes with Spring Cloud appeared first on Piotr's TechBlog.

Before we start with a list of security “golden rules”, let’s analyze a typical microservices architecture. We will focus on components important for building a secure solution.

The picture visible below illustrates a typical microservices architecture built with Spring Cloud. There is an API gateway built on top of Spring Cloud Gateway. Since it is an entry point to our system, we will enable some important security mechanisms on it. There are several microservices hidden behind the gateway. There is also a discovery server, which allows localizing IP addresses using the name of services. And finally, there are some components that do not take part in communication directly. It is just a proposition of a few selected tools. You may choose other solutions providing the same features. Vault is a tool for securely storing and accessing secrets. Keycloak is an open-source identity and access management solution. Spring Cloud Config Server provides an HTTP API for external configuration. It may integrate with several third-party tools including Vault.

Let’s begin. Here’s our list of Spring Security best practices.

1. Enable rate limiting on the API gateway

An API gateway is an important pattern in microservice-based architecture. It acts as a single entry point into the whole system. It is responsible not only for request routing but also for several other things including security. Consequently, one of the most essential components of security we should enable on the gateway is rate limiting. It protects your API against DoS attacks, which can tank a server with unlimited HTTP requests.

Spring provides its own implementation of the API gateway pattern called Spring Cloud Gateway. On the other hand, Spring Cloud Gateway comes with a built-in implementation of a rate-limiting component. To sum up, you just need to include a single dependency to build a gateway application. Then you have to provide some configuration settings to enable rate limiting for a single route.

In order to enable a rate limiter on a gateway, we need to use a component called RequestRateLimiter GatewayFilter factory. It uses a RateLimiter implementation to determine if the current request is allowed to proceed. Otherwise, the gateway returns a status of HTTP 429 - Too Many Requests. The RequestRateLimiter implementation uses Redis as a backend.

Let’s take a look at a typical configuration of routes handled by Spring Cloud Gateway. There are three parameters that can be used to configure the rate limiter: replenishRate, burstCapacity, and requestedTokens. The replenishRate property is how many requests per second a single user may send. The burstCapacity property is the maximum number of requests a user can send in a single second. With the requestedTokens property, we may set the cost of a single token.

spring:
  cloud:
    gateway:
      routes:
        - id: account-service
          uri: http://localhost:8090
          predicates:
            - Path=/account/**
          filters:
            - RewritePath=/account/(?<path>.*), /$\{path}
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 1
                redis-rate-limiter.burstCapacity: 60
                redis-rate-limiter.requestedTokens: 15

Then we should define a KeyResolver bean. A rate limiter defines all parameters per a single key returned by the resolver.

@Bean
KeyResolver authUserKeyResolver() {
   return exchange -> ReactiveSecurityContextHolder.getContext()
            .map(ctx -> ctx.getAuthentication().getPrincipal().toString());
}

For more details, you may refer to the article Secure Rate Limiting with Spring Cloud Gateway.

2. Generate and propagate certificates dynamically

Should we use SSL in microservice-to-microservice communication? Of course yes. But the question is how will you handle certificates used by your microservices. There are several best practices related to SSL certificate management. For example, you should not issue certificates for long time periods. You should also automatically renew or refresh them.

There are some tools that can help in following best practices. One of the most popular of them is Vault from Hashicorp. It provides the PKI secrets engine, which is responsible for generating dynamic X.509 certificates. The simplest way to try Vault is to run it locally on a Docker container.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

Then, we may enable and configure the PKI engine in some simple steps. You can do it using Vault CLI or UI. For some more detailed information about it read my article SSL with Spring WebFlux and Vault PKI. For now, let’s enable PKI with TTL and then configure CA using CLI as shown below.

$ vault secrets enable pki
$ vault secrets tune -max-lease-ttl=8760h pki
$ vault write pki/root/generate/internal \
    common_name=my-website.com \
    ttl=8760h

In the next step, we will use Spring VaultTemplate to issue a certificate dynamically. The fragment of code visible below shows how to create a certificate request with 12h TTL and localhost as a Common Name. Firstly, let’s build such a request using the VaultCertificateRequest object. Then we will invoke the issueCertificate method on the VaultPkiOperations object. The generated CertificateBundle contains both a certificate and a private key.

private CertificateBundle issueCertificate() throws Exception {
   VaultPkiOperations pkiOperations = vaultTemplate.opsForPki("pki");
   VaultCertificateRequest request = VaultCertificateRequest.builder()
        .ttl(Duration.ofHours(12))
        .commonName("localhost")
        .build();
   VaultCertificateResponse response = pkiOperations
      .issueCertificate("default", request);
   CertificateBundle certificateBundle = response.getRequiredData();
   return certificateBundle;
}

Finally, we just need to use the method for generating a certificate in Vault on runtime. The default behavior of our web server needs to be overridden. To do that, we need to create a Spring @Component that implements WebServerFactoryCustomizer. Depending on the web server we need to customize a different WebServerFactory. Typically, for Spring MVC it is Tomcat and Netty for Spring WebFlux. Inside the customize method, we are generating a certificate and store it inside the keystore (cert + private key) and truststore (CA).

@Component
public class GatewayServerCustomizer implements 
         WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
   @SneakyThrows
   @Override
   public void customize(NettyReactiveWebServerFactory factory) {
      String keyAlias = "vault";
      CertificateBundle bundle = issueCertificate();
      KeyStore keyStore = bundle.createKeyStore(keyAlias);
      String keyStorePath = saveKeyStoreToFile("server-key.pkcs12", keyStore);
      Ssl ssl = new Ssl();
      ssl.setEnabled(true);
      ssl.setClientAuth(Ssl.ClientAuth.NEED);
      ssl.setKeyStore(keyStorePath);
      ssl.setKeyAlias(keyAlias);
      ssl.setKeyStoreType(keyStore.getType());
      ssl.setKeyPassword("");
      ssl.setKeyStorePassword("123456");
      X509Certificate caCert = bundle.getX509IssuerCertificate();
      log.info("CA-SerialNumber: {}", caCert.getSerialNumber());
      KeyStore trustStore = KeyStore.getInstance("pkcs12");
      trustStore.load(null, null);
      trustStore.setCertificateEntry("ca", caCert);
      String trustStorePath = saveKeyStoreToFile("server-trust.pkcs12", trustStore);
      ssl.setTrustStore(trustStorePath);
      ssl.setTrustStorePassword("123456");
      ssl.setTrustStoreType(trustStore.getType());
      factory.setSsl(ssl);
      factory.setPort(8443);
   }
}

3. Use SSL in microservices communication

Since using SSL on the edge of a microservices-based system is obvious, inter-service communication is sometimes considered to be non-secure. My recommendation is always the same. Better use SSL or not But we should think about securing at least components that store sensitive data. One of them will probably be a config server. That’s obviously one of the Spring Security best practices. For Spring Boot microservices we can use a component called Spring Cloud Config Server. Since it is built on top of Spring MVC we may easily enable a secure connection on the server-side.

server:
  port: ${PORT:8888}
  ssl:
    enabled: true
    client-auth: need
    key-store: classpath: server-key.jks
    key-store-password: 123456
    key-alias: configserver
    trust-store: classpath: server-trust.jks
    trust-store-password: 123456

On the client side, we use a component called Spring Cloud Config Client. Since it is responsible for connecting with the server, we also need to handle SSL there. To do that we need to override the RestTemplate SSL configuration on the ConfigServicePropertySourceLocator bean. The fragment of code visible below uses a self-signed certificate, but we can easily implement here a strategy described in the previous section.

@Configuration
public class SSLConfigServiceBootstrapConfiguration {
   @Autowired
   ConfigClientProperties properties;
   @Bean
   public ConfigServicePropertySourceLocator configServicePropertySourceLocator() throws Exception {
      final char[] password = "123456".toCharArray();
      final ClassPathResource resource = new ClassPathResource("account.jks");
      SSLContext sslContext = SSLContexts.custom()
         .loadKeyMaterial(resource.getFile(), password, password)
         .loadTrustMaterial(resource.getFile(), password, new TrustSelfSignedStrategy()).build();
      CloseableHttpClient httpClient = HttpClients.custom()
         .setSSLContext(sslContext)
         .setSSLHostnameVerifier((s, sslSession) -> true)
         .build();
      HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
      ConfigServicePropertySourceLocator configServicePropertySourceLocator = new ConfigServicePropertySourceLocator(properties);
      configServicePropertySourceLocator.setRestTemplate(new RestTemplate(requestFactory));
      return configServicePropertySourceLocator;
   }
}

In the latest version of Spring Cloud Config, we can enable TLS traffic encryption in configuration. We just need to define the right settings using properties with a prefix spring.cloud.config.tls.*.

What about encrypting communication between applications and a discovery server? You can choose between several available discovery servers supported in Spring Cloud. But let’s assume we use Eureka. Similarly to Spring Cloud Config, we use a high-level client to communicate with a server. So, in that case, we need to define a bean DiscoveryClientOptionalArgs, and also override SSL settings on the HTTP client there. The Eureka client uses the Jersey HTTP client, so we need to create an instance of EurekaJerseyClientBuilder to override the SSL configuration.

@Bean
public DiscoveryClient.DiscoveryClientOptionalArgs discoveryClientOptionalArgs() throws Exception {
   DiscoveryClient.DiscoveryClientOptionalArgs args = new DiscoveryClient.DiscoveryClientOptionalArgs();
   final char[] password = "123456".toCharArray();
   final ClassPathResource resource = new ClassPathResource("account.jks");
   SSLContext sslContext = SSLContexts.custom()
	.loadKeyMaterial(resource.getFile(), password, password)
	.loadTrustMaterial(resource.getFile(), password, new TrustSelfSignedStrategy()).build();
   EurekaJerseyClientBuilder builder = new EurekaJerseyClientBuilder();	 
   builder.withClientName("account-client");
   builder.withMaxTotalConnections(10);
   builder.withMaxConnectionsPerHost(10);
   builder.withCustomSSL(sslContext);
   args.setEurekaJerseyClient(builder.build());
   return args;
}

Finally, we may configure HTTPS in communication between microservices. Since we use RestTemplate or WebClient instances directly on the client side it is relatively easy to implement secure communication in that case.

4. Keep configuration data encrypted

The current one of the best practices for Spring microservices security is related to a configuration server. We should encrypt at least sensitive data like passwords or secrets stored there. Spring Cloud Config Server provides a built-in mechanism for that. But we can also use Vault as a backend store for Spring Cloud Config Server, where all data is encrypted by default.

We will start with a default encrypt mechanism provided by Spring Cloud Config Server. Firstly, we need to enable it in the configuration properties.

spring:
  cloud:
    config:
      server:
        encrypt:
          enabled: true

Then, we have to configure a key store responsible for encrypting our sensitive data.

encrypt:
  keyStore:
    location: classpath:/config.jks
    password: 123456
    alias: config
    secret: 123456

Finally, we can set encrypted data instead of plain string with {cipher} prefix.

spring:  
  application:
    name: account-service
  security:
    user:
      password: '{cipher}AQBhpDVYHANrg59OGY7ioSbMdOrH7ZA0vfa2VqIvfxJK5vQp...'

Alternatively, you can use it as a configuration data backend. To do that you should enable a Spring profile called vault.

spring.profiles.active=vault

Then, we may add an example secret.

$ vault write secret/hello value=world
$ vault read secret/hello

5. Restrict access to the API resources

In the previous sections, we discussed such topics as authentication, traffic, and data encryption. But another important aspect of securing your applications is authorization and access to the API resources. If you think about web app authorization, the first approach that probably comes to your mind is OAuth 2.0 or OpenID Connect. OAuth 2.0 is the industry-standard protocol for authorization. Of course, it is supported by Spring Security. There are also multiple OAuth2 providers you can integrate your application with. One of them is Keycloak. I will use it in the example in this article. Firstly, let’s run Keycloak on a Docker container. By default, it exposes API and a web console on the port 8080.

$ docker run -d --name keycloak -p 8888:8080 \
   -e KEYCLOAK_USER=spring \
   -e KEYCLOAK_PASSWORD=spring123 \
   jboss/keycloak

We are going to enable and configure OAuth 2.0 support on the API gateway. Besides spring-cloud-starter-gateway dependency, we need to include spring-boot-starter-oauth2-client and spring-cloud-starter-security to activate the TokenRelay filter. Then we have to provide the Spring Security configuration settings for the OAuth2 client.

spring:
  security:
    oauth2:
      client:
        provider:
          keycloak:
            token-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/token
            authorization-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/auth
            userinfo-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/userinfo
            user-name-attribute: preferred_username
        registration:
          keycloak-with-test-scope:
            provider: keycloak
            client-id: spring-with-test-scope
            client-secret: c6480137-1526-4c3e-aed3-295aabcb7609
            authorization-grant-type: authorization_code
            redirect-uri: "{baseUrl}/login/oauth2/code/keycloak"
          keycloak-without-test-scope:
            provider: keycloak
            client-id: spring-without-test-scope
            client-secret: f6fc369d-49ce-4132-8282-5b5d413eba23
            authorization-grant-type: authorization_code
            redirect-uri: "{baseUrl}/login/oauth2/code/keycloak"

In the last step, we need to configure the Spring Security filter. Since Spring Cloud Gateway is built on top of Spring WebFlux, we need to annotate the configuration bean with @EnableWebFluxSecurity. Inside the filterChain method we are going to enable authorization for all the exchanges. We will also set OAuth2 as a default login method and finally disable CSRF.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {
   @Bean
   public SecurityWebFilterChain filterChain(ServerHttpSecurity http) {
      http.authorizeExchange(exchanges -> exchanges.anyExchange().authenticated())
         .oauth2Login(withDefaults());
      http.csrf().disable();
      return http.build();
   }
}

As I mentioned before, we will have a token relay pattern between the gateway and microservices. A Token Relay is where an OAuth2 consumer acts as a Client and forwards the incoming token to outgoing resource requests. So, now let’s enable global method security and OAuth2 resources server for the downstream services.

@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfig extends WebSecurityConfigurerAdapter {
   protected void configure(HttpSecurity http) throws Exception {
      http.authorizeRequests(authorize -> authorize.anyRequest().authenticated())
         .oauth2ResourceServer(OAuth2ResourceServerConfigurer::jwt);
   }
}

After that, it is possible to configure role-based access using @PreAuthorize and @PostAuthorize. Let’s take a look at the implementation of the REST controller class. It is a single ping method. That method may be accessed only by the client with the TEST scope. For more implementation details you may refer to the article Spring Cloud Gateway OAuth2 with Keycloak.

@RestController
@RequestMapping("/callme")
public class CallmeController {
   @PreAuthorize("hasAuthority('SCOPE_TEST')")
   @GetMapping("/ping")
   public String ping() {
      SecurityContext context = SecurityContextHolder.getContext();
      Authentication authentication = context.getAuthentication();
      return "Scopes: " + authentication.getAuthorities();
   }
}

6. Dynamically generate credentials to the external systems

Does your application connect to external systems like databases or message brokers? How do you store the credentials used by your application? Of course, we can always encrypt sensitive data, but if we work with many microservices having separate databases it may not be a very comfortable solution. Here comes Vault with another handy mechanism. Its database secrets engine generates database credentials dynamically based on configured roles. We may also take advantage of dynamically generated credentials for RabbitMQ, Nomad, and Consul.

Firstly, let’s enable the Vault database engine, which is disabled by default.

$ vault secrets enable database

Let’s assume our application connects to the Postgres database. Therefore, we need to configure a Vault plugin for PostgreSQL database and then provide connection settings and credentials.

$ vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="default" \
  connection_url="postgresql://{{username}}:{{password}}@localhost:5432?sslmode=disable" \
  username="postgres" \
  password="postgres123456"

Then we need to create a database role. The name of the role should be the same as the name passed in field allowed_roles in the previous step. We also have to set a target database name and SQL statement that creates users with privileges.

$ vault write database/roles/default db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE,  SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" max_ttl="24h"

Thanks to Spring Cloud Vault project we can easily integrate any Spring Boot application with the Vault databases engine. Two dependencies need to be included in Maven pom.xml to enable that support. Of course, we also need dependencies for the JPA and Postgres driver.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>org.postgresql</groupId>
   <artifactId>postgresql</artifactId>
</dependency>

The only thing we have to do is to configure integration with Vault via Spring Cloud Vault. The following configuration settings should be placed in bootstrap.yml (no application.yml). You may consider running the application on Nomad.

spring:
  cloud:
    vault:
      uri: http://localhost:8200
      token: ${VAULT_TOKEN}
      postgresql:
        enabled: true
        role: default
        backend: database
  datasource:
    url: jdbc:postgresql://localhost:5432/posgtres

The important part of the configuration visible above is under the property spring.cloud.vault.postgresql. Following Spring Cloud documentation “Username and password are stored in spring.datasource.username and spring.datasource.password so using Spring Boot will pick up the generated credentials for your DataSource without further configuration”. For more details about integration between Spring Cloud and Vault database engine, you may refer to my article Secure Spring Cloud Microservices with Vault and Nomad.

7. Always be up to date

This one of the best practices may be applied anywhere not only as a rule to Spring microservices security. We usually use open-source libraries in our applications, so it is important to include the latest versions of them. They may contain critical updates for publicly disclosed vulnerabilities contained within a project’s dependencies. There are also several dependency scanners like Snyk or OWASP.

Final thoughts

That is my private list of best practices for Spring Boot microservices security. Of course, most of them are not related just to a single framework, and we apply them for any other framework or toolkit. Do you have your own list of Spring Security best practices? Don’t afraid to share it in the comments.

The post Spring Microservices Security Best Practices appeared first on Piotr's TechBlog.

Let’s say a little bit more about Vault. It allows you to secure, store, and control access to tokens, passwords, certificates, and encryption keys using UI, CLI, or HTTP API. It is a really powerful tool. With Vault, instead of a traditional approach, you can manage your security in a more dynamic, cloud-native way. For example, you can integrate Vault with a database backend, and then generate user login and password on the fly. Moreover, for Spring Boot applications you can take an advantage of the Spring Cloud Vault project. If you are interested in more information about it read my article Testing Spring Boot Integration with Vault and Postgres using Testcontainers.

That’s not all. You can integrate Vault with other tools from Hashicorp like Consul or Nomad. In other words, it allows us to build a cloud-native platform in a secure way. For more details please refer to the article Secure Spring Cloud Microservices with Vault and Nomad.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my repository sample-spring-cloud-security. Then you should go to the gateway-service directory, and just follow my instructions in the next sections. The sample application acts as an API gateway for microservices. We use Spring Cloud Gateway. Since it is built on top of Spring WebFlux, that example is perfectly right for our current article.

1. Running Vault

We will run Vault inside the Docker container in development mode. The server running in that mode does not require any further setup, it is ready to use just after startup. After startup, our instance of Vault is available on port 8200. The version of Vault used in this article is 1.7.1.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

It is possible to login using different methods, but the most suitable way for us is through a token. To do that we have to display container logs using command docker logs vault, and then copy Root Token as shown below.

Finally, we are able to login to the Vault web console.

2. Enable and configure Vault PKI

There are two ways to enable and configure Vault PKI: with CLI or via UI. Most articles describe a list of CLI commands required to configure the PKI engine. One of them is available on the Hashicorp site. However, I’m going to use Vault UI for that. So first, let’s enable a new engine on the main site.

Then, we need to choose a type of engine to enable. In our case it is the option PKI Certificates.

During creation let’s leave a default name pki. Then, we need to navigate into the newly enabled engine and create a new role. A role is used for generating certificates. The name of my role is default. This name is important because we would have to call it from the code using VaultTemplate.

The type of the key used in our role is rsa.

Before creating it, we should set some important parameters. One of them is TTL, which is set to 3 days. Also, don’t forget to check fields Allow any name and Require Common Name. Both of them are related to the CN field inside the certificate. Because we will store a username inside the CN field, we need to allow any name for it.

Once a role is created, we need to configure CA. To do that, we should first switch to the Configuration tab and then click Configure button.

After that, let’s choose Configure CA.

Finally, we can create a new CA certificate. We should leave the root value as CA Type and internal as Type. The default key format is pem. We can also set a Common Name for the CA certificate. For both role and CA it worth filling additional fields like e.g. the name of an organization or organization unit.

3. Integrating Spring WebFlux with Vault PKI

Let’s begin with dependencies. We need to include a Spring WebFlux starter for reactive API and a Spring Security starter to secure API. Integration with Vault API can be provided by spring-vault-core. I also had to include Jackson libraries in order to be able to start the application with Spring Vault.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-cloud-starter-webflux</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-core</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-annotations</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.vault</groupId>
   <artifactId>spring-vault-core</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Then, let’s configure a VaultTemplate bean. It should use the http scheme and an authentication token injected from the configuration.

@Value("vault.token")
private String vaultToken;

@Bean
VaultTemplate vaultTemplate() {
   VaultEndpoint e =  new VaultEndpoint();
   e.setScheme("http");
   VaultTemplate template = new VaultTemplate(e, new TokenAuthentication(vaultToken));
   return template;
}

The VaultTemplate provides dedicated support for interaction with the PKI engine. We just need to call the method opsForPki passing the PKI engine name to obtain the VaultPkiOperations instance (1). Then we need to build a certificate request with VaultCertificateRequest. We may set several parameters, but the most important is CN and certificate TTL (2). Finally, we should invoke the issueCertificate method passing the request and the name of the role configured on Vault PKI (3). Our certificate has been successfully generated. Now, we just need to obtain it from the response. The generated certificate, CA certificate, and a private key are available inside the CertificateBundle object, which is returned by the method.

private CertificateBundle issueCertificate() throws Exception {
   VaultPkiOperations pkiOperations = vaultTemplate.opsForPki("pki"); // (1)
   VaultCertificateRequest request = VaultCertificateRequest.builder()
        .ttl(Duration.ofHours(12))
        .commonName("localhost")
        .build(); // (2)
   VaultCertificateResponse response = pkiOperations.issueCertificate("default", request); // (3)
   CertificateBundle certificateBundle = response.getRequiredData(); // (4)

   log.info("Cert-SerialNumber: {}", certificateBundle.getSerialNumber());
   return certificateBundle;
}

4. Enable Spring WebFlux security

We have already integrated Spring WebFlux with Vault PKI in the previous section. Finally, we can proceed to the last step in our implementation – enable security based on X.509 certificates. To do that we need to create a @Configuration class. It should be annotated with @EnableWebFluxSecurity (1). We also need to obtain a username from the certificate by implementing the principal extractor. We are going to use SubjectDnX509PrincipalExtractor (2) with the right regex that reads data from the CN field. The final configuration disables CSRF, basic auth, and enables SSL with X509 certificates (3). We also need to provide an implementation of the UserDetails interface (4) with a single username piotrm.

@Configuration
@EnableWebFluxSecurity // (1)
public class SecurityConfig {

   @Bean
   public SecurityWebFilterChain filterChain(ServerHttpSecurity http) {
      SubjectDnX509PrincipalExtractor principalExtractor =
             new SubjectDnX509PrincipalExtractor(); // (2)
      principalExtractor.setSubjectDnRegex("CN=(.*?)(?:,|$)");

      return http.csrf().disable()
             .authorizeExchange(exchanges -> 
                    exchanges.anyExchange().authenticated())
             .x509()
                .principalExtractor(principalExtractor)
             .and()
                .httpBasic().disable().build(); // (3)
   }

   @Bean
   public MapReactiveUserDetailsService users() { // (4)
      UserDetails user1 = User.builder()
             .username("piotrm")
             .password("{noop}1234")
             .roles("USER")
             .build();
      return new MapReactiveUserDetailsService(user1);
   }
}

In the last step, we need to override the Netty server SSL configuration on runtime. Our customizer should implement the WebServerFactoryCustomizer interface, and use NettyReactiveWebServerFactory. Inside customize method we first invoke the method issueCertificate responsible for generating a certificate in Vault (you can refer to the previous section to see the implementation of that method) (1). The CertificateBundle contains all required data. We can invoke the method createKeyStore on it to create a keystore (2) and then save it in the file (3).

To override Netty SSL settings we should use the Ssl object. The client authentication needs to be enabled (4). We will also set the location of the currently created KeyStore (5). After that, we may proceed to the truststore creation. The issuer certificate may be obtained from CertificateBundle (6). Then we should create a new keystore, and set the CA certificate as an entry there (7). Finally, we will save the truststore to the file and set its location in the Ssl object.

@Component
@Slf4j
public class GatewayServerCustomizer implements 
         WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {

   @SneakyThrows
   @Override
   public void customize(NettyReactiveWebServerFactory factory) {
      String keyAlias = "vault";
      CertificateBundle bundle = issueCertificate(); // (1)
      KeyStore keyStore = bundle.createKeyStore(keyAlias); // (2)
      String keyStorePath = saveKeyStoreToFile("server-key.pkcs12", keyStore); // (3)

      Ssl ssl = new Ssl();
      ssl.setEnabled(true);
      ssl.setClientAuth(Ssl.ClientAuth.NEED); // (4)

      ssl.setKeyStore(keyStorePath); // (5)
      ssl.setKeyAlias(keyAlias);
      ssl.setKeyStoreType(keyStore.getType());
      ssl.setKeyPassword("");
      ssl.setKeyStorePassword("123456");

      X509Certificate caCert = bundle.getX509IssuerCertificate(); // (6)
      log.info("CA-SerialNumber: {}", caCert.getSerialNumber());
      KeyStore trustStore = KeyStore.getInstance("pkcs12");
      trustStore.load(null, null);
      trustStore.setCertificateEntry("ca", caCert); // (7)
      String trustStorePath = saveKeyStoreToFile("server-trust.pkcs12", trustStore); // (8)

      ssl.setTrustStore(trustStorePath); // (9)
      ssl.setTrustStorePassword("123456");
      ssl.setTrustStoreType(trustStore.getType());

      factory.setSsl(ssl);
      factory.setPort(8443);
   }
}

5. Testing Spring WebFlux with Vault PKI

Let’s run our sample application. It is available under the 8443 port. We will test it using the curl tool. Before doing it we need to generate a client certificate with a private key. Let’s go to the Vault UI once again. If you click a default Vault UI redirects to form responsible for certificate generation as shown below. In the Common Name field, we should provide the test username configured inside the UserDetails implementation. For me, it is piotrm. Also, don’t forget to set the right TTL.

After generating a certificate you will be redirected to the site with the results. First, you should copy the string with your certificate, and save it to the file. For me it is piotrm.crt. You can also display the content of a generated private key. Then, do the same as with the certificate. My filename is piotrm.key.

Finally, we can send a test request to our sample application passing the names of key and certificate files.

$ curl https://localhost:8443/hello -v --key piotrm.key --cert piotrm.crt

The post SSL with Spring WebFlux and Vault PKI appeared first on Piotr's TechBlog.

Ok, we know how it works, now the question is how to automatically test it. With Testcontainers it is possible with just a few lines of code.

1. Building application

Let’s begin from a short intro to the application code. It is very simple. Here’s the list of dependencies required for building an application that exposes REST API, and integrates with Postgres and Vault.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>org.postgresql</groupId>
   <artifactId>postgresql</artifactId>
   <version>42.2.5</version>
</dependency>

Application connects to Postgres, enables integration with Vault via Spring Cloud Vault, and automatically creates/updates tables on startup.

spring:
  application:
    name: callme-service
  cloud:
    vault:
      uri: http://192.168.99.100:8200
      token: ${VAULT_TOKEN}
      postgresql:
        enabled: true
        role: default
        backend: database
  datasource:
    url: jdbc:postgresql://192.168.99.100:5432/postgres
  jpa.hibernate.ddl-auto: update

It exposes the single endpoint. The following method is responsible for handling incoming requests. It just inserts a record to the database and returns a response with app name, version and id of inserted record.

@RestController
@RequestMapping("/callme")
public class CallmeController {

   private static final Logger LOGGER = LoggerFactory.getLogger(CallmeController.class);
   
   @Autowired
   Optional<BuildProperties> buildProperties;
   @Autowired
   CallmeRepository repository;
   
   @GetMapping("/message/{message}")
   public String ping(@PathVariable("message") String message) {
      Callme c = repository.save(new Callme(message, new Date()));
      if (buildProperties.isPresent()) {
         BuildProperties infoProperties = buildProperties.get();
         LOGGER.info("Ping: name={}, version={}", infoProperties.getName(), infoProperties.getVersion());
         return infoProperties.getName() + ":" + infoProperties.getVersion() + ":" + c.getId();
      } else {
         return "callme-service:"  + c.getId();
      }
   }
   
}

2. Enabling Testcontainers

To enable Testcontainers for our project we need to include some dependencies to our Maven pom.xml. We have dedicated modules for Postgres and Vault. We also include Spring Boot Test dependency, because we would like to test the whole Spring Boot app.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-test</artifactId>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.testcontainers</groupId>
   <artifactId>vault</artifactId>
   <version>1.10.5</version>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.testcontainers</groupId>
   <artifactId>testcontainers</artifactId>
   <version>1.10.5</version>
   <scope>test</scope>
</dependency>
<dependency>
   <groupId>org.testcontainers</groupId>
   <artifactId>postgresql</artifactId>
   <version>1.10.5</version>
   <scope>test</scope>
</dependency>

3. Running Vault test container

Testcontainers framework supports JUnit 4/JUnit 5 and Spock. The Vault container can be started before tests if it is annotated with @Rule or @ClassRule. By default it uses version 0.7, but we can override it with the newest version, which is 1.0.2. We also may set a root token, which is then required by Spring Cloud Vault for integration with Vault.

@ClassRule
public static VaultContainer vaultContainer = new VaultContainer<>("vault:1.0.2")
   .withVaultToken("123456")
   .withVaultPort(8200);

That root token can be overridden before starting a JUnit test on the test class.

@RunWith(SpringRunner.class)
@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT, properties = {
    "spring.cloud.vault.token=123456"
})
public class CallmeTest { ... }

4. Running Postgres with Testcontainers

As an alternative to @ClassRule, we can manually start the container in a @BeforeClass or @Before method in the test. With this approach you will also have to stop it manually in @AfterClass or @After method. We start the Postgres container manually, because by default it is exposed on a dynamically generated port, which needs to be set for the Spring Boot application before starting the test. The listen port is returned by method getFirstMappedPort invoked on PostgreSQLContainer.

private static PostgreSQLContainer postgresContainer = new PostgreSQLContainer()
   .withDatabaseName("postgres")
   .withUsername("postgres")
   .withPassword("postgres123");
   
@BeforeClass
public static void init() throws IOException, InterruptedException {
   postgresContainer.start();
   int port = postgresContainer.getFirstMappedPort();
   System.setProperty("spring.datasource.url", String.format("jdbc:postgresql://192.168.99.100:%d/postgres", postgresContainer.getFirstMappedPort()));
   // ...
}

@AfterClass
public static void shutdown() {
   postgresContainer.stop();
}

5. Integrating Vault and Postgres containers

Once we have succesfully started both Vault and Postgres containers, we need to integrate them via Vault secret engine. First, we need to enable database secret engine Vault. After that we must configure a connection to Postgres. The last step is to configure a role. A role is a logical name that maps to a policy used to generate those credentials. All these actions may be performed using Vault commands. You can launch commands on the Vault container using execInContainer method. Vault configuration commands should be executed just after Postgres container startup.

@BeforeClass
public static void init() throws IOException, InterruptedException {
   postgresContainer.start();
   int port = postgresContainer.getFirstMappedPort();
   System.setProperty("spring.datasource.url", String.format("jdbc:postgresql://192.168.99.100:%d/postgres", postgresContainer.getFirstMappedPort()));
   vaultContainer.execInContainer("vault", "secrets", "enable", "database");
   String url = String.format("connection_url=postgresql://{{username}}:{{password}}@192.168.99.100:%d?sslmode=disable", port);
   vaultContainer.execInContainer("vault", "write", "database/config/postgres", "plugin_name=postgresql-database-plugin", "allowed_roles=default", url, "username=postgres", "password=postgres123");
   vaultContainer.execInContainer("vault", "write", "database/roles/default", "db_name=postgres",
      "creation_statements=CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE,  SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";",
      "default_ttl=1h", "max_ttl=24h");
}

6. Running application tests

Finally, we may run application tests. We just call the single endpoint exposed by the app using TestRestTemplate, and verify the output.

@Autowired
TestRestTemplate template;

@Test
public void test() {
   String res = template.getForObject("/callme/message/{message}", String.class, "Test");
   Assert.assertNotNull(res);
   Assert.assertTrue(res.endsWith("1"));
}

If you are interested in what exactly happens during the test you can set a breakpoint inside the test method and execute docker ps command manually.

The post Testing Spring Boot Integration with Vault and Postgres using Testcontainers Framework appeared first on Piotr's TechBlog.

• Manually with rabbitmqctl (as described in my article RabbitMQ in cluster)
• Declaratively by listing cluster nodes in config file
• Using DNS-based discovery
• Using AWS (EC2) instance discovery via a dedicated plugin
• Using Kubernetes discovery via a dedicated plugin
• Using Consul discovery via a dedicated plugin
• Using etcd-based discovery via a dedicated plugin

Today, I’m going to show you how to create RabbitMQ cluster using service discovery based on HashiCorp’s Consul. Additionally, we will include Vault to our architecture in order to use its interesting feature called secrets engine for managing credentials used for accessing RabbitMQ. We will set up this sample on the local machine using Docker images of RabbitMQ, Consul and Vault. Finally, we will test our solution using a simple Spring Boot application that sends and listens for incoming messages to the cluster. That application is available on GitHub repository sample-haclustered-rabbitmq-service in the branch consul.

Architecture

We use Vault as a credentials manager when applications try to authenticate against RabbitMQ nodes or when a user tries to login to RabbitMQ web admin console. Each RabbitMQ node registers itself after startup in Consul and retrieves a list of nodes running inside a cluster. Vault is integrated with RabbitMQ using a dedicated secrets engine. Here’s an architecture of our sample solution.

1. Configure RabbitMQ Consul plugin

The integration between RabbitMQ and Consul is realized via plugin rabbitmq-peer-discovery-consul. This plugin is not enabled by default on the official RabbitMQ Docker container. So, the first step is to build our own Docker image based on the official RabbitMQ image that installs and enables the required plugin. By default, RabbitMQ main configuration file is available under path /etc/rabbitmq/rabbitmq.conf inside a Docker container. To override it we just use the COPY statement as shown below. The following Dockerfile definition takes RabbitMQ with a management web console as base image and enables rabbitmq_peer_discovery_consul plugin.

FROM rabbitmq:3.7.8-management
COPY rabbitmq.conf /etc/rabbitmq
RUN rabbitmq-plugins enable --offline rabbitmq_peer_discovery_consul

Now, let’s take a closer look on our plugin configuration settings. Because I run Docker on Windows Consul is not available under default localhost address, but on 192.168.99.100. So, first we need to set that IP address using property cluster_formation.consul.host. We also need to set Consul as a default peer discovery implementation by setting the name of plugin for property cluster_formation.peer_discovery_backend. Finally, we have to set two additional properties to make it work in our local Docker environment. It is related to the address of RabbitMQ node sent to Consul during the registration process. It is important to compute it properly, and not to send for example localhost. After setting property cluster_formation.consul.svc_addr_use_nodename to false node will register itself using host name instead of node name. We can set the name of the host for the container inside its running command. Here’s my full RabbitMQ configuration file used in the demo for this article.

loopback_users.guest = false
listeners.tcp.default = 5672
hipe_compile = false
management.listener.port = 15672
management.listener.ssl = false
cluster_formation.peer_discovery_backend = rabbit_peer_discovery_consul
cluster_formation.consul.host = 192.168.99.100
cluster_formation.consul.svc_addr_auto = true
cluster_formation.consul.svc_addr_use_nodename = false

After saving the configuration visible above in the file rabbitmq.conf we can proceed to building our custom Docker image with RabbitMQ. This image is available in my Docker repository under alias piomin/rabbitmq, but you can also build it by yourself from Dockerfile by executing the following command.

$ docker build -t piomin/rabbitmq:1.0 .
Sending build context to Docker daemon  3.072kB
Step 1 : FROM rabbitmq:3.7.8-management
 ---> d69a5113ceae
Step 2 : COPY rabbitmq.conf /etc/rabbitmq
 ---> aa306ef88085
Removing intermediate container fda0e21178f9
Step 3 : RUN rabbitmq-plugins enable --offline rabbitmq_peer_discovery_consul
 ---> Running in 0892a42bffef
The following plugins have been configured:
  rabbitmq_management
  rabbitmq_management_agent
  rabbitmq_peer_discovery_common
  rabbitmq_peer_discovery_consul
  rabbitmq_web_dispatch
Applying plugin configuration to rabbit@fda0e21178f9...
The following plugins have been enabled:
  rabbitmq_peer_discovery_common
  rabbitmq_peer_discovery_consul

set 5 plugins.
Offline change; changes will take effect at broker restart.
 ---> cfe73f9d9904
Removing intermediate container 0892a42bffef
Successfully built cfe73f9d9904

2. Running RabbitMQ cluster on Docker

In the previous step we have succesfully created a Docker image of RabbitMQ configured to run in cluster mode using Consul discovery. Before running this image we need to start an instance of Consul. Here’s the command that starts the Docker container with Consul and exposes it on port 8500.

$ docker run -d --name consul -p 8500:8500 consul

We will also create a Docker network to enable communication between containers by hostname. It is required in this scenario, because each RabbitMQ container is registered using container hostname.

$ docker network create rabbitmq

Now, we can run our three clustered RabbitMQ containers. We will set a unique hostname for every single container (using -h option) and set the same Docker network everywhere. We also have to set the environment variable RABBITMQ_ERLANG_COOKIE.

$ docker run -d --name rabbit1 -h rabbit1 --network rabbitmq -p 30000:5672 -p 30010:15672 -e RABBITMQ_ERLANG_COOKIE='rabbitmq' piomin/rabbitmq:1.0
$ docker run -d --name rabbit2 -h rabbit2 --network rabbitmq -p 30001:5672 -p 30011:15672 -e RABBITMQ_ERLANG_COOKIE='rabbitmq' piomin/rabbitmq:1.0
$ docker run -d --name rabbit3 -h rabbit3 --network rabbitmq -p 30002:5672 -p 30012:15672 -e RABBITMQ_ERLANG_COOKIE='rabbitmq' piomin/rabbitmq:1.0

After running all three instances of RabbitMQ we can first take a look on the Consul web console. You should see there the new service called rabbitmq. This value is the default name of a cluster set by RabbitMQ Consul plugin. We can override inside rabbitmq.conf using cluster_formation.consul.svc property.

We can check out if the cluster has been succesfully started using RabbitMQ web management console. Every node is exposing it. I just had to override default port 15672 to avoid port conflicts between three running instances.

3. Integrating RabbitMQ with Vault

In the two previous steps we have succesfully run the cluster of three RabbitMQ nodes based on Consul discovery. Now, we will include Vault to our sample system to dynamically generate user credentials. Let’s begin from running Vault on Docker. You can find detailed information about it in my previous article Secure Spring Cloud Microservices with Vault and Nomad. We will run Vault in development mode using the following command.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

You can copy the root token from container logs using docker logs -f vault command. Then you have to login to the Vault web console available under address http://192.168.99.100:8200 using this token and enable RabbitMQ secret engine as shown below.

And confirm.

You can easily run Vault commands using a terminal provided by the web admin console or do the same thing using HTTP API. The first command visible below is used for writing connection details. We just need to pass RabbitMQ address and admin user credentials. The provided configuration settings points to #1 RabbitMQ node, but the changes are then replicated to the whole cluster.

$ vault write rabbitmq/config/connection connection_uri="http://192.168.99.100:30010" username="guest" password="guest"

The next step is to configure a role that maps a name in Vault to virtual host permissions.

$ vault write rabbitmq/roles/default vhosts='{"/":{"write": ".*", "read": ".*"}}'

We can test our newly created configuration by running command vault read rabbitmq/creds/default as shown below.

4. Sample application

Our sample application is pretty simple. It consists of two modules. First of them sender is responsible for sending messages to RabbitMQ, while second listener for receiving incoming messages. Both of them are Spring Boot applications that integrate with RabbitMQ and Vault using the following dependencies.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-amqp</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-vault-config-rabbitmq</artifactId>
   <version>2.0.2.RELEASE</version>
</dependency>

We need to provide some configuration settings in bootstrap.yml file to integrate our application with Vault. First, we need to enable plugin for that integration by setting property spring.cloud.vault.rabbitmq.enabled to true. Of course, Vault address and root token are required. It is also important to set property spring.cloud.vault.rabbitmq.role with the name of Vault role configured in step 3. Spring Cloud Vault injects username and password generated by Vault to the application properties spring.rabbitmq.username and spring.rabbitmq.password, so the only thing we need to configure in bootstrap.yml file is the list of available cluster nodes.

spring:
  rabbitmq:
    addresses: 192.168.99.100:30000,192.168.99.100:30001,192.168.99.100:30002
  cloud:
    vault:
      uri: http://192.168.99.100:8200
      token: s.7DaENeiqLmsU5ZhEybBCRJhp
      rabbitmq:
        enabled: true
        role: default
        backend: rabbitmq

For the test purposes you should enable high-available queues on RabbitMQ. For instructions how to configure them using policies you can refer to my article RabbitMQ in cluster. The application works at the level of exchanges. Auto-configured connection factory is injected into the application and set for RabbitTemplate bean.

@SpringBootApplication
public class Sender {
   
   private static final Logger LOGGER = LoggerFactory.getLogger("Sender");
   
   @Autowired
   RabbitTemplate template;

   public static void main(String[] args) {
      SpringApplication.run(Sender.class, args);
   }

   @PostConstruct
   public void send() {
      for (int i = 0; i < 1000; i++) {
         int id = new Random().nextInt(100000);
         template.convertAndSend(new Order(id, "TEST"+id, OrderType.values()[(id%2)]));
      }
      LOGGER.info("Sending completed.");
   }
    
    @Bean
    public RabbitTemplate template(ConnectionFactory connectionFactory) {
        RabbitTemplate rabbitTemplate = new RabbitTemplate(connectionFactory);
        rabbitTemplate.setExchange("ex.example");
        return rabbitTemplate;
    }
    
}

Our listener app is connected only to the third node of the cluster (spring.rabbitmq.addresses=192.168.99.100:30002). However, the test queue is mirrored between all clustered nodes, so it is able to receive messages sent by sender app. You can easily test using my sample applications.

@SpringBootApplication
@EnableRabbit
public class Listener {

   private static final  Logger LOGGER = LoggerFactory.getLogger("Listener");

   private Long timestamp;

   public static void main(String[] args) {
      SpringApplication.run(Listener.class, args);
   }

   @RabbitListener(queues = "q.example")
   public void onMessage(Order order) {
      if (timestamp == null)
         timestamp = System.currentTimeMillis();
      LOGGER.info((System.currentTimeMillis() - timestamp) + " : " + order.toString());
   }

   @Bean
   public SimpleRabbitListenerContainerFactory rabbitListenerContainerFactory(ConnectionFactory connectionFactory) {
      SimpleRabbitListenerContainerFactory factory = new SimpleRabbitListenerContainerFactory();
      factory.setConnectionFactory(connectionFactory);
      factory.setConcurrentConsumers(10);
      factory.setMaxConcurrentConsumers(20);
      return factory;
   }
   
}

The post RabbitMQ Cluster with Consul and Vault appeared first on Piotr's TechBlog.

Today I’m going to show you how to integrate your Spring Boot application with HashiCorp’s Vault in order to store your sensitive data properly. The first good news is that you don’t have to create any keys or certificates for encryption and decryption, because Vault will do it in your place. In this article in a few areas I’ll refer to my previous article about HashiCorp’s solutions Deploying Spring Cloud Microservices on HashiCorp’s Nomad. Now, as then, I also deploy my sample applications on Nomad to take advantage of built-in integration between those two very interesting HashiCorp’s tools. We will also use another HashiCorp’s solution for service discovery in inter-service communication – Consul. It’s also worth mentioning that Spring Cloud provides a dedicated project for integration with Vault – Spring Cloud Vault.

Architecture

The sample presented in this article will consist of two applications deployed on HashiCorp’s Nomad callme-service and caller-service. Microservice caller-service is the endpoint exposed by callme-service. An inter-service communication is performed using the name of the target application registered in Consul server. Microservice callme-service will store the history of all interactions triggered by caller-service in the database. The credentials to the database are stored on Vault. Nomad is integrated with Vault and stores the root token, which is not visible by the applications. The architecture of the described solution is visible on the following picture.

The current sample is pretty similar to the sample presented in my article Deploying Spring Cloud Microservices on Hashicorp’s Nomad. It is also available in the same repository on GitHub sample-nomad-java-service, but in the different branch vault. The current sample add an integration with PostgreSQL and Vault server for managing credentials to database.

1. Running Vault

We will run Vault inside the Docker container in a development mode. Server in development mode does not require any further setup, it is ready to use just after startup. It provides in-memory encrypted storage and unsecure (HTTP) connection, which is not a problem for a demo purposes. We can override default server IP address and listening port by setting environment property VAULT_DEV_LISTEN_ADDRESS, but we won’t do that. After startup our instance of Vault is available on port 8200. We can use the admin web console, which is for me available under address http://192.168.99.100:8200. The current version of Vault is 1.0.0.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

It is possible to login using different methods, but the most suitable way for us is through a token. To do that we have to display container logs using command docker logs vault, and then copy Root Token as shown below.

Now you can login to the Vault web console.

2. Integration with Postgres database

In Vault we can create Secret Engine that connects to other services and generates dynamic credentials on demand. Secrets engines are available under path. There is a dedicated engine for the various databases, for example PostgreSQL. Before activating such an engine we should run an instance of Postgres database. This time we will also use a Docker container. It is possible to set login and password to the database using environment variables.

$ docker run -d --name postgres -p 5432:5432 -e POSTGRES_PASSWORD=postgres123456 -e POSTGRES_USER=postgres postgres

After starting the database, we may proceed to the engine configuration in the Vault web console. First, let’s create our first secret engine. We may choose between some different types of engine. The right choice for now is Databases.

You can apply a new configuration to Vault using vault command or by HTTP API. Vault web console provides a terminal for running CLI commands, but it could be problematic in some cases. For example, I have a problem with escaping strings in some SQL commands, and therefore I had to add it using HTTP API. No matter which method you use, the next steps are the same. Following Vault documentation we first need to configure a plugin for PostgreSQL database and then provide connection settings and credentials.

$ vault write database/config/postgres plugin_name=postgresql-database-plugin allowed_roles="default" connection_url="postgresql://{{username}}:{{password}}@192.168.99.100:5432?sslmode=disable" username="postgres" password="postgres123456"

Alternatively, you can perform the same action using the HTTP API method. To authenticate against Vault we need to add header X-Vault-Token with root token. I have disabled SSL for connection with Postgres by setting sslmode=disable. There is only one role allowed to use this plugin: default. Now, let’s configure that role.

$ curl --header "X-Vault-Token: s.44GiacPqbV78fNbmoWK4mdYq" --request POST --data '{"plugin_name": "postgresql-database-plugin","allowed_roles": "default","connection_url": "postgresql://{{username}}:{{password}}@localhost:5432?sslmode=disable","username": "postgres","password": "postgres123456"}' http://192.168.99.100:8200/v1/database/config/postgres

The role can be created either with CLI or with HTTP API. The name of the role should be the same as the name passed in field allowed_roles in the previous step. We also have to set a target database name and SQL statement that creates users with privileges.

$ vault write database/roles/default db_name=postgres creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE,  SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" default_ttl="1h" max_ttl="24h"

Alternatively you can call the following HTTP API endpoint.

$ curl --header "X-Vault-Token: s.44GiacPqbV78fNbmoWK4mdYq" --request POST --data '{"db_name":"postgres", "creation_statements": ["CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE, SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";"]}' http://192.168.99.100:8200/v1/database/roles/default

And that’s all. Now, we can test our configuration using command with role’s name vault read database/creds/default as shown below. You can login to the database using returned credentials. By default, they are valid for one hour.

3. Enabling Spring Cloud Vault

We have succesfully configured a secret engine that is responsible for creating users on Postgres. Now, we can proceed to the development and integrate our application with Vault. Fortunately, there is a project Spring Cloud Vault, which provides out-of-the-box integration with Vault database secret engines. The only thing we have to do is to include Spring Cloud Vault to our project and provide some configuration settings. Let’s start from setting Spring Cloud Release Train. We use the newest stable version Finchley.SR2.

<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Finchley.SR2</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

We have to include two dependencies to our pom.xml. Starter spring-cloud-starter-vault-config is responsible for loading configuration from Vault and spring-cloud-vault-config-databases responsible for integration with secret engines for databases.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>

The sample application also connects to Postgres database, so we will include the following dependencies.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>org.postgresql</groupId>
   <artifactId>postgresql</artifactId>
   <version>42.2.5</version>
</dependency>

The only thing we have to do is to configure integration with Vault via Spring Cloud Vault. The following configuration settings should be placed in bootstrap.yml (no application.yml). Because we run our application on Nomad server, we use the port number dynamically set by Nomad available under environment property NOMAD_HOST_PORT_http and secret token from Vault available under environment property VAULT_TOKEN.

server:
  port: ${NOMAD_HOST_PORT_http:8091}

spring:
  application:
    name: callme-service
  cloud:
    vault:
      uri: http://192.168.99.100:8200
      token: ${VAULT_TOKEN}
      postgresql:
        enabled: true
        role: default
        backend: database
  datasource:
    url: jdbc:postgresql://192.168.99.100:5432/postgres

The important part of the configuration visible above is under the property spring.cloud.vault.postgresql. Following Spring Cloud documentation “Username and password are stored in spring.datasource.username and spring.datasource.password so using Spring Boot will pick up the generated credentials for your DataSource without further configuration”. Spring Cloud Vault is connecting with Vault, and then using role default (previously created on Vault) to generate new credentials to database. Those credentials are injected into spring.datasource properties. Then, the application is connected to the database using injected credentials. Everything works fine. Now, let’s try to run our applications on Nomad.

4. Deploying apps on Nomad

Before starting Nomad node we should also run Consul using its Docker container. Here’s a Docker command that starts a single node Consul instance.

$ docker run -d --name consul -p 8500:8500 consul

After that we can configure connection settings to Consul and Vault in Nomad configuration. I have created the file nomad.conf. Nomad is authenticating itself against Vault using root token. Connection with Consul is not secured. Sometimes it is also required to set network interface name and total CPU on the machine for the Nomad client. Most clients are able to determine it automatically, but it does not work for me.

client {
  network_interface = "Połączenie lokalne 4"
  cpu_total_compute = 10400
}

consul {
  address = "192.168.99.100:8500"
}

vault {
  enabled = true
  address = "http://192.168.99.100:8200"
  token = "s.6jhQ1WdcYrxpZmpa0RNd0LMw"
}

Let’s run Nomad in development mode passing configuration file location.

$ nomad agent -dev -config=nomad.conf

If everything works fine you should see the similar log on startup.

Once we have succesfully started Nomad agent integrated with Consul and Vault, we can proceed to the application deployment. First build the whole project with mvn clean install command. The next step is to prepare Nomad’s job descriptor file. For more details about the Nomad deployment process and its descriptor file you can refer to my previous article about it (mentioned in the preface of this article). Descriptor file is available inside application GitHub under path callme-service/job.nomad for callme-service, and caller-service/job.nomad for caller-service.

job "callme-service" {
   datacenters = ["dc1"]
   type = "service"
   group "callme" {
      count = 2
      task "api" {
         driver = "java"
         config {
            jar_path    = "C:\\Users\\minkowp\\git\\sample-nomad-java-services-idea\\callme-service\\target\\callme-service-1.0.0-SNAPSHOT.jar"
            jvm_options = ["-Xmx256m", "-Xms128m"]
         }
         resources {
            cpu    = 500 # MHz
            memory = 300 # MB
            network {
               port "http" {}
            }
         }
         service {
            name = "callme-service"
            port = "http"
         }
         vault {
            policies = ["nomad"]
         }
      }
      restart {
         attempts = 1
      }
   }
}

You will have to change the value of jar_path property with your path of application binaries. Before applying this deployment to Nomad we will have to add some additional configuration on Vault. When adding integration with Vault we have to pass the name of policies used for checking permissions. I set the policy with the name nomad, which now has to be created in Vault. Our application requires a permission for reading paths /secret/* and /database/* as shown below.

Finally, we can deploy our application callme-service on Nomad by executing the following command.

$ nomad job run job.nomad

The similar descriptor file is available for caller-service, so we can also deploy it. All the microservice has been registered in Consul as shown below.

Here is the list of registered instances of caller-service. As you can see on the picture below it is available under port 25816.

You can also take a look at the Nomad jobs view.

The post Secure Spring Cloud Microservices with Vault and Nomad appeared first on Piotr's TechBlog.