There is a similar article on my blog about “cert-manager” and Spring Boot SSL hot reload more focused on Kubernetes. You can also read the post on how to integrate Spring Boot with Vault PKI to generate certs automatically.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Let’s check out the openshift branch. Then, we need to go to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

$ git clone https://github.com/piomin/sample-spring-security-microservices.git
$ git checkout openshift
$ cd ssl

How It Works

OpenShift brings a lot of interesting features to the Kubernetes platform. The “Service Serving Certificates” is one of the examples. Thanks to that mechanism, we don’t need to install any external tool for generating and rotating certificates dynamically like “cert-manager”. The OpenShift CA service automatically rotates certificates 13 months before expiration and issues new certificates for 26 months. After a rotation, the previous CA is trusted until it expires. To achieve it, OpenShift implements the root CA key rollover process described in  https://tools.ietf.org/html/rfc4210#section-4.4. This allows a grace period for all affected services to refresh their key material before the expiration. Sounds interesting? Let’s proceed to the implementation.

In order to test a mechanism, we will run two simple Spring Boot apps on OpenShift that communicate using SSL certificates. OpenShift takes care of issuing/rotating certificates and exposing the CA bundle as a ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using Service Serving Certificates”. Here’s the illustration of our apps’ architecture on OpenShift.

Create a Server Side App

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now, our main goal is to enable HTTPS for that app and make it work properly with the OpenShift “Service Serving Certificates”. Firstly, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure keystores for the web server (3). We can configure SSL bundles using PEM-encoded text files directly in the app with the spring.ssl.bundle.pem properties group. The certificate (tls.crt) and private key (tls.key) will be mounted as a volume to the pod. Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property (2). Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl.bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

The only thing we need to do to generate SSL certificates for our app is to annotate the Kubernetes Service with service.beta.openshift.io/serving-cert-secret-name. It should contain the name of the target Kubernetes Secret as a value.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Once we apply such a Service to the cluster, OpenShift generates the secure-callme-bundle Secret that contains the certificate and private key.

After that, we just need to deploy the app and mount the generated Secret as a volume. The volume path is passed to the app as the CERT_PATH environment variable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-bundle

Finally, we need to expose the HTTP endpoint outside of the cluster. We can easily do it with the OpenShift Route. Don’t forget the set the TLS termination as passthrough. Thanks to that OpenShift router sends traffic directly to the destination without TLS termination.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-callme-bundle
  labels:
    app: secure-callme-bundle
    app.kubernetes.io/component: secure-callme-bundle
    app.kubernetes.io/instance: secure-callme-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-callme-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Create a Client Side App

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the truststore taken from ConfigMap with CA certificates. With the certs inside the service-ca.crt file, the RestTemplate bean can authenticate against the secure-callme-bundle app.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key
  client:
    truststore:
      certificate: ${CLIENT_CERT_PATH}/service-ca.crt

Don’t forget about creating the RestTemplate bean. It needs to the SslBundles bean and set it on the RestTemplateBuilder.

@Bean
RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
   return builder
      .setSslBundle(sslBundles.getBundle("client")).build();
}

We need to create the ConfigMap annotated with service.beta.openshift.io/inject-cabundle=true. After that, the OpenShift cluster automatically injects the service CA certificate into the service-ca.crt key on the config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-caller-bundle
  namespace: demo-secure
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
data: {}

Let’s display the details of the secure-caller-bundle ConfigMap. As you see, OpenShift injects the PEM-encoded CA certificates under the service-ca.crt key to that config map.

Finally, we can deploy our client-side app. Here’s the YAML manifest with the Deployment object. Since the secure-caller-bundle app acts as a web server and also uses a REST client to invoke the secure-callme-bundle we need to mount two volumes. The first one (/opt/secret) contains a certificate and private key for the web server (1). Consequently, the second one (/opt/client-cert) holds the CA bundle used by the RestTemplate client (2). This client communicates with secure-callme-bundle internally through the Kubernetes Service (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-cert
            # (3)
            - name: HOST
              value: secure-callme-bundle.demo-secure.svc
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            # (1)
            - mountPath: /opt/secret
              name: cert
            # (2)
            - mountPath: /opt/client-cert
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-bundle
        - name: client-cert
          configMap:
            name: secure-caller-bundle

Of course, we should also create a passthrough Route to expose our app the outside of the cluster:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-caller-bundle
  labels:
    app: secure-caller-bundle
    app.kubernetes.io/component: secure-caller-bundle
    app.kubernetes.io/instance: secure-caller-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-caller-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Testing OpenShift Certs SSL Rotation with Spring Boot

The main problem with testing the certs rotation scenario is in the 13-month interval. We can’t change that period in the configuration. However, there is a way how we can force OpenShift to rotate the CA certs on demand. In order to do that, we need to set the spec.unsupportedConfigOverrides.forceRotation.reason property on the cluster-wide ServiceCA object with any value. Of course, you shouldn’t do that at production, since it is not supported by OpenShift. It is just designed for testing or development purposes.

apiVersion: operator.openshift.io/v1
kind: ServiceCA
metadata:
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  unsupportedConfigOverrides:
    forceRotation:
      reason: testing

We will call our apps through the OpenShift Route. So, let’s display a current list of routes related to our sample apps:

$ oc get route
NAME                   HOST/PORT                                                     PATH   SERVICES               PORT    TERMINATION   WILDCARD
secure-caller-bundle   secure-caller-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-caller-bundle   https   passthrough   None
secure-callme-bundle   secure-callme-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-callme-bundle   https   passthrough   None

Once we set that property, OpenShift immediately rotates that CA certificate. It will also take care of synchronizing the latest certificates with all the secrets and config maps containing annotations required by the “Service Serving Certificates” mechanism. As you see, now, our secure-caller-bundle ConfigMap contains two CAs: an old one and a new one.

$ oc describe cm secure-caller-bundle
Name:         secure-caller-bundle
Namespace:    demo-secure
Labels:       <none>
Annotations:  service.beta.openshift.io/inject-cabundle: true

Data
====
service-ca.crt:
----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIITnr+EqYsYtEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAzMDExMDM5MTVaFw0yNjA0MzAxMDM5MTZaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDN0No0DIZ+BCdPvQ9TIvEjwGUSoUXM+DYI
Dxlk2Ef6goFv3YsAgNJTxznntRc69dJTrLtRrmFN0d31JPM/gJOMkKMvTWnz8Qzx
athuvvwbivJNKh/qn+Dhewbjrx5LGpq3v9VQ7t/5Rf5F9VMpyp738EwfAy2cfxTA
sxgYpvU/foivM9U0uMaZyXA5vLsGN6cKpEREGMqKbIKSXNYXD9/lbHv2eyr+5+s9
keYEDkGTWMceISihm5mGNwOZpLZhNwFgSZR9O63TIgGc/TEZ3EYwcIjmlVOSwQ/u
sM43HL1jtYKw3AovQOc5qc2kT5eSPV7jVCCfNYnj3BLvt0AYUXE/AgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT3iI1f
71ZVt6OVLC/+0oxNL7BqhzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEATcT/LK5pAaOGz0j3MR4NhsfmAjo4K8SoGmFE
F4lDzDWZTaJuJHCHqpblgGKLfokU9BQC3L4NZLIisOj3CWF3zIpC6MOsU7x9qDm0
aZpI1gqhRXQZOqVpE9XJ0XBEOLP9hlzxa66SQlALlU5CWbDNEMtXsMMQf1lSgIRz
arWK8QwmPsmg6duoVtFcCMY2klkVacZctiMgM4wvf1CCJt3TkiggwO3t4IdiV8Xa
Z9d/LlV/bFzboxGxOb2IMHj5VXKPM6HmC4slBSqKeyr7PYyM7rjhorIOOner9JrX
RhbC8/uARciLbcFKyOOCSLoe5m1i8QPrRmCm/GhVS9M4x86TXw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIFlMUWK4HjW4wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAyMjgxMTUyMjNaFw0yNjA0MjgxMTUyMjRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY6AZbLN0kT2BPXbDn7iZmbyYZgyI8Y3jh
hRq9JrX0ZIIs5BvFcRRkm2IcJ8gsMF4VztGCkNoDQ07ojAIHI/FxssAU8wwoz2QP
eH2qzBpLA1lBAYvtjki//55STJvJF+7Z6qbcDwUVX6/r+hrpy5MIjaQmRV4DRLpp
ZD3HrqYozJEvCKaA2pinC2VyW8IBDB0FDvPdBxvzjrCDjssr/v0jqFgUGFJJmKEj
EaFQACeMlOS4Q5avEglfwoLS+RGgPjE1s1gpxir2dZCdnM8b9B3CFy2sS+SSHN4y
+FJRJBv4CbMAMP+DLIsHY872c2XmOlleJNFpiFWi9r33LCc7MnbjAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRey/oP
FWW5kG8ifSXqIXBh2L+FdzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEAhbimn2Tv5w9Cy7i6dNdYYdVhA52cCnWWDDhf
s/dZ+Zl1aP3f179XO0xz3ovLuE227Z9AYPGtPW6c0FQgxgDgl2pYUX4J88caTUG/
eR9EG9LEfrL6OlKMjMvAFWrL64bcAAiuoz/gWB0a29xMnUsA52+7OpnzmHMduaaz
5JfRaTwkjGGe8ay2XsTUNrNGqZJGKHhE+hrUVlFsS7GpaSEe9GllkS9lGUZp2Hei
lTxwnBr9j+aP/QWXwuAVxT1DUAByJYoTPrpMhk+hKQUj/8TvxmMmsksZ57steqRI
/gWYd4j2SohN3jd1mv6rQ+jsGGAfCNmYVZL1oevNaqfnENt0IA==
-----END CERTIFICATE-----

However, if we don’t restart the pods, our apps still use the old keystore and truststore (unless they have the reload-on-update option enabled).

Here’s a list of running pods.

oc get po
NAME                                    READY   STATUS    RESTARTS   AGE
secure-caller-bundle-57c7658d85-bj4q5   1/1     Running   0          2d23h
secure-callme-bundle-cbf59569f-n6gg5    1/1     Running   0          2d23h

Let’s just restart the secure-callme-bundle app by deleting its pod. Do not restart the secure-callme-bundle pod, because we want to test the scenario where it uses the old truststore.

$ oc delete po secure-callme-bundle-cbf59569f-n6gg5
pod "secure-callme-bundle-cbf59569f-n6gg5" deleted

Then, we can call the GET /callme endpoint exposed by the secure-callme-bundle app through the Route once again. As you see, it servers the latest, reloaded certificate.

Let’s perform a final test. Now, we call the GET /caller endpoint exposed by the secure-caller-bundle app. Under the hood, it uses RestTemplate to call the GET /callme endpoint exposed by the secure-callme-bundle app via secure-callme-bundle.demo-secure.svc address. As you see the communication still works fine although the secure-caller-bundle app still uses the keystore and truststore.

Final Thoughts

The OpenShift “Service Serving Certificates” mechanism is simple to configure and doesn’t require that you handle the process of reloading apps with the rotated trusted material. It also has some drawbacks. You cannot change the rotation period from 13 months into a different value, and there is a global CA bundle shared across all the services running on the cluster. If those things are problematic for you I advise you to use “cert-manager” and a tool that allows you to restart pods automatically on the Secret or ConfigMap change. It is also possible to leverage the Spring SslBundles to reload certs without restarting the pod. With Spring Boot apps you can also use OpenShift Service Mesh, which can handle SSL communication at the platform level.

The post Rotate SSL Certificates with OpenShift and Spring Boot appeared first on Piotr's TechBlog.

In order to generate SSL certificates on Kubernetes we will use cert-manager. “Cert-manager” can rotate certificates after a specified period and save them as Kubernetes Secrets. I have already described how to implement a similar scenario with an automatic restart of a pod on a secret update in the article here. We were using the Stakater Reloader tool to restart the pod automatically on a new version of Secret. However, this time we use Spring Boot features to avoid having to restart an app (pod).

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

How It Works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need to design a solution for enabling SSL/TLS communication between the services running on Kubernetes. This solution must take into account a scenario of certificates reloading. Moreover, it must happen at the same time for the both server and client sides to avoid errors in the communication. On the server side, we use an embedded Tomcat server. In the client-side app, we use the Spring RestTemplate object.

“Cert-manager” can generate certificates automatically, based on the provided CRD object. It ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It serves all the required staff as the Kubernetes Secret. Such a secret is then mounted as a volume into the app pod. Thanks to that we don’t need to restart a pod, to see the latest certificates or “keystores” inside the pod. Here is the visualization of the described architecture.

Install cert-manager on Kubernetes

In order to install both “cert-manager” on Kubernetes we will use its Helm chart. We don’t need any specific settings. Before installing the chart we have to add CRD resources for the latest version 1.14.2:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart in the cert-manager namespace using the following command:

$ helm install my-release cert-manager jetstack/cert-manager \
    -n cert-manager

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0 

That’s all. Now we can proceed to the implementation.

Spring Boot SSL Hot Reload on the Embedded Server

Sample App Implementation

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our main goal is to enable HTTPS for that app and make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure SSL trust material for the web server (3). There are two types of trusted material it can support. In order to configure bundles using Java keystore files, we have to use the spring.ssl.bundle.jks group. On the other hand, it is possible to configure bundles using PEM-encoded text files with the spring.ssl.bundle.pem properties group.

In the exercise, we will use the Java keystore files (JKS). We define a single SSL bundle under the server name. It contains both keystore and truststore locations. With the reload-on-update property, we can instruct Spring Boot to watch the files in the background and trigger a web server reload if they change. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property (2). Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property. Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl:
  client-auth: NEED
  bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Generate Certificates with Cert-manager

Before we deploy the callme-secure-bundle app on Kubernetes, we need to configure “cert-manager” and generate the required certificates. Firstly, we need to define the CRD object responsible for issuing certificates. Here’s the ClusterIssuer object that generates self-signed certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-cluster-issuer
spec:
  selfSigned: {}

Here’s the Kubernetes Secret with the password used for securing generated keystores:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

After that, we can generate certificates. Here’s the Certificate object for the app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer, which has been created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme-bundle, so the cert needs to have that name as CN. In order to enable certificate rotation, we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration “cert-manager” will automatically renew a certificate (5). However, it won’t rotate the private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme-bundle
    - localhost
  secretName: secure-callme-cert
  commonName: secure-callme-bundle
  duration: 1h
  renewBefore: 5m

Deploy on Kubernetes

After creating a certificate we can proceed to the secure-callme-bundle app deployment. It mounts the Secret containing certificates and keystores as a volume. The name of the output Secret is determined by the value of the spec.secretName defined in the Certificate object. We need to inject some environment variables into the Spring Boot app. It requires the password to the keystores (PASSWORD), the location of the mounted trusted material inside the pod (CERT_PATH), and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert

Here’s the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Firstly, make sure you are inside the secure-callme-bundle directory. Let’s build and run the app on Kubernetes with Skaffold and enable “port-forwarding” under 8443 port:

$ skaffold dev --port-forward

Skaffold will not only run the app but also apply all required Kubernetes objects defined in the app k8s directory. It applies also to the “cert-manager” Certificate object. Once the skaffold dev command finishes successfully, we access our HTTP endpoint under the http://127.0.0.1:8443 address.

Let’s call the GET /callme endpoint. Although, we enabled the --insecure option the request failed since the web server requires client authentication. To avoid it, we should include both key and certificate files in the curl command. However,

$ curl https://localhost:8443/callme --insecure -v
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-callme-bundle
*  start date: Feb 18 20:13:00 2024 GMT
*  expire date: Feb 18 21:13:00 2024 GMT
*  issuer: CN=secure-callme-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /callme HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
* LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
* Closing connection
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0

Spring Boot SSL Hot Reload with RestTemplate

Sample App Implementation

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the keystore and truststore taken from Secret generated for the server-side app. With those files, the RestTemplate bean can authenticate against the secure-callme-bundle app. Of course, we also need to automatically reload the SslBundle bean after a certificate rotation.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
  client:
    reload-on-update: true
    keystore:
      location: ${CLIENT_CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CLIENT_CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Spring Boot 3.1 with the bundles’ concept extremely simplifies SSL context configuration for Spring REST clients like RestTemplate or WebClient. However, currently (Spring Boot 3.2.2) there is no built-in implementation for reloading e.g. Spring RestTemplate on the SslBundle update. Therefore we need to add a portion of code to achieve that. Fortunately, SslBundles allows us to define a custom handler that fires on the bundle update event. We need to define the handler for the client bundle. Once it receives a rotated version of SslBundle, it replaces the existing RestTemplate bean in the context with a new one using RestTemplateBuilder.

@SpringBootApplication
public class SecureCallerBundle {

   private static final Logger LOG = LoggerFactory
      .getLogger(SecureCallerBundle.class);

   public static void main(String[] args) {
      SpringApplication.run(SecureCallerBundle.class, args);
   }

   @Autowired
   ApplicationContext context;

   @Bean("restTemplate")
   RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
      sslBundles.addBundleUpdateHandler("client", sslBundle -> {
         try {
            LOG.info("Bundle updated: " + sslBundle.getStores().getKeyStore().getCertificate("certificate"));
         } catch (KeyStoreException e) {
            LOG.error("Error on getting certificate", e);
         }
         DefaultSingletonBeanRegistry registry = (DefaultSingletonBeanRegistry) context
            .getAutowireCapableBeanFactory();
         registry.destroySingleton("restTemplate");
         registry.registerSingleton("restTemplate", 
            builder.setSslBundle(sslBundle).build());
      });
      return builder.setSslBundle(sslBundles.getBundle("client")).build();
   }
}

Deploy on Kubernetes

Let’s take a look at the Kubernetes Deployment manifest for the current app. This time, we are mounting two secrets as volumes. The first one is generated for the current app web server, while the second one is generated for the secure-callme-bundle app and is used by the RestTemplate in establishing secure communication. We also set the address of the target service to inject it into the app (HOST) and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme-bundle
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Let’s deploy the app with the skaffold dev --port-forward command. Once again, it will deploy all the required staff on Kubernetes. Since we already exposed the secure-callme-bundle app with the “port-forward” option, the current app is exposed under the 8444 port.

Let’s try to call the GET /caller endpoint. Under the hood, it calls the endpoint exposed by the secure-callme-bundle app with RestTemplate. As you see, the secure communication is successfully established.

curl https://localhost:8444/caller --insecure -v
*   Trying [::1]:8444...
* Connected to localhost (::1) port 8444
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-caller-bundle
*  start date: Feb 18 20:40:11 2024 GMT
*  expire date: Feb 18 21:40:11 2024 GMT
*  issuer: CN=secure-caller-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /caller HTTP/1.1
> Host: localhost:8444
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 57
< Date: Sun, 18 Feb 2024 21:26:42 GMT
<
* Connection #0 to host localhost left intact
I'm `secure-caller`! calling... I'm secure-callme-bundle!

Now, we can wait one hour until the “cert-manager” rotates the certificate inside the secure-callme-cert Secret. However, we can also remove the secret, since “cert-manager” will regenerate it based on the Certificate object. Here’s the secret with certificates and keystores used to establish secure communication between both our sample Spring Boot apps.

No matter if you wait until the 1h rotation occurs or do it manually by removing the secret, you should see the following log inside the secure-callme-bundle app pod. It means that Spring Boot has received the SslBundle update event and then reloaded a Tomcat server.

The SslBundle event is also handled on the secure-caller-bundle app side. It refreshes the RestTemplate bean and prints information with the latest certificate in the logs.

Final Thoughts

The latest releases of Spring Boot simplify the management of SSL certificates on the both server and client sides a lot. Thanks to SslBundles we can easily handle the certificate rotation process without restarting the pod on Kubernetes. There are still some other things to consider don’t covered by this article. It includes the mechanism of distributing the trust bundles across the apps. However, for example, to manage trust bundles in the Kubernetes environment we can use the “cert-manager” trust-manager feature.

The post Spring Boot SSL Hot Reload on Kubernetes appeared first on Piotr's TechBlog.

Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme and secure-caller. After that, you should just follow my instructions. Let’s begin.

How it works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.

Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap or Secret. Then it performs a rolling upgrade on pods, which use the particular ConfigMap or Secret. Here is the visualization of the described architecture.

Prerequisites

Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:

1. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
2. Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager

Install Cert Manager and Stakater Reloader

In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart using the following command:

$ helm install my-release cert-manager jetstack/cert-manager

The same with the Stakater Reloader – first we need to add the stakater charts repository:

$ helm repo add stakater https://stakater.github.io/stakater-charts

Then, we can install the latest version of the chart:

$ helm install my-reloader stakater/reloader

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0          1m
my-reloader-reloader-7566fdc68c-qj9l4         1/1     Running   0          1m

That’s all. Now we can proceed to the implementation.

HTTPS with Spring Boot

Our first app secure-callme exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property. Here’s the configuration for our Spring Boot inside the application.yml file.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

We will set the values of CERT_PATH and PASSWORD at the level of Kubernetes Deployment. Now, let’s switch to the secure-caller implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient dependency.

<dependency>
  <groupId>org.apache.httpcomponents.client5</groupId>
  <artifactId>httpclient5</artifactId>
</dependency>

Now, we will use Apache HttpClient as a low-level client for the Spring RestTemplate. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate @Bean we use RestTemplateBuilder.

@SpringBootApplication
public class SecureCaller {

   public static void main(String[] args) {
      SpringApplication.run(SecureCaller.class, args);
   }

   @Autowired
   ClientSSLProperties clientSSLProperties;

   @Bean
   RestTemplate builder(RestTemplateBuilder builder) throws 
      GeneralSecurityException, IOException {

      final SSLContext sslContext = new SSLContextBuilder()
         .loadTrustMaterial(
            new File(clientSSLProperties.getTrustStore()),
            clientSSLProperties.getTrustStorePassword().toCharArray())
         .loadKeyMaterial(
            new File(clientSSLProperties.getKeyStore()),
            clientSSLProperties.getKeyStorePassword().toCharArray(),
            clientSSLProperties.getKeyStorePassword().toCharArray()
         )
         .build();

      final SSLConnectionSocketFactory sslSocketFactory = 
         SSLConnectionSocketFactoryBuilder.create()
                .setSslContext(sslContext)
                .build();

      final HttpClientConnectionManager cm = 
         PoolingHttpClientConnectionManagerBuilder.create()
                .setSSLSocketFactory(sslSocketFactory)
                .build();

      final HttpClient httpClient = HttpClients.custom()
         .setConnectionManager(cm)
         .evictExpiredConnections()
         .build();

      return builder
         .requestFactory(() -> 
            new HttpComponentsClientHttpRequestFactory(httpClient))
         .build();
   }
}

The client credentials are taken from configuration settings under the client.ssl key. Here is the @ConfigurationProperties class used the RestTemplateBuilder in the previous step.

@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {

   private String keyStore;
   private String keyStorePassword;
   private String trustStore;
   private String trustStorePassword;

   // GETTERS AND SETTERS ...

}

Here’s the configuration for the secure-caller inside application.yml file. The same as for the secure-callme we expose the REST endpoint over HTTPS.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

client.url: https://${HOST}:8443/callme
client.ssl:
  key-store: ${CLIENT_CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CLIENT_CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}

The secure-caller app calls GET /callme exposed by the secure-callme app using customized RestTemplate.

@RestController
public class SecureCallerController {

   RestTemplate restTemplate;

   @Value("${client.url}")
   String clientUrl;

   public SecureCallerController(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @GetMapping("/caller")
   public String call() {
      return "I'm `secure-caller`! calling... " +
             restTemplate.getForObject(clientUrl, String.class);
   }

}

Generate and Renew Certificates on Kubernetes with Cert Manager

With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-clusterissuer
spec:
  selfSigned: {}

After that, we can generate certificates. Here’s the cert-manager Certificate object for the secure-callme app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy to Always.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores: # (1)
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef: # (2)
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme
  secretName: secure-callme-cert
  commonName: secure-callme # (3)
  duration: 1h # (4)
  renewBefore: 5m  # (5)

The Certificate object for secure-caller is very similar. The only difference is in the CN field. We will use the port-forward option during the test, so I’ll set the domain name to localhost (1).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-caller-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - localhost
    - secure-caller
  secretName: secure-caller-cert
  commonName: localhost # (1)
  duration: 1h
  renewBefore: 5m

After applying both manifests we can display a list of Certificates. Each of them is related to the Secret with the same name:

$ kubectl get certificate
NAME                 READY   SECRET               AGE
secure-caller-cert   True    secure-caller-cert   1m
secure-callme-cert   True    secure-callme-cert   1m

Here are the details of the secure-callme-cert Secret. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store and server.ssl.key-store properties). There is also a certificate (tls.crt), a private key (tls.key), and CA (ca.crt).

$ kubectl describe secret secure-callme-cert
Name:         secure-callme-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: secure-callme
              cert-manager.io/certificate-name: secure-callme-cert
              cert-manager.io/common-name: secure-callme
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: ss-clusterissuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:          550 bytes
keystore.jks:    1029 bytes
tls.crt:         550 bytes
tls.key:         227 bytes
truststore.jks:  422 bytes

Deploy and Reload Apps on Kubernetes

Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme app.

First of all, we need to reload the app each time the secure-callme-cert changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload (1). The annotation should contain the name of Secret, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH env variable (2). We are mounting the whole secure-callme-cert Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme
  annotations:
    # (1)
    secret.reloader.stakater.com/reload: "secure-callme-cert" 
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme
    spec:
      containers:
        - image: piomin/secure-callme
          name: secure-callme
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH # (2)
              value: /opt/secret
          volumeMounts:
            - mountPath: /opt/secret # (3)
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert # (4)

The password to the key store files is available inside the jks-password-secret:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

There is also the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme
  name: secure-callme
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme
  type: ClusterIP

Now, go to the secure-callme directory and just run the following command:

$ skaffold run

The Deployment manifest of the secure-caller app is a little bit more complicated. The same as before we need to reload the app on Secret change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller
  # (1)
  annotations:
    secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller
    spec:
      containers:
        - image: piomin/secure-caller
          name: secure-caller
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            # (2)
            - name: CERT_PATH
              value: /opt/secret
            # (3)
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        # (5)
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Then, go to the secure-caller directory and deploy the app. This time we enable port-forward to easily test the app locally.

$ skaffold run --port-forward

Let’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.

$ kubectl get deploy 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
my-cert-manager              1/1     1            1           1h
my-cert-manager-cainjector   1/1     1            1           1h
my-cert-manager-webhook      1/1     1            1           1h
my-reloader-reloader         1/1     1            1           1h
secure-caller                1/1     1            1           1h
secure-callme                1/1     1            1           1h

Testing Renew of Certificates on Kubernetes

The secure-caller app is available on the 8443 port locally, while the secure-callme app is available inside the cluster under the service name secure-callme. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.key}' | base64 --decode > tls.key

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.crt}' | base64 --decode > tls.crt

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.ca\.crt}' | base64 --decode > ca.crt

Now, we can call the GET /caller endpoint using the following curl command. Under the hood, the secure-caller calls the endpoint GET /callme exposed by the secure-callme also over HTTPS. If you did everything according to the instruction you should have the same result as below.

$ curl https://localhost:8443/caller \
  --key tls.key \
  --cert tls.crt \
  --cacert ca.crt 
I'm `secure-caller`! calling... I'm `secure-callme`!

Our certificate is valid for one hour.

Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.

Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.

Final Thoughts

Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.

The post Renew Certificates on Kubernetes with Cert Manager and Reloader appeared first on Piotr's TechBlog.

There are several topics you should be familiar with before start reading the following article. At least, it is worth reading about the basics related to service mesh on Kubernetes with Istio and Spring Boot here. You may also be interested in the article about security best practices for Spring Boot apps.

Prerequisites

Before we begin we need to prepare a test cluster. Personally, I’m using a local OpenShift cluster that simplifies the installation of several tools we need. But you can as well use any other Kubernetes distribution like minikube or kind. No matter which version you choose, you need to install at least:

1. Istio – you can do it in several ways, here’s an instruction with Istio CLI
2. Cert Manager – the simplest way to install it with the kubectl
3. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
4. Helm – used to install additional tools on Kubernetes (or OpenShift)

Since I’m using OpenShift I can install everything using the UI console and Operator Hub. Here’s the list of my tools:

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the tls branch. After that, you should just follow my instructions. Let’s begin.

Generate Certificates with Cert Manager

You can as well generate TLS/SSL certs using e.g. openssl and then apply them on Kubernetes. However, Cert Manager simplifies that process. It automatically creates a Kubernetes Secret with all the required staff. Before we create a certificate we should configure a default ClusterIssuer. I’m using the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

In the next step, we are going to create the Certificate object. It is important to set the commonName parameter properly. It should be the same as the hostname of our gateway. Since I’m running the local instance of OpenShift the default domain suffix is apps-crc.testing. The name of the application is sample-spring-kotlin in that case. We will also need to generate a keystore and truststore for configuring 2-way SSL in the second scenario with the passthrough gateway. Finally, we should have the same Kubernetes Secret with a certificate placed in the namespace with the app (demo-apps) and the namespace with Istio workloads (istio-system in our case). We can sync secrets across namespaces in several different ways. I’ll use the built-in feature of Cert Manager based on the secretTemplate field and a tool called reflector. Here’s the final version of the Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sample-spring-boot-cert
  namespace: demo-apps
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: selfsigned-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - sample-spring-kotlin.apps-crc.testing
  secretName: sample-spring-boot-cert
  commonName: sample-spring-kotlin.apps-crc.testing
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"  
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "istio-system"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "istio-system"

Before we apply the Certificate object we need to create the Secret with a keystore password:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
  namespace: demo-apps
data:
  password: MTIzNDU2
type: Opaque

Of course, we also need to install the reflector tool on our Kubernetes cluster. We can easily do it using the following Helm commands:

$ helm repo add emberstack https://emberstack.github.io/helm-charts
$ helm repo update
$ helm upgrade --install reflector emberstack/reflector

Here’s the final result. There is the Secret sample-spring-kotlin-cert in the demo-apps namespace. It contains a TLS certificate, private key, CA, JKS keystore, and truststore. You can also verify that the same Secret is available in the istio-system namespace.

Create Istio Gateway with Mutual TLS

Let’s begin with our first scenario. In order to create a gateway with mTLS, we should set MUTUAL as a mode and set the name of the Secret containing the certificate and private key. The name of the Istio Gateway host is sample-spring-kotlin.apps-crc.testing. Gateway is available on Kubernetes under the default HTTPS port.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: MUTUAL
        credentialName: sample-spring-boot-cert

The Spring Boot application is available under the HTTP port. Therefore we should create a standard Istio VirtualService that refers to the already created gateway.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  http:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8080

Finally, we can run the Spring Boot app on Kubernetes using skaffold. We need to activate the istio-mutual profile that exposes the HTTP (8080) port.

$ skaffold dev -p istio-mutual

OpenShift Service Mesh automatically exposes Istio Gateway as the Route object. Let’s just verify if everything works fine. Assuming that the app has started successfully we can display a list of Istio gateways in the demo-apps namespace:

$ kubectl get gw -n demo-apps
NAME                    AGE
microservices-gateway   3m4s

Then let’s display a list of Istio virtual services in the same namespace:

$ kubectl get vs -n demo-apps
NAME                           GATEWAYS                    HOSTS                                       AGE
sample-spring-boot-vs-via-gw   ["microservices-gateway"]   ["sample-spring-kotlin.apps-crc.testing"]   4m2s

If you are running it on OpenShift you should also check the Route object in the istio-system namespace:

$ oc get route -n istio-system | grep sample-spring-kotlin

If you test it on Kubernetes you just need to set the Host header on your request. Here’s the curl command for testing our secure gateway. Since we enabled mutual TLS auth we need to provide the client key and certificate. We can copy them to the local machine from Kubernetes Secret generated by the Cert Manager.

Let’s call the REST endpoint GET /persons exposed by our sample Spring Boot app:

$ curl -v https://sample-spring-kotlin.apps-crc.testing/persons \
    --key tls.key \
    --cert tls.crt

You will probably receive the following response:

Ok, we forgot to add our CA to the trusted certificates. Let’s do that. Alternatively, we can set the parameter --cacert on the curl command.

Now, we can run again the same curl command as before. The secure communication with our app should work perfectly fine. We can proceed to the second scenario.

Mutual Auth for Spring Boot and Passthrough Istio Gateway

Let’s proceed to the second scenario. Now, our sample Spring Boot is exposing the HTTPS port with the client cert verification. In order to enable it on the app side, we need to provide some configuration settings.

Here’s our application.yml file. Firstly, we need to enable SSL and set the default port to 8443. It is important to force client certificate authentication with the server.ssl.client-auth property. As a result, we also need to provide a truststore file location. Finally, we don’t want to expose Spring Boot Actuator endpoint over SSL, so we force to expose them under the default plain port.

server.port: 8443
server.ssl.enabled: true
server.ssl.key-store: /opt/secret/keystore.jks
server.ssl.key-store-password: ${PASSWORD}

server.ssl.trust-store: /opt/secret/truststore.jks
server.ssl.trust-store-password: ${PASSWORD}
server.ssl.client-auth: NEED

management.server.port: 8080
management.server.ssl.enabled: false

In the next step, we need to inject both keystore.jks and truststore.jks into the app container. Therefore we need to modify Deployment to mount a secret generated by the Cert Manager as a volume. Once again, the name of that Secret is sample-spring-boot-cert. The content of that Secret would be available for the app under the /opt/secret directory. Of course, we also need to expose the port 8443 outside and inject a secret with the keystone and truststore password.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin-microservice
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: sample-spring-kotlin-microservice
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app.kubernetes.io/name: sample-spring-kotlin-microservice
    spec:
      containers:
      - image: piomin/sample-spring-kotlin
        name: sample-spring-kotlin-microservice
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8443
          name: https
        env:
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: jks-password-secret
        volumeMounts:
          - mountPath: /opt/secret
            name: sample-spring-boot-cert
      volumes:
        - name: sample-spring-boot-cert
          secret:
            secretName: sample-spring-boot-cert

Here’s the definition of our Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: PASSTHROUGH

The definition of the Istio VirtualService is slightly different than before. It contains the TLS route with the name of the host to match SNI.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  tls:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8443
          weight: 100
      match:
        - port: 443
          sniHosts:
            - sample-spring-kotlin.apps-crc.testing

Finally, we can deploy the current version of the app using the following skaffold command to enable HTTPS port for the app running on Kubernetes:

$ skaffold dev -p istio-passthrough

Now, you can repeat the same steps as in the previous section to verify that the current configuration works fine.

Final Thoughts

In this article, I showed how to use some interesting tools that simplify the configuration of HTTPS for apps running on Kubernetes. You could see how Istio, Cert Manager, or reflector can work together. We considered two variants of making secure HTTPS Istio gateways for the Spring Boot application.

The post HTTPS on Kubernetes with Spring Boot, Istio and Cert Manager appeared first on Piotr's TechBlog.

Before we start with a list of security “golden rules”, let’s analyze a typical microservices architecture. We will focus on components important for building a secure solution.

The picture visible below illustrates a typical microservices architecture built with Spring Cloud. There is an API gateway built on top of Spring Cloud Gateway. Since it is an entry point to our system, we will enable some important security mechanisms on it. There are several microservices hidden behind the gateway. There is also a discovery server, which allows localizing IP addresses using the name of services. And finally, there are some components that do not take part in communication directly. It is just a proposition of a few selected tools. You may choose other solutions providing the same features. Vault is a tool for securely storing and accessing secrets. Keycloak is an open-source identity and access management solution. Spring Cloud Config Server provides an HTTP API for external configuration. It may integrate with several third-party tools including Vault.

Let’s begin. Here’s our list of Spring Security best practices.

1. Enable rate limiting on the API gateway

An API gateway is an important pattern in microservice-based architecture. It acts as a single entry point into the whole system. It is responsible not only for request routing but also for several other things including security. Consequently, one of the most essential components of security we should enable on the gateway is rate limiting. It protects your API against DoS attacks, which can tank a server with unlimited HTTP requests.

Spring provides its own implementation of the API gateway pattern called Spring Cloud Gateway. On the other hand, Spring Cloud Gateway comes with a built-in implementation of a rate-limiting component. To sum up, you just need to include a single dependency to build a gateway application. Then you have to provide some configuration settings to enable rate limiting for a single route.

In order to enable a rate limiter on a gateway, we need to use a component called RequestRateLimiter GatewayFilter factory. It uses a RateLimiter implementation to determine if the current request is allowed to proceed. Otherwise, the gateway returns a status of HTTP 429 - Too Many Requests. The RequestRateLimiter implementation uses Redis as a backend.

Let’s take a look at a typical configuration of routes handled by Spring Cloud Gateway. There are three parameters that can be used to configure the rate limiter: replenishRate, burstCapacity, and requestedTokens. The replenishRate property is how many requests per second a single user may send. The burstCapacity property is the maximum number of requests a user can send in a single second. With the requestedTokens property, we may set the cost of a single token.

spring:
  cloud:
    gateway:
      routes:
        - id: account-service
          uri: http://localhost:8090
          predicates:
            - Path=/account/**
          filters:
            - RewritePath=/account/(?<path>.*), /$\{path}
            - name: RequestRateLimiter
              args:
                redis-rate-limiter.replenishRate: 1
                redis-rate-limiter.burstCapacity: 60
                redis-rate-limiter.requestedTokens: 15

Then we should define a KeyResolver bean. A rate limiter defines all parameters per a single key returned by the resolver.

@Bean
KeyResolver authUserKeyResolver() {
   return exchange -> ReactiveSecurityContextHolder.getContext()
            .map(ctx -> ctx.getAuthentication().getPrincipal().toString());
}

For more details, you may refer to the article Secure Rate Limiting with Spring Cloud Gateway.

2. Generate and propagate certificates dynamically

Should we use SSL in microservice-to-microservice communication? Of course yes. But the question is how will you handle certificates used by your microservices. There are several best practices related to SSL certificate management. For example, you should not issue certificates for long time periods. You should also automatically renew or refresh them.

There are some tools that can help in following best practices. One of the most popular of them is Vault from Hashicorp. It provides the PKI secrets engine, which is responsible for generating dynamic X.509 certificates. The simplest way to try Vault is to run it locally on a Docker container.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

Then, we may enable and configure the PKI engine in some simple steps. You can do it using Vault CLI or UI. For some more detailed information about it read my article SSL with Spring WebFlux and Vault PKI. For now, let’s enable PKI with TTL and then configure CA using CLI as shown below.

$ vault secrets enable pki
$ vault secrets tune -max-lease-ttl=8760h pki
$ vault write pki/root/generate/internal \
    common_name=my-website.com \
    ttl=8760h

In the next step, we will use Spring VaultTemplate to issue a certificate dynamically. The fragment of code visible below shows how to create a certificate request with 12h TTL and localhost as a Common Name. Firstly, let’s build such a request using the VaultCertificateRequest object. Then we will invoke the issueCertificate method on the VaultPkiOperations object. The generated CertificateBundle contains both a certificate and a private key.

private CertificateBundle issueCertificate() throws Exception {
   VaultPkiOperations pkiOperations = vaultTemplate.opsForPki("pki");
   VaultCertificateRequest request = VaultCertificateRequest.builder()
        .ttl(Duration.ofHours(12))
        .commonName("localhost")
        .build();
   VaultCertificateResponse response = pkiOperations
      .issueCertificate("default", request);
   CertificateBundle certificateBundle = response.getRequiredData();
   return certificateBundle;
}

Finally, we just need to use the method for generating a certificate in Vault on runtime. The default behavior of our web server needs to be overridden. To do that, we need to create a Spring @Component that implements WebServerFactoryCustomizer. Depending on the web server we need to customize a different WebServerFactory. Typically, for Spring MVC it is Tomcat and Netty for Spring WebFlux. Inside the customize method, we are generating a certificate and store it inside the keystore (cert + private key) and truststore (CA).

@Component
public class GatewayServerCustomizer implements 
         WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {
   @SneakyThrows
   @Override
   public void customize(NettyReactiveWebServerFactory factory) {
      String keyAlias = "vault";
      CertificateBundle bundle = issueCertificate();
      KeyStore keyStore = bundle.createKeyStore(keyAlias);
      String keyStorePath = saveKeyStoreToFile("server-key.pkcs12", keyStore);
      Ssl ssl = new Ssl();
      ssl.setEnabled(true);
      ssl.setClientAuth(Ssl.ClientAuth.NEED);
      ssl.setKeyStore(keyStorePath);
      ssl.setKeyAlias(keyAlias);
      ssl.setKeyStoreType(keyStore.getType());
      ssl.setKeyPassword("");
      ssl.setKeyStorePassword("123456");
      X509Certificate caCert = bundle.getX509IssuerCertificate();
      log.info("CA-SerialNumber: {}", caCert.getSerialNumber());
      KeyStore trustStore = KeyStore.getInstance("pkcs12");
      trustStore.load(null, null);
      trustStore.setCertificateEntry("ca", caCert);
      String trustStorePath = saveKeyStoreToFile("server-trust.pkcs12", trustStore);
      ssl.setTrustStore(trustStorePath);
      ssl.setTrustStorePassword("123456");
      ssl.setTrustStoreType(trustStore.getType());
      factory.setSsl(ssl);
      factory.setPort(8443);
   }
}

3. Use SSL in microservices communication

Since using SSL on the edge of a microservices-based system is obvious, inter-service communication is sometimes considered to be non-secure. My recommendation is always the same. Better use SSL or not But we should think about securing at least components that store sensitive data. One of them will probably be a config server. That’s obviously one of the Spring Security best practices. For Spring Boot microservices we can use a component called Spring Cloud Config Server. Since it is built on top of Spring MVC we may easily enable a secure connection on the server-side.

server:
  port: ${PORT:8888}
  ssl:
    enabled: true
    client-auth: need
    key-store: classpath: server-key.jks
    key-store-password: 123456
    key-alias: configserver
    trust-store: classpath: server-trust.jks
    trust-store-password: 123456

On the client side, we use a component called Spring Cloud Config Client. Since it is responsible for connecting with the server, we also need to handle SSL there. To do that we need to override the RestTemplate SSL configuration on the ConfigServicePropertySourceLocator bean. The fragment of code visible below uses a self-signed certificate, but we can easily implement here a strategy described in the previous section.

@Configuration
public class SSLConfigServiceBootstrapConfiguration {
   @Autowired
   ConfigClientProperties properties;
   @Bean
   public ConfigServicePropertySourceLocator configServicePropertySourceLocator() throws Exception {
      final char[] password = "123456".toCharArray();
      final ClassPathResource resource = new ClassPathResource("account.jks");
      SSLContext sslContext = SSLContexts.custom()
         .loadKeyMaterial(resource.getFile(), password, password)
         .loadTrustMaterial(resource.getFile(), password, new TrustSelfSignedStrategy()).build();
      CloseableHttpClient httpClient = HttpClients.custom()
         .setSSLContext(sslContext)
         .setSSLHostnameVerifier((s, sslSession) -> true)
         .build();
      HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
      ConfigServicePropertySourceLocator configServicePropertySourceLocator = new ConfigServicePropertySourceLocator(properties);
      configServicePropertySourceLocator.setRestTemplate(new RestTemplate(requestFactory));
      return configServicePropertySourceLocator;
   }
}

In the latest version of Spring Cloud Config, we can enable TLS traffic encryption in configuration. We just need to define the right settings using properties with a prefix spring.cloud.config.tls.*.

What about encrypting communication between applications and a discovery server? You can choose between several available discovery servers supported in Spring Cloud. But let’s assume we use Eureka. Similarly to Spring Cloud Config, we use a high-level client to communicate with a server. So, in that case, we need to define a bean DiscoveryClientOptionalArgs, and also override SSL settings on the HTTP client there. The Eureka client uses the Jersey HTTP client, so we need to create an instance of EurekaJerseyClientBuilder to override the SSL configuration.

@Bean
public DiscoveryClient.DiscoveryClientOptionalArgs discoveryClientOptionalArgs() throws Exception {
   DiscoveryClient.DiscoveryClientOptionalArgs args = new DiscoveryClient.DiscoveryClientOptionalArgs();
   final char[] password = "123456".toCharArray();
   final ClassPathResource resource = new ClassPathResource("account.jks");
   SSLContext sslContext = SSLContexts.custom()
	.loadKeyMaterial(resource.getFile(), password, password)
	.loadTrustMaterial(resource.getFile(), password, new TrustSelfSignedStrategy()).build();
   EurekaJerseyClientBuilder builder = new EurekaJerseyClientBuilder();	 
   builder.withClientName("account-client");
   builder.withMaxTotalConnections(10);
   builder.withMaxConnectionsPerHost(10);
   builder.withCustomSSL(sslContext);
   args.setEurekaJerseyClient(builder.build());
   return args;
}

Finally, we may configure HTTPS in communication between microservices. Since we use RestTemplate or WebClient instances directly on the client side it is relatively easy to implement secure communication in that case.

4. Keep configuration data encrypted

The current one of the best practices for Spring microservices security is related to a configuration server. We should encrypt at least sensitive data like passwords or secrets stored there. Spring Cloud Config Server provides a built-in mechanism for that. But we can also use Vault as a backend store for Spring Cloud Config Server, where all data is encrypted by default.

We will start with a default encrypt mechanism provided by Spring Cloud Config Server. Firstly, we need to enable it in the configuration properties.

spring:
  cloud:
    config:
      server:
        encrypt:
          enabled: true

Then, we have to configure a key store responsible for encrypting our sensitive data.

encrypt:
  keyStore:
    location: classpath:/config.jks
    password: 123456
    alias: config
    secret: 123456

Finally, we can set encrypted data instead of plain string with {cipher} prefix.

spring:  
  application:
    name: account-service
  security:
    user:
      password: '{cipher}AQBhpDVYHANrg59OGY7ioSbMdOrH7ZA0vfa2VqIvfxJK5vQp...'

Alternatively, you can use it as a configuration data backend. To do that you should enable a Spring profile called vault.

spring.profiles.active=vault

Then, we may add an example secret.

$ vault write secret/hello value=world
$ vault read secret/hello

5. Restrict access to the API resources

In the previous sections, we discussed such topics as authentication, traffic, and data encryption. But another important aspect of securing your applications is authorization and access to the API resources. If you think about web app authorization, the first approach that probably comes to your mind is OAuth 2.0 or OpenID Connect. OAuth 2.0 is the industry-standard protocol for authorization. Of course, it is supported by Spring Security. There are also multiple OAuth2 providers you can integrate your application with. One of them is Keycloak. I will use it in the example in this article. Firstly, let’s run Keycloak on a Docker container. By default, it exposes API and a web console on the port 8080.

$ docker run -d --name keycloak -p 8888:8080 \
   -e KEYCLOAK_USER=spring \
   -e KEYCLOAK_PASSWORD=spring123 \
   jboss/keycloak

We are going to enable and configure OAuth 2.0 support on the API gateway. Besides spring-cloud-starter-gateway dependency, we need to include spring-boot-starter-oauth2-client and spring-cloud-starter-security to activate the TokenRelay filter. Then we have to provide the Spring Security configuration settings for the OAuth2 client.

spring:
  security:
    oauth2:
      client:
        provider:
          keycloak:
            token-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/token
            authorization-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/auth
            userinfo-uri: http://localhost:8080/auth/realms/master/protocol/openid-connect/userinfo
            user-name-attribute: preferred_username
        registration:
          keycloak-with-test-scope:
            provider: keycloak
            client-id: spring-with-test-scope
            client-secret: c6480137-1526-4c3e-aed3-295aabcb7609
            authorization-grant-type: authorization_code
            redirect-uri: "{baseUrl}/login/oauth2/code/keycloak"
          keycloak-without-test-scope:
            provider: keycloak
            client-id: spring-without-test-scope
            client-secret: f6fc369d-49ce-4132-8282-5b5d413eba23
            authorization-grant-type: authorization_code
            redirect-uri: "{baseUrl}/login/oauth2/code/keycloak"

In the last step, we need to configure the Spring Security filter. Since Spring Cloud Gateway is built on top of Spring WebFlux, we need to annotate the configuration bean with @EnableWebFluxSecurity. Inside the filterChain method we are going to enable authorization for all the exchanges. We will also set OAuth2 as a default login method and finally disable CSRF.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {
   @Bean
   public SecurityWebFilterChain filterChain(ServerHttpSecurity http) {
      http.authorizeExchange(exchanges -> exchanges.anyExchange().authenticated())
         .oauth2Login(withDefaults());
      http.csrf().disable();
      return http.build();
   }
}

As I mentioned before, we will have a token relay pattern between the gateway and microservices. A Token Relay is where an OAuth2 consumer acts as a Client and forwards the incoming token to outgoing resource requests. So, now let’s enable global method security and OAuth2 resources server for the downstream services.

@Configuration
@EnableWebSecurity
@EnableGlobalMethodSecurity(prePostEnabled = true)
public class SecurityConfig extends WebSecurityConfigurerAdapter {
   protected void configure(HttpSecurity http) throws Exception {
      http.authorizeRequests(authorize -> authorize.anyRequest().authenticated())
         .oauth2ResourceServer(OAuth2ResourceServerConfigurer::jwt);
   }
}

After that, it is possible to configure role-based access using @PreAuthorize and @PostAuthorize. Let’s take a look at the implementation of the REST controller class. It is a single ping method. That method may be accessed only by the client with the TEST scope. For more implementation details you may refer to the article Spring Cloud Gateway OAuth2 with Keycloak.

@RestController
@RequestMapping("/callme")
public class CallmeController {
   @PreAuthorize("hasAuthority('SCOPE_TEST')")
   @GetMapping("/ping")
   public String ping() {
      SecurityContext context = SecurityContextHolder.getContext();
      Authentication authentication = context.getAuthentication();
      return "Scopes: " + authentication.getAuthorities();
   }
}

6. Dynamically generate credentials to the external systems

Does your application connect to external systems like databases or message brokers? How do you store the credentials used by your application? Of course, we can always encrypt sensitive data, but if we work with many microservices having separate databases it may not be a very comfortable solution. Here comes Vault with another handy mechanism. Its database secrets engine generates database credentials dynamically based on configured roles. We may also take advantage of dynamically generated credentials for RabbitMQ, Nomad, and Consul.

Firstly, let’s enable the Vault database engine, which is disabled by default.

$ vault secrets enable database

Let’s assume our application connects to the Postgres database. Therefore, we need to configure a Vault plugin for PostgreSQL database and then provide connection settings and credentials.

$ vault write database/config/postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="default" \
  connection_url="postgresql://{{username}}:{{password}}@localhost:5432?sslmode=disable" \
  username="postgres" \
  password="postgres123456"

Then we need to create a database role. The name of the role should be the same as the name passed in field allowed_roles in the previous step. We also have to set a target database name and SQL statement that creates users with privileges.

$ vault write database/roles/default db_name=postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}';GRANT SELECT, UPDATE, INSERT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";GRANT USAGE,  SELECT ON ALL SEQUENCES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="1h" max_ttl="24h"

Thanks to Spring Cloud Vault project we can easily integrate any Spring Boot application with the Vault databases engine. Two dependencies need to be included in Maven pom.xml to enable that support. Of course, we also need dependencies for the JPA and Postgres driver.

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-vault-config</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-vault-config-databases</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
   <groupId>org.postgresql</groupId>
   <artifactId>postgresql</artifactId>
</dependency>

The only thing we have to do is to configure integration with Vault via Spring Cloud Vault. The following configuration settings should be placed in bootstrap.yml (no application.yml). You may consider running the application on Nomad.

spring:
  cloud:
    vault:
      uri: http://localhost:8200
      token: ${VAULT_TOKEN}
      postgresql:
        enabled: true
        role: default
        backend: database
  datasource:
    url: jdbc:postgresql://localhost:5432/posgtres

The important part of the configuration visible above is under the property spring.cloud.vault.postgresql. Following Spring Cloud documentation “Username and password are stored in spring.datasource.username and spring.datasource.password so using Spring Boot will pick up the generated credentials for your DataSource without further configuration”. For more details about integration between Spring Cloud and Vault database engine, you may refer to my article Secure Spring Cloud Microservices with Vault and Nomad.

7. Always be up to date

This one of the best practices may be applied anywhere not only as a rule to Spring microservices security. We usually use open-source libraries in our applications, so it is important to include the latest versions of them. They may contain critical updates for publicly disclosed vulnerabilities contained within a project’s dependencies. There are also several dependency scanners like Snyk or OWASP.

Final thoughts

That is my private list of best practices for Spring Boot microservices security. Of course, most of them are not related just to a single framework, and we apply them for any other framework or toolkit. Do you have your own list of Spring Security best practices? Don’t afraid to share it in the comments.

The post Spring Microservices Security Best Practices appeared first on Piotr's TechBlog.

Let’s say a little bit more about Vault. It allows you to secure, store, and control access to tokens, passwords, certificates, and encryption keys using UI, CLI, or HTTP API. It is a really powerful tool. With Vault, instead of a traditional approach, you can manage your security in a more dynamic, cloud-native way. For example, you can integrate Vault with a database backend, and then generate user login and password on the fly. Moreover, for Spring Boot applications you can take an advantage of the Spring Cloud Vault project. If you are interested in more information about it read my article Testing Spring Boot Integration with Vault and Postgres using Testcontainers.

That’s not all. You can integrate Vault with other tools from Hashicorp like Consul or Nomad. In other words, it allows us to build a cloud-native platform in a secure way. For more details please refer to the article Secure Spring Cloud Microservices with Vault and Nomad.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my repository sample-spring-cloud-security. Then you should go to the gateway-service directory, and just follow my instructions in the next sections. The sample application acts as an API gateway for microservices. We use Spring Cloud Gateway. Since it is built on top of Spring WebFlux, that example is perfectly right for our current article.

1. Running Vault

We will run Vault inside the Docker container in development mode. The server running in that mode does not require any further setup, it is ready to use just after startup. After startup, our instance of Vault is available on port 8200. The version of Vault used in this article is 1.7.1.

$ docker run --cap-add=IPC_LOCK -d --name vault -p 8200:8200 vault

It is possible to login using different methods, but the most suitable way for us is through a token. To do that we have to display container logs using command docker logs vault, and then copy Root Token as shown below.

Finally, we are able to login to the Vault web console.

2. Enable and configure Vault PKI

There are two ways to enable and configure Vault PKI: with CLI or via UI. Most articles describe a list of CLI commands required to configure the PKI engine. One of them is available on the Hashicorp site. However, I’m going to use Vault UI for that. So first, let’s enable a new engine on the main site.

Then, we need to choose a type of engine to enable. In our case it is the option PKI Certificates.

During creation let’s leave a default name pki. Then, we need to navigate into the newly enabled engine and create a new role. A role is used for generating certificates. The name of my role is default. This name is important because we would have to call it from the code using VaultTemplate.

The type of the key used in our role is rsa.

Before creating it, we should set some important parameters. One of them is TTL, which is set to 3 days. Also, don’t forget to check fields Allow any name and Require Common Name. Both of them are related to the CN field inside the certificate. Because we will store a username inside the CN field, we need to allow any name for it.

Once a role is created, we need to configure CA. To do that, we should first switch to the Configuration tab and then click Configure button.

After that, let’s choose Configure CA.

Finally, we can create a new CA certificate. We should leave the root value as CA Type and internal as Type. The default key format is pem. We can also set a Common Name for the CA certificate. For both role and CA it worth filling additional fields like e.g. the name of an organization or organization unit.

3. Integrating Spring WebFlux with Vault PKI

Let’s begin with dependencies. We need to include a Spring WebFlux starter for reactive API and a Spring Security starter to secure API. Integration with Vault API can be provided by spring-vault-core. I also had to include Jackson libraries in order to be able to start the application with Spring Vault.

<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-cloud-starter-webflux</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-core</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-databind</artifactId>
</dependency>
<dependency>
   <groupId>com.fasterxml.jackson.core</groupId>
   <artifactId>jackson-annotations</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.vault</groupId>
   <artifactId>spring-vault-core</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-security</artifactId>
</dependency>

Then, let’s configure a VaultTemplate bean. It should use the http scheme and an authentication token injected from the configuration.

@Value("vault.token")
private String vaultToken;

@Bean
VaultTemplate vaultTemplate() {
   VaultEndpoint e =  new VaultEndpoint();
   e.setScheme("http");
   VaultTemplate template = new VaultTemplate(e, new TokenAuthentication(vaultToken));
   return template;
}

The VaultTemplate provides dedicated support for interaction with the PKI engine. We just need to call the method opsForPki passing the PKI engine name to obtain the VaultPkiOperations instance (1). Then we need to build a certificate request with VaultCertificateRequest. We may set several parameters, but the most important is CN and certificate TTL (2). Finally, we should invoke the issueCertificate method passing the request and the name of the role configured on Vault PKI (3). Our certificate has been successfully generated. Now, we just need to obtain it from the response. The generated certificate, CA certificate, and a private key are available inside the CertificateBundle object, which is returned by the method.

private CertificateBundle issueCertificate() throws Exception {
   VaultPkiOperations pkiOperations = vaultTemplate.opsForPki("pki"); // (1)
   VaultCertificateRequest request = VaultCertificateRequest.builder()
        .ttl(Duration.ofHours(12))
        .commonName("localhost")
        .build(); // (2)
   VaultCertificateResponse response = pkiOperations.issueCertificate("default", request); // (3)
   CertificateBundle certificateBundle = response.getRequiredData(); // (4)

   log.info("Cert-SerialNumber: {}", certificateBundle.getSerialNumber());
   return certificateBundle;
}

4. Enable Spring WebFlux security

We have already integrated Spring WebFlux with Vault PKI in the previous section. Finally, we can proceed to the last step in our implementation – enable security based on X.509 certificates. To do that we need to create a @Configuration class. It should be annotated with @EnableWebFluxSecurity (1). We also need to obtain a username from the certificate by implementing the principal extractor. We are going to use SubjectDnX509PrincipalExtractor (2) with the right regex that reads data from the CN field. The final configuration disables CSRF, basic auth, and enables SSL with X509 certificates (3). We also need to provide an implementation of the UserDetails interface (4) with a single username piotrm.

@Configuration
@EnableWebFluxSecurity // (1)
public class SecurityConfig {

   @Bean
   public SecurityWebFilterChain filterChain(ServerHttpSecurity http) {
      SubjectDnX509PrincipalExtractor principalExtractor =
             new SubjectDnX509PrincipalExtractor(); // (2)
      principalExtractor.setSubjectDnRegex("CN=(.*?)(?:,|$)");

      return http.csrf().disable()
             .authorizeExchange(exchanges -> 
                    exchanges.anyExchange().authenticated())
             .x509()
                .principalExtractor(principalExtractor)
             .and()
                .httpBasic().disable().build(); // (3)
   }

   @Bean
   public MapReactiveUserDetailsService users() { // (4)
      UserDetails user1 = User.builder()
             .username("piotrm")
             .password("{noop}1234")
             .roles("USER")
             .build();
      return new MapReactiveUserDetailsService(user1);
   }
}

In the last step, we need to override the Netty server SSL configuration on runtime. Our customizer should implement the WebServerFactoryCustomizer interface, and use NettyReactiveWebServerFactory. Inside customize method we first invoke the method issueCertificate responsible for generating a certificate in Vault (you can refer to the previous section to see the implementation of that method) (1). The CertificateBundle contains all required data. We can invoke the method createKeyStore on it to create a keystore (2) and then save it in the file (3).

To override Netty SSL settings we should use the Ssl object. The client authentication needs to be enabled (4). We will also set the location of the currently created KeyStore (5). After that, we may proceed to the truststore creation. The issuer certificate may be obtained from CertificateBundle (6). Then we should create a new keystore, and set the CA certificate as an entry there (7). Finally, we will save the truststore to the file and set its location in the Ssl object.

@Component
@Slf4j
public class GatewayServerCustomizer implements 
         WebServerFactoryCustomizer<NettyReactiveWebServerFactory> {

   @SneakyThrows
   @Override
   public void customize(NettyReactiveWebServerFactory factory) {
      String keyAlias = "vault";
      CertificateBundle bundle = issueCertificate(); // (1)
      KeyStore keyStore = bundle.createKeyStore(keyAlias); // (2)
      String keyStorePath = saveKeyStoreToFile("server-key.pkcs12", keyStore); // (3)

      Ssl ssl = new Ssl();
      ssl.setEnabled(true);
      ssl.setClientAuth(Ssl.ClientAuth.NEED); // (4)

      ssl.setKeyStore(keyStorePath); // (5)
      ssl.setKeyAlias(keyAlias);
      ssl.setKeyStoreType(keyStore.getType());
      ssl.setKeyPassword("");
      ssl.setKeyStorePassword("123456");

      X509Certificate caCert = bundle.getX509IssuerCertificate(); // (6)
      log.info("CA-SerialNumber: {}", caCert.getSerialNumber());
      KeyStore trustStore = KeyStore.getInstance("pkcs12");
      trustStore.load(null, null);
      trustStore.setCertificateEntry("ca", caCert); // (7)
      String trustStorePath = saveKeyStoreToFile("server-trust.pkcs12", trustStore); // (8)

      ssl.setTrustStore(trustStorePath); // (9)
      ssl.setTrustStorePassword("123456");
      ssl.setTrustStoreType(trustStore.getType());

      factory.setSsl(ssl);
      factory.setPort(8443);
   }
}

5. Testing Spring WebFlux with Vault PKI

Let’s run our sample application. It is available under the 8443 port. We will test it using the curl tool. Before doing it we need to generate a client certificate with a private key. Let’s go to the Vault UI once again. If you click a default Vault UI redirects to form responsible for certificate generation as shown below. In the Common Name field, we should provide the test username configured inside the UserDetails implementation. For me, it is piotrm. Also, don’t forget to set the right TTL.

After generating a certificate you will be redirected to the site with the results. First, you should copy the string with your certificate, and save it to the file. For me it is piotrm.crt. You can also display the content of a generated private key. Then, do the same as with the certificate. My filename is piotrm.key.

Finally, we can send a test request to our sample application passing the names of key and certificate files.

$ curl https://localhost:8443/hello -v --key piotrm.key --cert piotrm.crt

The post SSL with Spring WebFlux and Vault PKI appeared first on Piotr's TechBlog.

• Encryption and decryption of sensitive data
• Setting up SSL configuration on the server side
• SSL connection on the client side

1. Encryption and decryption

If you use JDK 8 or lower, you first need to download and install the Java Cryptography Extension (JCE) provided by Oracle. It consists of two JAR files (local_policy.jar and US_export_policy.jar), which need to override the existing policy files in the JRE lib/security directory. Since our sample applications use JDK 11 we don’t have to install any additional libraries. JDK 9 and later use by default the unlimited policy files. Sample applications source code is available in my GitHub repository sample-spring-cloud-security in the branch secure_config: https://github.com/piomin/sample-spring-cloud-security/tree/secure_config.
If the remote property sources stored on the config server contain encrypted data, their values should be prefixed with {cipher} and wrapped in quotes to designate it as a YAML file. Wrapping in quotes is not necessary for .properties files. If such a value cannot be decrypted, it is replaced by an additional value (usually ) under the same key prefixed with invalid.
Now, let’s consider the case where we are securing our API with Basic Auth mechanism. To enable it for our sample Spring Boot application (for example account-service) we first need to include spring-boot-starter-security artifact except standard Web and Cloud Config Client starters:

<dependency>
   <groupId>org.springframework.cloud</groupId>
   <artifactId>spring-cloud-starter-config</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-security</artifactId>
</dependency>   

When adding Security starter Basic Auth is enabled by default. We can override username and password. First, let’s encrypt a sample password using encrypt endpoint exposed by Spring Cloud Config Server. This endpoint is responsible for encrypting a given parameter using the key stored on the server. Our test password is demotest. Let’s encrypt it:

There is also an opposite endpoint that allows to decrypt already encrypted parameter. Let’s use it just for check:

Now, we will prepare the following YAML properties file for account-service, that is stored on the config server.

spring:  
  application:
    name: account-service
  security:
    user:
      password: '{cipher}AQDDqhpwfKGBPf1KZ0lPuHpRHb1mGTdRt3VMioFhTFyNaVJB5Sn+3zguMJUUntXq3Sib/2zg8n6RG5DHKApXCd5DeVicrOkarqrWVRFc2VO10Yg6anGvHRfqWKD0D7T8xXGrDx0i06UR2sj5ZyNRiGT/Lzy+2pWMxZrSepP9caiVKElLt1a/oTwyiTECA2A/5KulHG39+bJPEeZqycLffmjdwWAy3DXxDmDbEqsxsTnCD9wE9Dc4etl1eWhtHG1UR8QkPmUti+eqj2rDzAqr59p4FHlbc6VnGgUmL2fz2MQFqzghdGaBhWLuC5v/TdpqJsyred2o7n653opDD5NeEZRy1/Uipwu9hf563mM9wCLyKj6lx2ieuNag6YSOiKTHVCU='

The sample application account-service is retrieving properties from config server. Before using encrypted values it has to decrypt it. Therefore, it needs to have a keystore that allows to decrypt the password. The architecture is visualized in the following picture.

Here’s the required configuration on the client side. Like I have mentioned before, account-service need to use a key for decryption. It has to be set using encrypt.keyStore.* properties.

encrypt:
  keyStore:
    location: classpath:/account.jks
    password: 123456
    alias: account
    secret: 123456

After startup application is available on port 8091. If we try to call any endpoint without Authorization header we should receive HTTP 401.

In the request visible below we set Authorization header. The default username is user. The password has been already set to demotest.

2. Configure SSL on Spring Cloud Config Server

You have probably noticed that I used parameter --insecure for curl examples in the previous section. I didn’t pass any key and the server does not reject my requests. Now, we will force Spring Cloud Config Server to verify client SSL certificate during connection attempt. The appropriate configuration is visible below. We are forcing client authentication by setting property server.ssl.client-auth to true. We also should set truststore and keystore locations.

server:  
  port: ${PORT:8888}
  ssl:
    enabled: true
    client-auth: need
    key-store: classpath:config.jks
    key-store-password: 123456
    trust-store: classpath:config.jks
    trust-store-password: 123456
    key-alias: discovery

The next step is to generate keys and certificates. We will use the command-line tool provided under JRE — keytool. Let’s begin with a well-known command for generating a keystore file with a key pair. One keystore is generated for a config
server, while the second for the client application, in this particular case, for account-service:

$ keytool -genkey -alias account -store type JKS -keyalg RSA -keysize 2048 -
$ keystore account.jks -validity 3650
$ keytool -genkey -alias config -storetype JKS -keyalg RSA -keysize 2048 -
$ keystore config.jks -validity 3650

Then, the self-signed certificate has to be exported from a keystore to a file — for example, with a .cer or .crt extension. You will then be prompted for the password you provided during the keystore generation:

$ keytool -exportcert -alias account -keystore account.jks -file account.cer
$ keytool -exportcert -alias config -keystore config.jks -file config.cer

The certificate corresponding to the public key has been extracted from the keystore, so now it can be distributed to all interested parties. The public certificate from account-service should be included in the discovery server’s trust store and vice-versa:

$ keytool -importcert -alias config -keystore account.jks -file config.cer
$ keytool -importcert -alias account -keystore config.jks -file account.cer

There is also some additional configuration on the server-side. We have to enable native profile and encryption.

spring:
  application:
    name: config-service
  cloud:
    config:
      server:
        encrypt:
          enabled: false
  profiles:
    active: native

3. Configure SSL for Spring Cloud Config Client

After enabling SSL on the server side we may proceed to the client side implementation. We will have to use HttpClient to implement secure connection with a server. So, first let’s include the following artifact to the Maven dependencies:

<dependency>
   <groupId>org.apache.httpcomponents</groupId>
   <artifactId>httpclient</artifactId>
</dependency>

We use 2.2.1.RELEASE version of Spring Boot and Hoxton.RELEASE Release Train of Spring Cloud.

<properties>
   <java.version>11</java.version>
</properties>

<parent>
   <groupId>org.springframework.boot</groupId>
   <artifactId>spring-boot-starter-parent</artifactId>
   <version>2.2.1.RELEASE</version>
</parent>

<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.springframework.cloud</groupId>
         <artifactId>spring-cloud-dependencies</artifactId>
         <version>Hoxton.RELEASE</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

Spring Cloud Config Client uses RestTemplate for HTTP communication with a config server. This communication is performed in the bootstrap phase. That’s very important information, because we have to override RestTemplate bean properties before retrieving configuration sources from a config server. First, we need to implement secure connection using HttpClient and ConfigServicePropertySourceLocator bean. Here’s the implementation of custom ConfigServicePropertySourceLocator bean inside @Configuration class.

@Configuration
public class SSLConfigServiceBootstrapConfiguration {

   @Autowired
   ConfigClientProperties properties;

   @Bean
   public ConfigServicePropertySourceLocator configServicePropertySourceLocator() throws Exception {
      final char[] password = "123456".toCharArray();
      final ClassPathResource resource = new ClassPathResource("config.jks");

      SSLContext sslContext = SSLContexts.custom()
         .loadKeyMaterial(resource.getFile(), password, password)
         .loadTrustMaterial(resource.getFile(), password, new TrustSelfSignedStrategy()).build();
      CloseableHttpClient httpClient = HttpClients.custom()
         .setSSLContext(sslContext)
         .setSSLHostnameVerifier((s, sslSession) -> true)
         .build();
      HttpComponentsClientHttpRequestFactory requestFactory = new HttpComponentsClientHttpRequestFactory(httpClient);
      ConfigServicePropertySourceLocator configServicePropertySourceLocator = new ConfigServicePropertySourceLocator(properties);
      configServicePropertySourceLocator.setRestTemplate(new RestTemplate(requestFactory));
      return configServicePropertySourceLocator;
   }

}

The configuration class defined above needs to be loaded on a bootstrap phase. To do that we have to create file spring.factories in the src/main/resources/META-INF directory:

org.springframework.cloud.bootstrap.BootstrapConfiguration = pl.piomin.services.account.SSLConfigServiceBootstrapConfiguration

Finally we may start the account-service application. It establishes a secure SSL connection with Spring cloud config Server using two-way (mutual) authentication. Here’s the fragment of logs during application startup.

Conclusion

Our scenario may be extended with a secure SSL discovery connection based on Spring Cloud Netflix Eureka server. An example of secure discovery between Spring microservice and Eureka has been described in the following article: Secure Discovery with Spring Cloud Netflix Eureka. In this article, I show you the most interesting aspects related to distributed configuration security with Spring Cloud Config. For the working example of application you should refer to my GitHub repository https://github.com/piomin/sample-spring-cloud-security/tree/secure_config.

The post Secure Spring Cloud Config appeared first on Piotr's TechBlog.