There is a similar article on my blog about “cert-manager” and Spring Boot SSL hot reload more focused on Kubernetes. You can also read the post on how to integrate Spring Boot with Vault PKI to generate certs automatically.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Let’s check out the openshift branch. Then, we need to go to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

$ git clone https://github.com/piomin/sample-spring-security-microservices.git
$ git checkout openshift
$ cd ssl

How It Works

OpenShift brings a lot of interesting features to the Kubernetes platform. The “Service Serving Certificates” is one of the examples. Thanks to that mechanism, we don’t need to install any external tool for generating and rotating certificates dynamically like “cert-manager”. The OpenShift CA service automatically rotates certificates 13 months before expiration and issues new certificates for 26 months. After a rotation, the previous CA is trusted until it expires. To achieve it, OpenShift implements the root CA key rollover process described in  https://tools.ietf.org/html/rfc4210#section-4.4. This allows a grace period for all affected services to refresh their key material before the expiration. Sounds interesting? Let’s proceed to the implementation.

In order to test a mechanism, we will run two simple Spring Boot apps on OpenShift that communicate using SSL certificates. OpenShift takes care of issuing/rotating certificates and exposing the CA bundle as a ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using Service Serving Certificates”. Here’s the illustration of our apps’ architecture on OpenShift.

Create a Server Side App

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now, our main goal is to enable HTTPS for that app and make it work properly with the OpenShift “Service Serving Certificates”. Firstly, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure keystores for the web server (3). We can configure SSL bundles using PEM-encoded text files directly in the app with the spring.ssl.bundle.pem properties group. The certificate (tls.crt) and private key (tls.key) will be mounted as a volume to the pod. Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property (2). Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl.bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

The only thing we need to do to generate SSL certificates for our app is to annotate the Kubernetes Service with service.beta.openshift.io/serving-cert-secret-name. It should contain the name of the target Kubernetes Secret as a value.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Once we apply such a Service to the cluster, OpenShift generates the secure-callme-bundle Secret that contains the certificate and private key.

After that, we just need to deploy the app and mount the generated Secret as a volume. The volume path is passed to the app as the CERT_PATH environment variable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-bundle

Finally, we need to expose the HTTP endpoint outside of the cluster. We can easily do it with the OpenShift Route. Don’t forget the set the TLS termination as passthrough. Thanks to that OpenShift router sends traffic directly to the destination without TLS termination.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-callme-bundle
  labels:
    app: secure-callme-bundle
    app.kubernetes.io/component: secure-callme-bundle
    app.kubernetes.io/instance: secure-callme-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-callme-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Create a Client Side App

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the truststore taken from ConfigMap with CA certificates. With the certs inside the service-ca.crt file, the RestTemplate bean can authenticate against the secure-callme-bundle app.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key
  client:
    truststore:
      certificate: ${CLIENT_CERT_PATH}/service-ca.crt

Don’t forget about creating the RestTemplate bean. It needs to the SslBundles bean and set it on the RestTemplateBuilder.

@Bean
RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
   return builder
      .setSslBundle(sslBundles.getBundle("client")).build();
}

We need to create the ConfigMap annotated with service.beta.openshift.io/inject-cabundle=true. After that, the OpenShift cluster automatically injects the service CA certificate into the service-ca.crt key on the config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-caller-bundle
  namespace: demo-secure
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
data: {}

Let’s display the details of the secure-caller-bundle ConfigMap. As you see, OpenShift injects the PEM-encoded CA certificates under the service-ca.crt key to that config map.

Finally, we can deploy our client-side app. Here’s the YAML manifest with the Deployment object. Since the secure-caller-bundle app acts as a web server and also uses a REST client to invoke the secure-callme-bundle we need to mount two volumes. The first one (/opt/secret) contains a certificate and private key for the web server (1). Consequently, the second one (/opt/client-cert) holds the CA bundle used by the RestTemplate client (2). This client communicates with secure-callme-bundle internally through the Kubernetes Service (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-cert
            # (3)
            - name: HOST
              value: secure-callme-bundle.demo-secure.svc
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            # (1)
            - mountPath: /opt/secret
              name: cert
            # (2)
            - mountPath: /opt/client-cert
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-bundle
        - name: client-cert
          configMap:
            name: secure-caller-bundle

Of course, we should also create a passthrough Route to expose our app the outside of the cluster:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-caller-bundle
  labels:
    app: secure-caller-bundle
    app.kubernetes.io/component: secure-caller-bundle
    app.kubernetes.io/instance: secure-caller-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-caller-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Testing OpenShift Certs SSL Rotation with Spring Boot

The main problem with testing the certs rotation scenario is in the 13-month interval. We can’t change that period in the configuration. However, there is a way how we can force OpenShift to rotate the CA certs on demand. In order to do that, we need to set the spec.unsupportedConfigOverrides.forceRotation.reason property on the cluster-wide ServiceCA object with any value. Of course, you shouldn’t do that at production, since it is not supported by OpenShift. It is just designed for testing or development purposes.

apiVersion: operator.openshift.io/v1
kind: ServiceCA
metadata:
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  unsupportedConfigOverrides:
    forceRotation:
      reason: testing

We will call our apps through the OpenShift Route. So, let’s display a current list of routes related to our sample apps:

$ oc get route
NAME                   HOST/PORT                                                     PATH   SERVICES               PORT    TERMINATION   WILDCARD
secure-caller-bundle   secure-caller-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-caller-bundle   https   passthrough   None
secure-callme-bundle   secure-callme-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-callme-bundle   https   passthrough   None

Once we set that property, OpenShift immediately rotates that CA certificate. It will also take care of synchronizing the latest certificates with all the secrets and config maps containing annotations required by the “Service Serving Certificates” mechanism. As you see, now, our secure-caller-bundle ConfigMap contains two CAs: an old one and a new one.

$ oc describe cm secure-caller-bundle
Name:         secure-caller-bundle
Namespace:    demo-secure
Labels:       <none>
Annotations:  service.beta.openshift.io/inject-cabundle: true

Data
====
service-ca.crt:
----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIITnr+EqYsYtEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAzMDExMDM5MTVaFw0yNjA0MzAxMDM5MTZaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDN0No0DIZ+BCdPvQ9TIvEjwGUSoUXM+DYI
Dxlk2Ef6goFv3YsAgNJTxznntRc69dJTrLtRrmFN0d31JPM/gJOMkKMvTWnz8Qzx
athuvvwbivJNKh/qn+Dhewbjrx5LGpq3v9VQ7t/5Rf5F9VMpyp738EwfAy2cfxTA
sxgYpvU/foivM9U0uMaZyXA5vLsGN6cKpEREGMqKbIKSXNYXD9/lbHv2eyr+5+s9
keYEDkGTWMceISihm5mGNwOZpLZhNwFgSZR9O63TIgGc/TEZ3EYwcIjmlVOSwQ/u
sM43HL1jtYKw3AovQOc5qc2kT5eSPV7jVCCfNYnj3BLvt0AYUXE/AgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT3iI1f
71ZVt6OVLC/+0oxNL7BqhzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEATcT/LK5pAaOGz0j3MR4NhsfmAjo4K8SoGmFE
F4lDzDWZTaJuJHCHqpblgGKLfokU9BQC3L4NZLIisOj3CWF3zIpC6MOsU7x9qDm0
aZpI1gqhRXQZOqVpE9XJ0XBEOLP9hlzxa66SQlALlU5CWbDNEMtXsMMQf1lSgIRz
arWK8QwmPsmg6duoVtFcCMY2klkVacZctiMgM4wvf1CCJt3TkiggwO3t4IdiV8Xa
Z9d/LlV/bFzboxGxOb2IMHj5VXKPM6HmC4slBSqKeyr7PYyM7rjhorIOOner9JrX
RhbC8/uARciLbcFKyOOCSLoe5m1i8QPrRmCm/GhVS9M4x86TXw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIFlMUWK4HjW4wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAyMjgxMTUyMjNaFw0yNjA0MjgxMTUyMjRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY6AZbLN0kT2BPXbDn7iZmbyYZgyI8Y3jh
hRq9JrX0ZIIs5BvFcRRkm2IcJ8gsMF4VztGCkNoDQ07ojAIHI/FxssAU8wwoz2QP
eH2qzBpLA1lBAYvtjki//55STJvJF+7Z6qbcDwUVX6/r+hrpy5MIjaQmRV4DRLpp
ZD3HrqYozJEvCKaA2pinC2VyW8IBDB0FDvPdBxvzjrCDjssr/v0jqFgUGFJJmKEj
EaFQACeMlOS4Q5avEglfwoLS+RGgPjE1s1gpxir2dZCdnM8b9B3CFy2sS+SSHN4y
+FJRJBv4CbMAMP+DLIsHY872c2XmOlleJNFpiFWi9r33LCc7MnbjAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRey/oP
FWW5kG8ifSXqIXBh2L+FdzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEAhbimn2Tv5w9Cy7i6dNdYYdVhA52cCnWWDDhf
s/dZ+Zl1aP3f179XO0xz3ovLuE227Z9AYPGtPW6c0FQgxgDgl2pYUX4J88caTUG/
eR9EG9LEfrL6OlKMjMvAFWrL64bcAAiuoz/gWB0a29xMnUsA52+7OpnzmHMduaaz
5JfRaTwkjGGe8ay2XsTUNrNGqZJGKHhE+hrUVlFsS7GpaSEe9GllkS9lGUZp2Hei
lTxwnBr9j+aP/QWXwuAVxT1DUAByJYoTPrpMhk+hKQUj/8TvxmMmsksZ57steqRI
/gWYd4j2SohN3jd1mv6rQ+jsGGAfCNmYVZL1oevNaqfnENt0IA==
-----END CERTIFICATE-----

However, if we don’t restart the pods, our apps still use the old keystore and truststore (unless they have the reload-on-update option enabled).

Here’s a list of running pods.

oc get po
NAME                                    READY   STATUS    RESTARTS   AGE
secure-caller-bundle-57c7658d85-bj4q5   1/1     Running   0          2d23h
secure-callme-bundle-cbf59569f-n6gg5    1/1     Running   0          2d23h

Let’s just restart the secure-callme-bundle app by deleting its pod. Do not restart the secure-callme-bundle pod, because we want to test the scenario where it uses the old truststore.

$ oc delete po secure-callme-bundle-cbf59569f-n6gg5
pod "secure-callme-bundle-cbf59569f-n6gg5" deleted

Then, we can call the GET /callme endpoint exposed by the secure-callme-bundle app through the Route once again. As you see, it servers the latest, reloaded certificate.

Let’s perform a final test. Now, we call the GET /caller endpoint exposed by the secure-caller-bundle app. Under the hood, it uses RestTemplate to call the GET /callme endpoint exposed by the secure-callme-bundle app via secure-callme-bundle.demo-secure.svc address. As you see the communication still works fine although the secure-caller-bundle app still uses the keystore and truststore.

Final Thoughts

The OpenShift “Service Serving Certificates” mechanism is simple to configure and doesn’t require that you handle the process of reloading apps with the rotated trusted material. It also has some drawbacks. You cannot change the rotation period from 13 months into a different value, and there is a global CA bundle shared across all the services running on the cluster. If those things are problematic for you I advise you to use “cert-manager” and a tool that allows you to restart pods automatically on the Secret or ConfigMap change. It is also possible to leverage the Spring SslBundles to reload certs without restarting the pod. With Spring Boot apps you can also use OpenShift Service Mesh, which can handle SSL communication at the platform level.

The post Rotate SSL Certificates with OpenShift and Spring Boot appeared first on Piotr's TechBlog.

In order to generate SSL certificates on Kubernetes we will use cert-manager. “Cert-manager” can rotate certificates after a specified period and save them as Kubernetes Secrets. I have already described how to implement a similar scenario with an automatic restart of a pod on a secret update in the article here. We were using the Stakater Reloader tool to restart the pod automatically on a new version of Secret. However, this time we use Spring Boot features to avoid having to restart an app (pod).

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

How It Works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need to design a solution for enabling SSL/TLS communication between the services running on Kubernetes. This solution must take into account a scenario of certificates reloading. Moreover, it must happen at the same time for the both server and client sides to avoid errors in the communication. On the server side, we use an embedded Tomcat server. In the client-side app, we use the Spring RestTemplate object.

“Cert-manager” can generate certificates automatically, based on the provided CRD object. It ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It serves all the required staff as the Kubernetes Secret. Such a secret is then mounted as a volume into the app pod. Thanks to that we don’t need to restart a pod, to see the latest certificates or “keystores” inside the pod. Here is the visualization of the described architecture.

Install cert-manager on Kubernetes

In order to install both “cert-manager” on Kubernetes we will use its Helm chart. We don’t need any specific settings. Before installing the chart we have to add CRD resources for the latest version 1.14.2:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart in the cert-manager namespace using the following command:

$ helm install my-release cert-manager jetstack/cert-manager \
    -n cert-manager

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0 

That’s all. Now we can proceed to the implementation.

Spring Boot SSL Hot Reload on the Embedded Server

Sample App Implementation

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our main goal is to enable HTTPS for that app and make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure SSL trust material for the web server (3). There are two types of trusted material it can support. In order to configure bundles using Java keystore files, we have to use the spring.ssl.bundle.jks group. On the other hand, it is possible to configure bundles using PEM-encoded text files with the spring.ssl.bundle.pem properties group.

In the exercise, we will use the Java keystore files (JKS). We define a single SSL bundle under the server name. It contains both keystore and truststore locations. With the reload-on-update property, we can instruct Spring Boot to watch the files in the background and trigger a web server reload if they change. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property (2). Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property. Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl:
  client-auth: NEED
  bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Generate Certificates with Cert-manager

Before we deploy the callme-secure-bundle app on Kubernetes, we need to configure “cert-manager” and generate the required certificates. Firstly, we need to define the CRD object responsible for issuing certificates. Here’s the ClusterIssuer object that generates self-signed certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-cluster-issuer
spec:
  selfSigned: {}

Here’s the Kubernetes Secret with the password used for securing generated keystores:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

After that, we can generate certificates. Here’s the Certificate object for the app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer, which has been created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme-bundle, so the cert needs to have that name as CN. In order to enable certificate rotation, we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration “cert-manager” will automatically renew a certificate (5). However, it won’t rotate the private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme-bundle
    - localhost
  secretName: secure-callme-cert
  commonName: secure-callme-bundle
  duration: 1h
  renewBefore: 5m

Deploy on Kubernetes

After creating a certificate we can proceed to the secure-callme-bundle app deployment. It mounts the Secret containing certificates and keystores as a volume. The name of the output Secret is determined by the value of the spec.secretName defined in the Certificate object. We need to inject some environment variables into the Spring Boot app. It requires the password to the keystores (PASSWORD), the location of the mounted trusted material inside the pod (CERT_PATH), and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert

Here’s the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Firstly, make sure you are inside the secure-callme-bundle directory. Let’s build and run the app on Kubernetes with Skaffold and enable “port-forwarding” under 8443 port:

$ skaffold dev --port-forward

Skaffold will not only run the app but also apply all required Kubernetes objects defined in the app k8s directory. It applies also to the “cert-manager” Certificate object. Once the skaffold dev command finishes successfully, we access our HTTP endpoint under the http://127.0.0.1:8443 address.

Let’s call the GET /callme endpoint. Although, we enabled the --insecure option the request failed since the web server requires client authentication. To avoid it, we should include both key and certificate files in the curl command. However,

$ curl https://localhost:8443/callme --insecure -v
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-callme-bundle
*  start date: Feb 18 20:13:00 2024 GMT
*  expire date: Feb 18 21:13:00 2024 GMT
*  issuer: CN=secure-callme-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /callme HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
* LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
* Closing connection
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0

Spring Boot SSL Hot Reload with RestTemplate

Sample App Implementation

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the keystore and truststore taken from Secret generated for the server-side app. With those files, the RestTemplate bean can authenticate against the secure-callme-bundle app. Of course, we also need to automatically reload the SslBundle bean after a certificate rotation.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
  client:
    reload-on-update: true
    keystore:
      location: ${CLIENT_CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CLIENT_CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Spring Boot 3.1 with the bundles’ concept extremely simplifies SSL context configuration for Spring REST clients like RestTemplate or WebClient. However, currently (Spring Boot 3.2.2) there is no built-in implementation for reloading e.g. Spring RestTemplate on the SslBundle update. Therefore we need to add a portion of code to achieve that. Fortunately, SslBundles allows us to define a custom handler that fires on the bundle update event. We need to define the handler for the client bundle. Once it receives a rotated version of SslBundle, it replaces the existing RestTemplate bean in the context with a new one using RestTemplateBuilder.

@SpringBootApplication
public class SecureCallerBundle {

   private static final Logger LOG = LoggerFactory
      .getLogger(SecureCallerBundle.class);

   public static void main(String[] args) {
      SpringApplication.run(SecureCallerBundle.class, args);
   }

   @Autowired
   ApplicationContext context;

   @Bean("restTemplate")
   RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
      sslBundles.addBundleUpdateHandler("client", sslBundle -> {
         try {
            LOG.info("Bundle updated: " + sslBundle.getStores().getKeyStore().getCertificate("certificate"));
         } catch (KeyStoreException e) {
            LOG.error("Error on getting certificate", e);
         }
         DefaultSingletonBeanRegistry registry = (DefaultSingletonBeanRegistry) context
            .getAutowireCapableBeanFactory();
         registry.destroySingleton("restTemplate");
         registry.registerSingleton("restTemplate", 
            builder.setSslBundle(sslBundle).build());
      });
      return builder.setSslBundle(sslBundles.getBundle("client")).build();
   }
}

Deploy on Kubernetes

Let’s take a look at the Kubernetes Deployment manifest for the current app. This time, we are mounting two secrets as volumes. The first one is generated for the current app web server, while the second one is generated for the secure-callme-bundle app and is used by the RestTemplate in establishing secure communication. We also set the address of the target service to inject it into the app (HOST) and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme-bundle
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Let’s deploy the app with the skaffold dev --port-forward command. Once again, it will deploy all the required staff on Kubernetes. Since we already exposed the secure-callme-bundle app with the “port-forward” option, the current app is exposed under the 8444 port.

Let’s try to call the GET /caller endpoint. Under the hood, it calls the endpoint exposed by the secure-callme-bundle app with RestTemplate. As you see, the secure communication is successfully established.

curl https://localhost:8444/caller --insecure -v
*   Trying [::1]:8444...
* Connected to localhost (::1) port 8444
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-caller-bundle
*  start date: Feb 18 20:40:11 2024 GMT
*  expire date: Feb 18 21:40:11 2024 GMT
*  issuer: CN=secure-caller-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /caller HTTP/1.1
> Host: localhost:8444
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 57
< Date: Sun, 18 Feb 2024 21:26:42 GMT
<
* Connection #0 to host localhost left intact
I'm `secure-caller`! calling... I'm secure-callme-bundle!

Now, we can wait one hour until the “cert-manager” rotates the certificate inside the secure-callme-cert Secret. However, we can also remove the secret, since “cert-manager” will regenerate it based on the Certificate object. Here’s the secret with certificates and keystores used to establish secure communication between both our sample Spring Boot apps.

No matter if you wait until the 1h rotation occurs or do it manually by removing the secret, you should see the following log inside the secure-callme-bundle app pod. It means that Spring Boot has received the SslBundle update event and then reloaded a Tomcat server.

The SslBundle event is also handled on the secure-caller-bundle app side. It refreshes the RestTemplate bean and prints information with the latest certificate in the logs.

Final Thoughts

The latest releases of Spring Boot simplify the management of SSL certificates on the both server and client sides a lot. Thanks to SslBundles we can easily handle the certificate rotation process without restarting the pod on Kubernetes. There are still some other things to consider don’t covered by this article. It includes the mechanism of distributing the trust bundles across the apps. However, for example, to manage trust bundles in the Kubernetes environment we can use the “cert-manager” trust-manager feature.

The post Spring Boot SSL Hot Reload on Kubernetes appeared first on Piotr's TechBlog.