In order to generate SSL certificates on Kubernetes we will use cert-manager. “Cert-manager” can rotate certificates after a specified period and save them as Kubernetes Secrets. I have already described how to implement a similar scenario with an automatic restart of a pod on a secret update in the article here. We were using the Stakater Reloader tool to restart the pod automatically on a new version of Secret. However, this time we use Spring Boot features to avoid having to restart an app (pod).

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

How It Works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need to design a solution for enabling SSL/TLS communication between the services running on Kubernetes. This solution must take into account a scenario of certificates reloading. Moreover, it must happen at the same time for the both server and client sides to avoid errors in the communication. On the server side, we use an embedded Tomcat server. In the client-side app, we use the Spring RestTemplate object.

“Cert-manager” can generate certificates automatically, based on the provided CRD object. It ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It serves all the required staff as the Kubernetes Secret. Such a secret is then mounted as a volume into the app pod. Thanks to that we don’t need to restart a pod, to see the latest certificates or “keystores” inside the pod. Here is the visualization of the described architecture.

Install cert-manager on Kubernetes

In order to install both “cert-manager” on Kubernetes we will use its Helm chart. We don’t need any specific settings. Before installing the chart we have to add CRD resources for the latest version 1.14.2:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart in the cert-manager namespace using the following command:

$ helm install my-release cert-manager jetstack/cert-manager \
    -n cert-manager

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0 

That’s all. Now we can proceed to the implementation.

Spring Boot SSL Hot Reload on the Embedded Server

Sample App Implementation

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our main goal is to enable HTTPS for that app and make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure SSL trust material for the web server (3). There are two types of trusted material it can support. In order to configure bundles using Java keystore files, we have to use the spring.ssl.bundle.jks group. On the other hand, it is possible to configure bundles using PEM-encoded text files with the spring.ssl.bundle.pem properties group.

In the exercise, we will use the Java keystore files (JKS). We define a single SSL bundle under the server name. It contains both keystore and truststore locations. With the reload-on-update property, we can instruct Spring Boot to watch the files in the background and trigger a web server reload if they change. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property (2). Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property. Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl:
  client-auth: NEED
  bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Generate Certificates with Cert-manager

Before we deploy the callme-secure-bundle app on Kubernetes, we need to configure “cert-manager” and generate the required certificates. Firstly, we need to define the CRD object responsible for issuing certificates. Here’s the ClusterIssuer object that generates self-signed certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-cluster-issuer
spec:
  selfSigned: {}

Here’s the Kubernetes Secret with the password used for securing generated keystores:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

After that, we can generate certificates. Here’s the Certificate object for the app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer, which has been created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme-bundle, so the cert needs to have that name as CN. In order to enable certificate rotation, we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration “cert-manager” will automatically renew a certificate (5). However, it won’t rotate the private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme-bundle
    - localhost
  secretName: secure-callme-cert
  commonName: secure-callme-bundle
  duration: 1h
  renewBefore: 5m

Deploy on Kubernetes

After creating a certificate we can proceed to the secure-callme-bundle app deployment. It mounts the Secret containing certificates and keystores as a volume. The name of the output Secret is determined by the value of the spec.secretName defined in the Certificate object. We need to inject some environment variables into the Spring Boot app. It requires the password to the keystores (PASSWORD), the location of the mounted trusted material inside the pod (CERT_PATH), and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert

Here’s the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Firstly, make sure you are inside the secure-callme-bundle directory. Let’s build and run the app on Kubernetes with Skaffold and enable “port-forwarding” under 8443 port:

$ skaffold dev --port-forward

Skaffold will not only run the app but also apply all required Kubernetes objects defined in the app k8s directory. It applies also to the “cert-manager” Certificate object. Once the skaffold dev command finishes successfully, we access our HTTP endpoint under the http://127.0.0.1:8443 address.

Let’s call the GET /callme endpoint. Although, we enabled the --insecure option the request failed since the web server requires client authentication. To avoid it, we should include both key and certificate files in the curl command. However,

$ curl https://localhost:8443/callme --insecure -v
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-callme-bundle
*  start date: Feb 18 20:13:00 2024 GMT
*  expire date: Feb 18 21:13:00 2024 GMT
*  issuer: CN=secure-callme-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /callme HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
* LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
* Closing connection
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0

Spring Boot SSL Hot Reload with RestTemplate

Sample App Implementation

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the keystore and truststore taken from Secret generated for the server-side app. With those files, the RestTemplate bean can authenticate against the secure-callme-bundle app. Of course, we also need to automatically reload the SslBundle bean after a certificate rotation.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
  client:
    reload-on-update: true
    keystore:
      location: ${CLIENT_CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CLIENT_CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Spring Boot 3.1 with the bundles’ concept extremely simplifies SSL context configuration for Spring REST clients like RestTemplate or WebClient. However, currently (Spring Boot 3.2.2) there is no built-in implementation for reloading e.g. Spring RestTemplate on the SslBundle update. Therefore we need to add a portion of code to achieve that. Fortunately, SslBundles allows us to define a custom handler that fires on the bundle update event. We need to define the handler for the client bundle. Once it receives a rotated version of SslBundle, it replaces the existing RestTemplate bean in the context with a new one using RestTemplateBuilder.

@SpringBootApplication
public class SecureCallerBundle {

   private static final Logger LOG = LoggerFactory
      .getLogger(SecureCallerBundle.class);

   public static void main(String[] args) {
      SpringApplication.run(SecureCallerBundle.class, args);
   }

   @Autowired
   ApplicationContext context;

   @Bean("restTemplate")
   RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
      sslBundles.addBundleUpdateHandler("client", sslBundle -> {
         try {
            LOG.info("Bundle updated: " + sslBundle.getStores().getKeyStore().getCertificate("certificate"));
         } catch (KeyStoreException e) {
            LOG.error("Error on getting certificate", e);
         }
         DefaultSingletonBeanRegistry registry = (DefaultSingletonBeanRegistry) context
            .getAutowireCapableBeanFactory();
         registry.destroySingleton("restTemplate");
         registry.registerSingleton("restTemplate", 
            builder.setSslBundle(sslBundle).build());
      });
      return builder.setSslBundle(sslBundles.getBundle("client")).build();
   }
}

Deploy on Kubernetes

Let’s take a look at the Kubernetes Deployment manifest for the current app. This time, we are mounting two secrets as volumes. The first one is generated for the current app web server, while the second one is generated for the secure-callme-bundle app and is used by the RestTemplate in establishing secure communication. We also set the address of the target service to inject it into the app (HOST) and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme-bundle
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Let’s deploy the app with the skaffold dev --port-forward command. Once again, it will deploy all the required staff on Kubernetes. Since we already exposed the secure-callme-bundle app with the “port-forward” option, the current app is exposed under the 8444 port.

Let’s try to call the GET /caller endpoint. Under the hood, it calls the endpoint exposed by the secure-callme-bundle app with RestTemplate. As you see, the secure communication is successfully established.

curl https://localhost:8444/caller --insecure -v
*   Trying [::1]:8444...
* Connected to localhost (::1) port 8444
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-caller-bundle
*  start date: Feb 18 20:40:11 2024 GMT
*  expire date: Feb 18 21:40:11 2024 GMT
*  issuer: CN=secure-caller-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /caller HTTP/1.1
> Host: localhost:8444
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 57
< Date: Sun, 18 Feb 2024 21:26:42 GMT
<
* Connection #0 to host localhost left intact
I'm `secure-caller`! calling... I'm secure-callme-bundle!

Now, we can wait one hour until the “cert-manager” rotates the certificate inside the secure-callme-cert Secret. However, we can also remove the secret, since “cert-manager” will regenerate it based on the Certificate object. Here’s the secret with certificates and keystores used to establish secure communication between both our sample Spring Boot apps.

No matter if you wait until the 1h rotation occurs or do it manually by removing the secret, you should see the following log inside the secure-callme-bundle app pod. It means that Spring Boot has received the SslBundle update event and then reloaded a Tomcat server.

The SslBundle event is also handled on the secure-caller-bundle app side. It refreshes the RestTemplate bean and prints information with the latest certificate in the logs.

Final Thoughts

The latest releases of Spring Boot simplify the management of SSL certificates on the both server and client sides a lot. Thanks to SslBundles we can easily handle the certificate rotation process without restarting the pod on Kubernetes. There are still some other things to consider don’t covered by this article. It includes the mechanism of distributing the trust bundles across the apps. However, for example, to manage trust bundles in the Kubernetes environment we can use the “cert-manager” trust-manager feature.

The post Spring Boot SSL Hot Reload on Kubernetes appeared first on Piotr's TechBlog.

Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme and secure-caller. After that, you should just follow my instructions. Let’s begin.

How it works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.

Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap or Secret. Then it performs a rolling upgrade on pods, which use the particular ConfigMap or Secret. Here is the visualization of the described architecture.

Prerequisites

Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:

1. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
2. Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager

Install Cert Manager and Stakater Reloader

In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart using the following command:

$ helm install my-release cert-manager jetstack/cert-manager

The same with the Stakater Reloader – first we need to add the stakater charts repository:

$ helm repo add stakater https://stakater.github.io/stakater-charts

Then, we can install the latest version of the chart:

$ helm install my-reloader stakater/reloader

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0          1m
my-reloader-reloader-7566fdc68c-qj9l4         1/1     Running   0          1m

That’s all. Now we can proceed to the implementation.

HTTPS with Spring Boot

Our first app secure-callme exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property. Here’s the configuration for our Spring Boot inside the application.yml file.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

We will set the values of CERT_PATH and PASSWORD at the level of Kubernetes Deployment. Now, let’s switch to the secure-caller implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient dependency.

<dependency>
  <groupId>org.apache.httpcomponents.client5</groupId>
  <artifactId>httpclient5</artifactId>
</dependency>

Now, we will use Apache HttpClient as a low-level client for the Spring RestTemplate. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate @Bean we use RestTemplateBuilder.

@SpringBootApplication
public class SecureCaller {

   public static void main(String[] args) {
      SpringApplication.run(SecureCaller.class, args);
   }

   @Autowired
   ClientSSLProperties clientSSLProperties;

   @Bean
   RestTemplate builder(RestTemplateBuilder builder) throws 
      GeneralSecurityException, IOException {

      final SSLContext sslContext = new SSLContextBuilder()
         .loadTrustMaterial(
            new File(clientSSLProperties.getTrustStore()),
            clientSSLProperties.getTrustStorePassword().toCharArray())
         .loadKeyMaterial(
            new File(clientSSLProperties.getKeyStore()),
            clientSSLProperties.getKeyStorePassword().toCharArray(),
            clientSSLProperties.getKeyStorePassword().toCharArray()
         )
         .build();

      final SSLConnectionSocketFactory sslSocketFactory = 
         SSLConnectionSocketFactoryBuilder.create()
                .setSslContext(sslContext)
                .build();

      final HttpClientConnectionManager cm = 
         PoolingHttpClientConnectionManagerBuilder.create()
                .setSSLSocketFactory(sslSocketFactory)
                .build();

      final HttpClient httpClient = HttpClients.custom()
         .setConnectionManager(cm)
         .evictExpiredConnections()
         .build();

      return builder
         .requestFactory(() -> 
            new HttpComponentsClientHttpRequestFactory(httpClient))
         .build();
   }
}

The client credentials are taken from configuration settings under the client.ssl key. Here is the @ConfigurationProperties class used the RestTemplateBuilder in the previous step.

@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {

   private String keyStore;
   private String keyStorePassword;
   private String trustStore;
   private String trustStorePassword;

   // GETTERS AND SETTERS ...

}

Here’s the configuration for the secure-caller inside application.yml file. The same as for the secure-callme we expose the REST endpoint over HTTPS.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

client.url: https://${HOST}:8443/callme
client.ssl:
  key-store: ${CLIENT_CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CLIENT_CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}

The secure-caller app calls GET /callme exposed by the secure-callme app using customized RestTemplate.

@RestController
public class SecureCallerController {

   RestTemplate restTemplate;

   @Value("${client.url}")
   String clientUrl;

   public SecureCallerController(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @GetMapping("/caller")
   public String call() {
      return "I'm `secure-caller`! calling... " +
             restTemplate.getForObject(clientUrl, String.class);
   }

}

Generate and Renew Certificates on Kubernetes with Cert Manager

With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-clusterissuer
spec:
  selfSigned: {}

After that, we can generate certificates. Here’s the cert-manager Certificate object for the secure-callme app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy to Always.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores: # (1)
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef: # (2)
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme
  secretName: secure-callme-cert
  commonName: secure-callme # (3)
  duration: 1h # (4)
  renewBefore: 5m  # (5)

The Certificate object for secure-caller is very similar. The only difference is in the CN field. We will use the port-forward option during the test, so I’ll set the domain name to localhost (1).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-caller-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - localhost
    - secure-caller
  secretName: secure-caller-cert
  commonName: localhost # (1)
  duration: 1h
  renewBefore: 5m

After applying both manifests we can display a list of Certificates. Each of them is related to the Secret with the same name:

$ kubectl get certificate
NAME                 READY   SECRET               AGE
secure-caller-cert   True    secure-caller-cert   1m
secure-callme-cert   True    secure-callme-cert   1m

Here are the details of the secure-callme-cert Secret. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store and server.ssl.key-store properties). There is also a certificate (tls.crt), a private key (tls.key), and CA (ca.crt).

$ kubectl describe secret secure-callme-cert
Name:         secure-callme-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: secure-callme
              cert-manager.io/certificate-name: secure-callme-cert
              cert-manager.io/common-name: secure-callme
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: ss-clusterissuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:          550 bytes
keystore.jks:    1029 bytes
tls.crt:         550 bytes
tls.key:         227 bytes
truststore.jks:  422 bytes

Deploy and Reload Apps on Kubernetes

Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme app.

First of all, we need to reload the app each time the secure-callme-cert changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload (1). The annotation should contain the name of Secret, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH env variable (2). We are mounting the whole secure-callme-cert Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme
  annotations:
    # (1)
    secret.reloader.stakater.com/reload: "secure-callme-cert" 
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme
    spec:
      containers:
        - image: piomin/secure-callme
          name: secure-callme
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH # (2)
              value: /opt/secret
          volumeMounts:
            - mountPath: /opt/secret # (3)
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert # (4)

The password to the key store files is available inside the jks-password-secret:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

There is also the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme
  name: secure-callme
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme
  type: ClusterIP

Now, go to the secure-callme directory and just run the following command:

$ skaffold run

The Deployment manifest of the secure-caller app is a little bit more complicated. The same as before we need to reload the app on Secret change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller
  # (1)
  annotations:
    secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller
    spec:
      containers:
        - image: piomin/secure-caller
          name: secure-caller
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            # (2)
            - name: CERT_PATH
              value: /opt/secret
            # (3)
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        # (5)
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Then, go to the secure-caller directory and deploy the app. This time we enable port-forward to easily test the app locally.

$ skaffold run --port-forward

Let’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.

$ kubectl get deploy 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
my-cert-manager              1/1     1            1           1h
my-cert-manager-cainjector   1/1     1            1           1h
my-cert-manager-webhook      1/1     1            1           1h
my-reloader-reloader         1/1     1            1           1h
secure-caller                1/1     1            1           1h
secure-callme                1/1     1            1           1h

Testing Renew of Certificates on Kubernetes

The secure-caller app is available on the 8443 port locally, while the secure-callme app is available inside the cluster under the service name secure-callme. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.key}' | base64 --decode > tls.key

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.crt}' | base64 --decode > tls.crt

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.ca\.crt}' | base64 --decode > ca.crt

Now, we can call the GET /caller endpoint using the following curl command. Under the hood, the secure-caller calls the endpoint GET /callme exposed by the secure-callme also over HTTPS. If you did everything according to the instruction you should have the same result as below.

$ curl https://localhost:8443/caller \
  --key tls.key \
  --cert tls.crt \
  --cacert ca.crt 
I'm `secure-caller`! calling... I'm `secure-callme`!

Our certificate is valid for one hour.

Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.

Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.

Final Thoughts

Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.

The post Renew Certificates on Kubernetes with Cert Manager and Reloader appeared first on Piotr's TechBlog.

There are several topics you should be familiar with before start reading the following article. At least, it is worth reading about the basics related to service mesh on Kubernetes with Istio and Spring Boot here. You may also be interested in the article about security best practices for Spring Boot apps.

Prerequisites

Before we begin we need to prepare a test cluster. Personally, I’m using a local OpenShift cluster that simplifies the installation of several tools we need. But you can as well use any other Kubernetes distribution like minikube or kind. No matter which version you choose, you need to install at least:

1. Istio – you can do it in several ways, here’s an instruction with Istio CLI
2. Cert Manager – the simplest way to install it with the kubectl
3. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
4. Helm – used to install additional tools on Kubernetes (or OpenShift)

Since I’m using OpenShift I can install everything using the UI console and Operator Hub. Here’s the list of my tools:

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the tls branch. After that, you should just follow my instructions. Let’s begin.

Generate Certificates with Cert Manager

You can as well generate TLS/SSL certs using e.g. openssl and then apply them on Kubernetes. However, Cert Manager simplifies that process. It automatically creates a Kubernetes Secret with all the required staff. Before we create a certificate we should configure a default ClusterIssuer. I’m using the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

In the next step, we are going to create the Certificate object. It is important to set the commonName parameter properly. It should be the same as the hostname of our gateway. Since I’m running the local instance of OpenShift the default domain suffix is apps-crc.testing. The name of the application is sample-spring-kotlin in that case. We will also need to generate a keystore and truststore for configuring 2-way SSL in the second scenario with the passthrough gateway. Finally, we should have the same Kubernetes Secret with a certificate placed in the namespace with the app (demo-apps) and the namespace with Istio workloads (istio-system in our case). We can sync secrets across namespaces in several different ways. I’ll use the built-in feature of Cert Manager based on the secretTemplate field and a tool called reflector. Here’s the final version of the Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sample-spring-boot-cert
  namespace: demo-apps
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: selfsigned-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - sample-spring-kotlin.apps-crc.testing
  secretName: sample-spring-boot-cert
  commonName: sample-spring-kotlin.apps-crc.testing
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"  
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "istio-system"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "istio-system"

Before we apply the Certificate object we need to create the Secret with a keystore password:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
  namespace: demo-apps
data:
  password: MTIzNDU2
type: Opaque

Of course, we also need to install the reflector tool on our Kubernetes cluster. We can easily do it using the following Helm commands:

$ helm repo add emberstack https://emberstack.github.io/helm-charts
$ helm repo update
$ helm upgrade --install reflector emberstack/reflector

Here’s the final result. There is the Secret sample-spring-kotlin-cert in the demo-apps namespace. It contains a TLS certificate, private key, CA, JKS keystore, and truststore. You can also verify that the same Secret is available in the istio-system namespace.

Create Istio Gateway with Mutual TLS

Let’s begin with our first scenario. In order to create a gateway with mTLS, we should set MUTUAL as a mode and set the name of the Secret containing the certificate and private key. The name of the Istio Gateway host is sample-spring-kotlin.apps-crc.testing. Gateway is available on Kubernetes under the default HTTPS port.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: MUTUAL
        credentialName: sample-spring-boot-cert

The Spring Boot application is available under the HTTP port. Therefore we should create a standard Istio VirtualService that refers to the already created gateway.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  http:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8080

Finally, we can run the Spring Boot app on Kubernetes using skaffold. We need to activate the istio-mutual profile that exposes the HTTP (8080) port.

$ skaffold dev -p istio-mutual

OpenShift Service Mesh automatically exposes Istio Gateway as the Route object. Let’s just verify if everything works fine. Assuming that the app has started successfully we can display a list of Istio gateways in the demo-apps namespace:

$ kubectl get gw -n demo-apps
NAME                    AGE
microservices-gateway   3m4s

Then let’s display a list of Istio virtual services in the same namespace:

$ kubectl get vs -n demo-apps
NAME                           GATEWAYS                    HOSTS                                       AGE
sample-spring-boot-vs-via-gw   ["microservices-gateway"]   ["sample-spring-kotlin.apps-crc.testing"]   4m2s

If you are running it on OpenShift you should also check the Route object in the istio-system namespace:

$ oc get route -n istio-system | grep sample-spring-kotlin

If you test it on Kubernetes you just need to set the Host header on your request. Here’s the curl command for testing our secure gateway. Since we enabled mutual TLS auth we need to provide the client key and certificate. We can copy them to the local machine from Kubernetes Secret generated by the Cert Manager.

Let’s call the REST endpoint GET /persons exposed by our sample Spring Boot app:

$ curl -v https://sample-spring-kotlin.apps-crc.testing/persons \
    --key tls.key \
    --cert tls.crt

You will probably receive the following response:

Ok, we forgot to add our CA to the trusted certificates. Let’s do that. Alternatively, we can set the parameter --cacert on the curl command.

Now, we can run again the same curl command as before. The secure communication with our app should work perfectly fine. We can proceed to the second scenario.

Mutual Auth for Spring Boot and Passthrough Istio Gateway

Let’s proceed to the second scenario. Now, our sample Spring Boot is exposing the HTTPS port with the client cert verification. In order to enable it on the app side, we need to provide some configuration settings.

Here’s our application.yml file. Firstly, we need to enable SSL and set the default port to 8443. It is important to force client certificate authentication with the server.ssl.client-auth property. As a result, we also need to provide a truststore file location. Finally, we don’t want to expose Spring Boot Actuator endpoint over SSL, so we force to expose them under the default plain port.

server.port: 8443
server.ssl.enabled: true
server.ssl.key-store: /opt/secret/keystore.jks
server.ssl.key-store-password: ${PASSWORD}

server.ssl.trust-store: /opt/secret/truststore.jks
server.ssl.trust-store-password: ${PASSWORD}
server.ssl.client-auth: NEED

management.server.port: 8080
management.server.ssl.enabled: false

In the next step, we need to inject both keystore.jks and truststore.jks into the app container. Therefore we need to modify Deployment to mount a secret generated by the Cert Manager as a volume. Once again, the name of that Secret is sample-spring-boot-cert. The content of that Secret would be available for the app under the /opt/secret directory. Of course, we also need to expose the port 8443 outside and inject a secret with the keystone and truststore password.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin-microservice
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: sample-spring-kotlin-microservice
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app.kubernetes.io/name: sample-spring-kotlin-microservice
    spec:
      containers:
      - image: piomin/sample-spring-kotlin
        name: sample-spring-kotlin-microservice
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8443
          name: https
        env:
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: jks-password-secret
        volumeMounts:
          - mountPath: /opt/secret
            name: sample-spring-boot-cert
      volumes:
        - name: sample-spring-boot-cert
          secret:
            secretName: sample-spring-boot-cert

Here’s the definition of our Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: PASSTHROUGH

The definition of the Istio VirtualService is slightly different than before. It contains the TLS route with the name of the host to match SNI.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  tls:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8443
          weight: 100
      match:
        - port: 443
          sniHosts:
            - sample-spring-kotlin.apps-crc.testing

Finally, we can deploy the current version of the app using the following skaffold command to enable HTTPS port for the app running on Kubernetes:

$ skaffold dev -p istio-passthrough

Now, you can repeat the same steps as in the previous section to verify that the current configuration works fine.

Final Thoughts

In this article, I showed how to use some interesting tools that simplify the configuration of HTTPS for apps running on Kubernetes. You could see how Istio, Cert Manager, or reflector can work together. We considered two variants of making secure HTTPS Istio gateways for the Spring Boot application.

The post HTTPS on Kubernetes with Spring Boot, Istio and Cert Manager appeared first on Piotr's TechBlog.