In our example, Keycloak will act as an Identity Provider. Keycloak supports SAML 2.0. We can also use Spring Security mechanisms supporting SAML authentication on the service provider side (our sample Spring Boot application). There are several articles about Spring Boot and SAML, but relatively few of them are up to date and use Keycloak as the IdP.

If you are interested in Keycloak and Spring Security you can read my article about microservices with Spring Cloud Gateway, OAuth2, and Keycloak. You can also take a look at the another post about best practices for securing Spring Boot microservices available here.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. First, you need to clone the following GitHub repository. It contains several sample Java applications for a Spring Boot security showcase. You must go to the saml directory, to proceed with exercise. The sample Spring Boot aplication is available in the callme-saml directory. Then you should follow my further instructions.

Prerequisites

Before we start the development, we must install some tools on our laptops. Of course, we should have Maven and at least Java 17 installed (Java 21 preferred). We must also have access to the container engine like Docker or Podman to run the Keycloak instance. 

Run Keycloak

We will run Keycloak as the Docker container. The repository contains the docker-compose.yml file in the saml directory and the realm manifest in the saml/config directory. Docker Compose run Keycloak in the development mode and imports the realm file on startup. Thanks to that you won’t have to create many resources in Keycloak by yourself. However, I’m describing step by step what should be done. The docker-compose.yml manifest also set the default administrator password to admin, enables HTTPS, and uses the Red Hat build of Keycloak instead of the community edition.

services:
  keycloak:
#    image: quay.io/keycloak/keycloak:24.0
    image: registry.redhat.io/rhbk/keycloak-rhel9:24-17
    environment:
      - KEYCLOAK_ADMIN=admin
      - KEYCLOAK_ADMIN_PASSWORD=admin
      - KC_HTTP_ENABLED=true
      - KC_HTTPS_CERTIFICATE_KEY_FILE=/opt/keycloak/conf/localhost.key.pem
      - KC_HTTPS_CERTIFICATE_FILE=/opt/keycloak/conf/localhost.crt.pem
      - KC_HTTPS_TRUST_STORE_FILE=/opt/keycloak/conf/truststore.jks
      - KC_HTTPS_TRUST_STORE_PASSWORD=123456
    ports:
      - 8080:8080
      - 8443:8443
    volumes:
      - /Users/pminkows/IdeaProjects/sample-spring-security-microservices/oauth/localhost-key.pem:/opt/keycloak/conf/localhost.key.pem
      - /Users/pminkows/IdeaProjects/sample-spring-security-microservices/oauth/localhost-crt.pem:/opt/keycloak/conf/localhost.crt.pem
      - /Users/pminkows/IdeaProjects/sample-spring-security-microservices/oauth/truststore.jks:/opt/keycloak/conf/truststore.jks
      - ./config/:/opt/keycloak/data/import:ro
    command: start-dev --import-realm

Finally, we must run the following command to start the instance of Keycloak.

docker compose up

Then, we should have the running instance of Keycloak exposes on the localhost over the HTTPS 8443 port.

After logging to the Keycloak UI with the admin / admin crendentials you should see the spring-boot-keycloak realm.

In the spring-boot-keycloak realm details there is a link to the SAML2 provider metadata file. We can download the file and pass directly to the Spring Boot application or save the link address for the future use. Keycloak published the IdP metadata for the spring-boot-keycloak realm under the https://localhost:8443/realms/spring-boot-keycloak/protocol/saml/descriptor link. Let’s copy that address to clipboard and proceed to the Spring Boot app implementation.

Create Spring Boot App with SAML2 Support

Let’s begin with the dependencies list. We must include the Spring Web and Spring Security modules. For the SAML2 support we must add the spring-security-saml2-service-provider dependency. That dependency uses the OpenSAML library, which is published in the dedicated Shibboleth Maven repository. Our application also requires Thymeleaf to provide a single web page with an authenticated principal details.

<dependencies>
  <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-security</artifactId>
  </dependency>
  <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-thymeleaf</artifactId>
  </dependency>
  <dependency>
     <groupId>org.thymeleaf.extras</groupId>
     <artifactId>thymeleaf-extras-springsecurity6</artifactId>
  </dependency>
  <dependency>
     <groupId>org.springframework.security</groupId>
     <artifactId>spring-security-saml2-service-provider</artifactId>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>shibboleth-build-releases</id>
    <name>Shibboleth Build Releases Repository</name>
    <url>https://build.shibboleth.net/nexus/content/repositories/releases/</url>
  </repository>
</repositories>

Our goal is to start with something basic. The application will publish a metadata endpoint using the saml2Metadata DSL method. We also enable authorization at the method level with the @EnableMethodSecurity annotation. We can access the resources the after authentication in Keycloak.

@Configuration
@EnableWebSecurity
@EnableMethodSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http.csrf(AbstractHttpConfigurer::disable)
                .authorizeHttpRequests(authorize -> authorize.anyRequest()
                        .authenticated())
                .saml2Login(withDefaults())
                .saml2Metadata(withDefaults());
        return http.build();
    }

}

Here’s the Spring Boot application.yml file. It changes the default app HTTP port to the 8081. The SAML2 configuration must be provided within the spring.security.saml2.relyingparty.registration.{registrationId}.* properties. We must set the address of the IdP metadata file in the assertingparty.metadata-uri property. We also need to set the entity-id and the SSO service address. Both those settings are exposed in the metadata IdP file. We should also provided certificates for signing requests and verifying SAML responses. The key and certificate all already present in the repository.

server.port: 8081

spring:
  security:
    saml2:
      relyingparty:
        registration:
          keycloak:
            identityprovider:
              entity-id: https://localhost:8443/realms/spring-boot-keycloak
              verification.credentials:
                - certificate-location: classpath:rp-certificate.crt
              singlesignon.url: https://localhost:8443/realms/spring-boot-keycloak/protocol/saml
              singlesignon.sign-request: false
            signing:
              credentials:
                - private-key-location: classpath:rp-key.key
                  certificate-location: classpath:rp-certificate.crt
            assertingparty:
              metadata-uri: https://localhost:8443/realms/spring-boot-keycloak/protocol/saml/descriptor

Lets’ run our application with the following command:

mvn spring-boot:run

After startup, we can display a metadata endpoint exposed with the saml2Metadata DSL method. The most important element in the file is the entityID. Let’s save it for the future usage.

<md:EntityDescriptor entityID="http://localhost:8081/saml2/service-provider-metadata/keycloak">
  <md:SPSSODescriptor protocolSupportEnumeration="urn:oasis:names:tc:SAML:2.0:protocol">
    <md:KeyDescriptor use="signing">
      <ds:KeyInfo>
        <ds:X509Data>
          <ds:X509Certificate>...</ds:X509Certificate>
        </ds:X509Data>
      </ds:KeyInfo>
    </md:KeyDescriptor>
    <md:AssertionConsumerService Binding="urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST" Location="http://localhost:8081/login/saml2/sso/keycloak" index="1"/>
  </md:SPSSODescriptor>
</md:EntityDescriptor>

Configure Keycloak SAML2 Client

Let’s switch to the Keycloak UI. We must create a SAML client. The client ID must be the same as the entityID retrieved in the previous section. We should also set the application root URL and valid redirects URIs. Notice that you don’t have do anything – Keycloak imports all requeired configuration at startup from the exported realm manifest.

Don’t forget to create a test user with password. We will use to autenticate against Keycloak in the step.

This is the optional step. We can add the client scope with some mappers. This force Keycloak to pass information about user group, email, and surname in the authentication response.

Let’s include the newly created scope to the client scopes.

We can also create the admins group and assign our test user to that group.

After providing the whole configuration we can make a first test. Our application is already running. It will print the authenticated user details on the main site after signing in. Here’s the MainController class.

@Controller
public class MainController {

    @GetMapping("/")
    public String getPrincipal(@AuthenticationPrincipal Saml2AuthenticatedPrincipal principal, Model model) {
        String emailAddress = principal.getFirstAttribute("email");
        model.addAttribute("emailAddress", emailAddress);
        model.addAttribute("userAttributes", principal.getAttributes());
        return "index";
    }

}

Here’s the main application site. It uses the Thymeleaf extension for Spring Security.

<!doctype html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:th="https://www.thymeleaf.org"
      xmlns:sec="https://www.thymeleaf.org/thymeleaf-extras-springsecurity6">

<head>
    <title>SAML 2.0 Login</title>
</head>

<body>
    <main role="main" class="container">
        <h1 class="mt-5">SAML 2.0 Login with Spring Security</h1>
        <p class="lead">You are successfully logged in as <span sec:authentication="name"></span></p>
        <h2 class="mt-2">User Identity Attributes</h2>
        <table class='table table-striped'>
            <thead>
            <tr>
                <th>Attribute</th>
                <th>Value</th>
            </tr>
            </thead>
            <tbody>
            <tr th:each="userAttribute : ${userAttributes}">
                <th th:text="${userAttribute.key}"></th>
                <td th:text="${userAttribute.value}"></td>
            </tr>
            </tbody>
        </table>
    </main>
</body>

</html>

Once we access the application main site over the http://localhost:8081 URL, Spring should redirect us to the Keycloak login site. Let’s provide our test user credentials.

Success! We are logged in to the application. Our Spring Boot app prints the details taken from SAML2 authentication token.

REST Methods Authorization

Let’s consider the following @RestController in our application. We have already enabled authorization at the method level with the @EnableMethodSecurity annotation. Our controller creates two REST endpoints. The GET /greetings/user endpoint requires the ROLE_USER authority granted, while the GET /greetings/admin requires the ROLE_ADMIN authority granted.

@RestController
@RequestMapping("/greetings")
public class GreetingController {

    @GetMapping("/user")
    @PreAuthorize("hasAuthority('ROLE_USER')")
    public String greeting() {
        return "I'm SAML user!";
    }

    @GetMapping("/admin")
    @PreAuthorize("hasAuthority('ROLE_ADMINS')")
    public String admin() {
        return "I'm SAML admin!";
    }

}

Let’s call the GET /greetings/user endpoint. It returns the expected response.

Then, let’s call the GET /greetings/admin endpoint. Unfortunately, we don’t have the access to that endpoint. That’s because it requires the user to have the ROLE_ADMINS authority.

Previosuly, we used a default implementation the SAML2 authentication provider in Spring Boot. It sets only the ROLE_USER as the granted authorities. We can override that behaviour with the setResponseAuthenticationConverter method SAML2 AuthenticationProvider implementation. Here’s the current implementation. Our method tries to find the member attrobute in the SAML2 response token. Then, it maps the group name taken from that attribute to the Spring Security GrantedAuthority name. Finally, it returns a new Saml2Authentication object with a new list of GrantedAuthority objects.

@Configuration
@EnableWebSecurity
@EnableMethodSecurity
public class SecurityConfig {

    private static final Logger LOG = LoggerFactory
        .getLogger(SecurityConfig.class);

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {

        OpenSaml4AuthenticationProvider provider = 
            new OpenSaml4AuthenticationProvider();

        provider.setResponseAuthenticationConverter(token -> {
            var auth = OpenSaml4AuthenticationProvider
                .createDefaultResponseAuthenticationConverter()
                .convert(token);
            LOG.info("AUTHORITIES: {}", auth.getAuthorities());

            var attrValues = token.getResponse().getAssertions().stream()
                    .flatMap(as -> as.getAttributeStatements().stream())
                    .flatMap(attrs -> attrs.getAttributes().stream())
                        .filter(attrs -> attrs.getName().equals("member"))
                        .findFirst().orElseThrow().getAttributeValues();

            if (!attrValues.isEmpty()) {
                var member = ((XSStringImpl) attrValues.getFirst()).getValue();
                LOG.info("MEMBER: {}", member);
                List<GrantedAuthority> authoritiesList = List.of(
                        new SimpleGrantedAuthority("ROLE_USER"),
                        new SimpleGrantedAuthority("ROLE_" + 
                            member.toUpperCase().replaceFirst("/", ""))
                );

                LOG.info("NEW AUTHORITIES: {}", authoritiesList);
                return new Saml2Authentication(
                    (AuthenticatedPrincipal) auth.getPrincipal(), 
                    auth.getSaml2Response(), 
                    authoritiesList);
            } else return auth;
        });

        http.csrf(AbstractHttpConfigurer::disable)
                .authorizeHttpRequests(authorize -> authorize.anyRequest()
                        .authenticated())
                .saml2Login(saml2 -> saml2
                        .authenticationManager(new ProviderManager(provider))
                )
                .saml2Metadata(withDefaults());
        return http.build();
    }

}

Let’s run a new version of our application.

mvn spring-boot:run

Once we open the http://localhost:8081 site once again we can take a look at the logs. I highlighted the part of the SAML response message that contains the member attribute. As you on the bottom, we created a new granted authorities list containing ROLE_USER and ROLE_ADMINS.

Fibnally, we can call the GET /greetings/admin endpoint once again. It works!

Final Thoughts

This article shows how to simply start with Spring Boot, SAML2 and Keycloak. It also provides more advanced solution with the custom authentication provider implementation that maps a user group name to the granted authority name. Although SAML2 is rather a mature technology, there are no many examples with Spring Boot, SAML2 and Keycloak. I hope that my article will fill this gap

The post Spring Boot with SAML2 and Keycloak appeared first on Piotr's TechBlog.

It is the first article on my blog after a long holiday break. I hope you enjoy it. If you are interested in Spring Boot you can find several other posts about it on my blog. I can recommend my article about another fresh Spring Boot feature related to security, that describes SSL certs hot reload on Kubernetes.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. Today you will have to clone two sample Git repositories. The first one contains automatically updated source code of microservices based on the latest version of the Spring Boot framework. The second repository contains an archived version of microservices based on the earlier, unsupported version of Spring Boot. Once you clone both of these repositories, you just need to follow my instructions.

By the way, you can verify SBOMs generated for your Spring Boot apps in various ways. I decided to use snyk CLI for that. Alternatively, you can use the web version of the Snyk SBOM checker available here. In order to install the snyk CLI on your machine you need to follow its documentation. I used homebrew to install it on my macOS:

$ brew tap snyk/tap
$ brew install snyk

Enable SBOM Support in Spring Boot

By default, Spring Boot supports the CycloneDX format for generating SBOMs. In order to enable it, we need to include the cyclonedx-maven-plugin Maven plugin in your project root pom.xml.

<build>
  <plugins>
    <plugin>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-maven-plugin</artifactId>
    </plugin>
    <plugin>
      <groupId>org.cyclonedx</groupId>
      <artifactId>cyclonedx-maven-plugin</artifactId>
    </plugin>
  </plugins>
</build>

There are several microservices defined in the same Git repository. They are all using the same root pom.xml. Each of them defines its list of dependencies. For this exercise, we need to have at least the Spring Boot Web and Actuator starters. However, let’s take a look at the whole list of dependencies for the employee-service (one of our sample microservices):

<dependencies>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-netflix-eureka-client</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-starter-config</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-zipkin</artifactId>
  </dependency>
  <dependency>
    <groupId>io.github.openfeign</groupId>
    <artifactId>feign-micrometer</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-starter-webmvc-api</artifactId>
    <version>2.6.0</version>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-test</artifactId>
    <scope>test</scope>
  </dependency>
  <dependency>
    <groupId>org.instancio</groupId>
    <artifactId>instancio-junit</artifactId>
    <version>4.8.1</version>
    <scope>test</scope>
  </dependency>
</dependencies>

After including the cyclonedx-maven-plugin plugin we need to execute the mvn package command in the repository root directory:

$ mvn clean package -DskipTests

The plugin will generate SBOM files for all the existing microservices and place them in the target/classes/META-INF/sbom directory for each Maven module.

The generated SBOM file will always be placed inside the JAR file as well. Let’s take a look at the location of the SBOM file inside the employee-service uber JAR.

In order to expose the actuator SBOM endpoint we need to include the following configuration property. Since our configuration is stored by the Spring Cloud Config server, we need to put such a property in the YAML files inside the config-service/src/main/resources/config directory.

management:
  endpoints:
    web:
      exposure:
        include: health,sbom

Then, let’s start the config-service with the following Maven command:

$ cd config-service
$ mvn clean spring-boot:run

After that, we can start our sample microservice. It loads the configuration properties from the config-service. It listens on the dynamically generated port number. For me, it is 53498. In order to see the contents of the generated SBOM file, we need to call the GET /actuator/sbom/application path.

Generate and Verify SBOMs with the Snyk CLI

The exact structure of the SBOM file is not very important from our perspective. We need a tool that allows us to verify components and dependencies published inside that file. As I mentioned before, we can use the snyk CLI for that. We will examine the file generated in the repository root directory. Here’s the snyk command that allows us to print all the detected vulnerabilities in the SBOM file:

$ snyk sbom test \
   --file=target/classes/META-INF/sbom/application.cdx.json \
   --experimental

Here’s the report created as the output of the command executed above. As you see, there are two detected issues related to the included dependencies. Of course, I’m not including those dependencies directly in the Maven pom.xml. They were automatically included by the Spring Boot starters used by the microservices. By the way, I was even not aware that Spring Boot includes kotlin-stdlib even if I’m not using any Kotlin library directly in the app.

Although there are two issues detected in the report, it doesn’t look very bad. Now, let’s try to analyze something more outdated. I have already mentioned my old repository with microservices: sample-spring-microservices. It is already in the archived status and uses Spring Boot in the 1.5 version. If we don’t want to modify anything there, we can also use snyk CLI to generate SBOM instead of the Maven plugin. Since built-in support for SBOM comes with Spring Boot 3.3 there is no sense in including a plugin for the apps with the 1.5 version. Here’s the snyk command that generates SBOM for all the projects inside the repository and exports it to the application.cdx.json file:

$ snyk sbom --format=cyclonedx1.4+json --all-projects > application.cdx.json

Then, let’s examine the SBOM file using the same command as before:

$ snyk sbom test --file=application.cdx.json --experimental

Now, the results are much more pessimistic. There are 211 detected issues, including 6 critical.

Final Thoughts

SBOMs allow organizations to identify and address potential security risks more effectively. Spring Boot support for generating SBOM files simplifies the process when incorporating them into the organizational software development life cycle.

The post SBOM with Spring Boot appeared first on Piotr's TechBlog.

There is a similar article on my blog about “cert-manager” and Spring Boot SSL hot reload more focused on Kubernetes. You can also read the post on how to integrate Spring Boot with Vault PKI to generate certs automatically.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Let’s check out the openshift branch. Then, we need to go to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

$ git clone https://github.com/piomin/sample-spring-security-microservices.git
$ git checkout openshift
$ cd ssl

How It Works

OpenShift brings a lot of interesting features to the Kubernetes platform. The “Service Serving Certificates” is one of the examples. Thanks to that mechanism, we don’t need to install any external tool for generating and rotating certificates dynamically like “cert-manager”. The OpenShift CA service automatically rotates certificates 13 months before expiration and issues new certificates for 26 months. After a rotation, the previous CA is trusted until it expires. To achieve it, OpenShift implements the root CA key rollover process described in  https://tools.ietf.org/html/rfc4210#section-4.4. This allows a grace period for all affected services to refresh their key material before the expiration. Sounds interesting? Let’s proceed to the implementation.

In order to test a mechanism, we will run two simple Spring Boot apps on OpenShift that communicate using SSL certificates. OpenShift takes care of issuing/rotating certificates and exposing the CA bundle as a ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using Service Serving Certificates”. Here’s the illustration of our apps’ architecture on OpenShift.

Create a Server Side App

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now, our main goal is to enable HTTPS for that app and make it work properly with the OpenShift “Service Serving Certificates”. Firstly, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure keystores for the web server (3). We can configure SSL bundles using PEM-encoded text files directly in the app with the spring.ssl.bundle.pem properties group. The certificate (tls.crt) and private key (tls.key) will be mounted as a volume to the pod. Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property (2). Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl.bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

The only thing we need to do to generate SSL certificates for our app is to annotate the Kubernetes Service with service.beta.openshift.io/serving-cert-secret-name. It should contain the name of the target Kubernetes Secret as a value.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Once we apply such a Service to the cluster, OpenShift generates the secure-callme-bundle Secret that contains the certificate and private key.

After that, we just need to deploy the app and mount the generated Secret as a volume. The volume path is passed to the app as the CERT_PATH environment variable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-bundle

Finally, we need to expose the HTTP endpoint outside of the cluster. We can easily do it with the OpenShift Route. Don’t forget the set the TLS termination as passthrough. Thanks to that OpenShift router sends traffic directly to the destination without TLS termination.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-callme-bundle
  labels:
    app: secure-callme-bundle
    app.kubernetes.io/component: secure-callme-bundle
    app.kubernetes.io/instance: secure-callme-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-callme-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Create a Client Side App

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the truststore taken from ConfigMap with CA certificates. With the certs inside the service-ca.crt file, the RestTemplate bean can authenticate against the secure-callme-bundle app.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key
  client:
    truststore:
      certificate: ${CLIENT_CERT_PATH}/service-ca.crt

Don’t forget about creating the RestTemplate bean. It needs to the SslBundles bean and set it on the RestTemplateBuilder.

@Bean
RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
   return builder
      .setSslBundle(sslBundles.getBundle("client")).build();
}

We need to create the ConfigMap annotated with service.beta.openshift.io/inject-cabundle=true. After that, the OpenShift cluster automatically injects the service CA certificate into the service-ca.crt key on the config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-caller-bundle
  namespace: demo-secure
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
data: {}

Let’s display the details of the secure-caller-bundle ConfigMap. As you see, OpenShift injects the PEM-encoded CA certificates under the service-ca.crt key to that config map.

Finally, we can deploy our client-side app. Here’s the YAML manifest with the Deployment object. Since the secure-caller-bundle app acts as a web server and also uses a REST client to invoke the secure-callme-bundle we need to mount two volumes. The first one (/opt/secret) contains a certificate and private key for the web server (1). Consequently, the second one (/opt/client-cert) holds the CA bundle used by the RestTemplate client (2). This client communicates with secure-callme-bundle internally through the Kubernetes Service (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-cert
            # (3)
            - name: HOST
              value: secure-callme-bundle.demo-secure.svc
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            # (1)
            - mountPath: /opt/secret
              name: cert
            # (2)
            - mountPath: /opt/client-cert
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-bundle
        - name: client-cert
          configMap:
            name: secure-caller-bundle

Of course, we should also create a passthrough Route to expose our app the outside of the cluster:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-caller-bundle
  labels:
    app: secure-caller-bundle
    app.kubernetes.io/component: secure-caller-bundle
    app.kubernetes.io/instance: secure-caller-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-caller-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Testing OpenShift Certs SSL Rotation with Spring Boot

The main problem with testing the certs rotation scenario is in the 13-month interval. We can’t change that period in the configuration. However, there is a way how we can force OpenShift to rotate the CA certs on demand. In order to do that, we need to set the spec.unsupportedConfigOverrides.forceRotation.reason property on the cluster-wide ServiceCA object with any value. Of course, you shouldn’t do that at production, since it is not supported by OpenShift. It is just designed for testing or development purposes.

apiVersion: operator.openshift.io/v1
kind: ServiceCA
metadata:
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  unsupportedConfigOverrides:
    forceRotation:
      reason: testing

We will call our apps through the OpenShift Route. So, let’s display a current list of routes related to our sample apps:

$ oc get route
NAME                   HOST/PORT                                                     PATH   SERVICES               PORT    TERMINATION   WILDCARD
secure-caller-bundle   secure-caller-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-caller-bundle   https   passthrough   None
secure-callme-bundle   secure-callme-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-callme-bundle   https   passthrough   None

Once we set that property, OpenShift immediately rotates that CA certificate. It will also take care of synchronizing the latest certificates with all the secrets and config maps containing annotations required by the “Service Serving Certificates” mechanism. As you see, now, our secure-caller-bundle ConfigMap contains two CAs: an old one and a new one.

$ oc describe cm secure-caller-bundle
Name:         secure-caller-bundle
Namespace:    demo-secure
Labels:       <none>
Annotations:  service.beta.openshift.io/inject-cabundle: true

Data
====
service-ca.crt:
----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIITnr+EqYsYtEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAzMDExMDM5MTVaFw0yNjA0MzAxMDM5MTZaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDN0No0DIZ+BCdPvQ9TIvEjwGUSoUXM+DYI
Dxlk2Ef6goFv3YsAgNJTxznntRc69dJTrLtRrmFN0d31JPM/gJOMkKMvTWnz8Qzx
athuvvwbivJNKh/qn+Dhewbjrx5LGpq3v9VQ7t/5Rf5F9VMpyp738EwfAy2cfxTA
sxgYpvU/foivM9U0uMaZyXA5vLsGN6cKpEREGMqKbIKSXNYXD9/lbHv2eyr+5+s9
keYEDkGTWMceISihm5mGNwOZpLZhNwFgSZR9O63TIgGc/TEZ3EYwcIjmlVOSwQ/u
sM43HL1jtYKw3AovQOc5qc2kT5eSPV7jVCCfNYnj3BLvt0AYUXE/AgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT3iI1f
71ZVt6OVLC/+0oxNL7BqhzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEATcT/LK5pAaOGz0j3MR4NhsfmAjo4K8SoGmFE
F4lDzDWZTaJuJHCHqpblgGKLfokU9BQC3L4NZLIisOj3CWF3zIpC6MOsU7x9qDm0
aZpI1gqhRXQZOqVpE9XJ0XBEOLP9hlzxa66SQlALlU5CWbDNEMtXsMMQf1lSgIRz
arWK8QwmPsmg6duoVtFcCMY2klkVacZctiMgM4wvf1CCJt3TkiggwO3t4IdiV8Xa
Z9d/LlV/bFzboxGxOb2IMHj5VXKPM6HmC4slBSqKeyr7PYyM7rjhorIOOner9JrX
RhbC8/uARciLbcFKyOOCSLoe5m1i8QPrRmCm/GhVS9M4x86TXw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIFlMUWK4HjW4wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAyMjgxMTUyMjNaFw0yNjA0MjgxMTUyMjRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY6AZbLN0kT2BPXbDn7iZmbyYZgyI8Y3jh
hRq9JrX0ZIIs5BvFcRRkm2IcJ8gsMF4VztGCkNoDQ07ojAIHI/FxssAU8wwoz2QP
eH2qzBpLA1lBAYvtjki//55STJvJF+7Z6qbcDwUVX6/r+hrpy5MIjaQmRV4DRLpp
ZD3HrqYozJEvCKaA2pinC2VyW8IBDB0FDvPdBxvzjrCDjssr/v0jqFgUGFJJmKEj
EaFQACeMlOS4Q5avEglfwoLS+RGgPjE1s1gpxir2dZCdnM8b9B3CFy2sS+SSHN4y
+FJRJBv4CbMAMP+DLIsHY872c2XmOlleJNFpiFWi9r33LCc7MnbjAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRey/oP
FWW5kG8ifSXqIXBh2L+FdzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEAhbimn2Tv5w9Cy7i6dNdYYdVhA52cCnWWDDhf
s/dZ+Zl1aP3f179XO0xz3ovLuE227Z9AYPGtPW6c0FQgxgDgl2pYUX4J88caTUG/
eR9EG9LEfrL6OlKMjMvAFWrL64bcAAiuoz/gWB0a29xMnUsA52+7OpnzmHMduaaz
5JfRaTwkjGGe8ay2XsTUNrNGqZJGKHhE+hrUVlFsS7GpaSEe9GllkS9lGUZp2Hei
lTxwnBr9j+aP/QWXwuAVxT1DUAByJYoTPrpMhk+hKQUj/8TvxmMmsksZ57steqRI
/gWYd4j2SohN3jd1mv6rQ+jsGGAfCNmYVZL1oevNaqfnENt0IA==
-----END CERTIFICATE-----

However, if we don’t restart the pods, our apps still use the old keystore and truststore (unless they have the reload-on-update option enabled).

Here’s a list of running pods.

oc get po
NAME                                    READY   STATUS    RESTARTS   AGE
secure-caller-bundle-57c7658d85-bj4q5   1/1     Running   0          2d23h
secure-callme-bundle-cbf59569f-n6gg5    1/1     Running   0          2d23h

Let’s just restart the secure-callme-bundle app by deleting its pod. Do not restart the secure-callme-bundle pod, because we want to test the scenario where it uses the old truststore.

$ oc delete po secure-callme-bundle-cbf59569f-n6gg5
pod "secure-callme-bundle-cbf59569f-n6gg5" deleted

Then, we can call the GET /callme endpoint exposed by the secure-callme-bundle app through the Route once again. As you see, it servers the latest, reloaded certificate.

Let’s perform a final test. Now, we call the GET /caller endpoint exposed by the secure-caller-bundle app. Under the hood, it uses RestTemplate to call the GET /callme endpoint exposed by the secure-callme-bundle app via secure-callme-bundle.demo-secure.svc address. As you see the communication still works fine although the secure-caller-bundle app still uses the keystore and truststore.

Final Thoughts

The OpenShift “Service Serving Certificates” mechanism is simple to configure and doesn’t require that you handle the process of reloading apps with the rotated trusted material. It also has some drawbacks. You cannot change the rotation period from 13 months into a different value, and there is a global CA bundle shared across all the services running on the cluster. If those things are problematic for you I advise you to use “cert-manager” and a tool that allows you to restart pods automatically on the Secret or ConfigMap change. It is also possible to leverage the Spring SslBundles to reload certs without restarting the pod. With Spring Boot apps you can also use OpenShift Service Mesh, which can handle SSL communication at the platform level.

The post Rotate SSL Certificates with OpenShift and Spring Boot appeared first on Piotr's TechBlog.

Our architecture consists of two Spring Boot microservices, an API gateway built on top of Spring Cloud Gateway, and a Keycloak authorization server. Spring Cloud Gateway acts here as an OAuth2 Client and OAuth2 Resource Server. For any incoming request, it verifies an access token before forwarding traffic to the downstream services. It initializes an authorization code flow procedure with Keycloak for any unauthenticated request. Our scenario needs to include the communication between internal microservices. They are both hidden behind the API gateway. The caller app invokes an endpoint exposed by the callme app. The HTTP client used in that communication has to use the access token sent by the gateway.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the oauth directory. You will find two Spring Boot microservices there: callme and caller. Of course, there is the gateway app built on top of Spring Cloud Gateway. After that, you should just follow my instructions. Let’s begin.

Run and Configure Keycloak

We are running Keycloak as a Docker container. By default, Keycloak exposes API and a web console on the port 8080. We also need to set an admin username and password with environment variables. Here’s the command used to run the Keycloak container:

$ docker run -d --name keycloak -p 8080:8080 \
    -e KEYCLOAK_ADMIN=admin \
    -e KEYCLOAK_ADMIN_PASSWORD=admin \
    quay.io/keycloak/keycloak:23.0.7 start-dev

Once the container starts, we can go to the UI admin console available under the http://localhost:8080/admin address. We will create a new realm. The name is that realm is demo. Instead of creating the required things manually, we can import the JSON resource file that contains the whole configuration of the realm. You can find such a resource file in my GitHub repository here: oauth/gateway/src/test/resources/realm-export.json. However, in the next parts of that section, we will use the Keycloak dashboard to create objects step by step. In case you import the configuration from the JSON resource file, you can just skip to the next section.

Then, we need to add a single OpenID Connect client to the demo realm. The name of our client is spring-with-test-scope. We should enable client authentication and put the right address in the “Valid redirect URIs” field (it can be the wildcard for testing purposes).

We need to save the name of the client and its secret. Those two settings have to be set on the application side.

Then, let’s create a new client scope with the TEST name.

Then, we have to add the TEST to the spring-with-test-scope client scopes.

We also need to create a user to authenticate against Keycloak. The name of our user is spring. In order to set the password, we need to switch to the “Credentials” tab. For my user, I choose the Spring_123 password.

Once we finish with the configuration, we can export it to the JSON file (the same file we can use when creating a new realm). Such a file will be useful later, for building automated tests with Testcontainers.

Unfortunately, Keycloak doesn’t export realm users to the file. Therefore, we need to add the following JSON to the users section in the exported file.

{
  "username": "spring",
  "email": "piotr.minkowski@gmail.com",
  "firstName": "Piotr",
  "lastName": "Minkowski",
  "enabled": true,
  "credentials": [
    {
      "type": "password",
      "value": "Spring_123"
    }
  ],
  "realmRoles": [
    "default-roles-demo",
    "USER"
  ]
}

Create Spring Cloud Gateway with OAuth2 Support and Keycloak

As I mentioned before, our gateway app will act as an OAuth2 Client and OAuth2 Resource Server. In that case, we include both the Spring Boot Auth2 Client Starter and the spring-security-oauth2-resource-server dependency. We also need to include the spring-security-oauth2-jose to decode JWT tokens automatically. Of course, we need to include the Spring Cloud Gateway Starter. Finally, we add dependencies for automated testing with JUnit. We will use Testcontainers to run the Keycloak container during the JUnit test. It can be achieved with the com.github.dasniko:testcontainers-keycloak dependency.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-oauth2-client</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-resource-server</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-jose</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.cloud</groupId>
  <artifactId>spring-cloud-starter-gateway</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-test</artifactId>
  <scope>test</scope>
</dependency>
<dependency>
  <groupId>com.github.dasniko</groupId>
  <artifactId>testcontainers-keycloak</artifactId>
  <version>3.2.0</version>
 <scope>test</scope>
</dependency>
<dependency>
  <groupId>org.testcontainers</groupId>
  <artifactId>junit-jupiter</artifactId>
  <version>1.19.6</version>
  <scope>test</scope>
</dependency>

Let’s begin with the Spring Security configuration. First, we need to annotate the Configuration bean with @EnableWebFluxSecurity. That’s because Spring Cloud Gateway uses the reactive version of the Spring web module. The oauth2Login() method is responsible for redirecting an unauthenticated request to the Keycloak login page. On the other hand, the oauth2ResourceServer() method verifies an access token before forwarding traffic to the downstream services.

@Configuration
@EnableWebFluxSecurity
public class SecurityConfig {

    @Bean
    public SecurityWebFilterChain springSecurityFilterChain(ServerHttpSecurity http) {
        http.authorizeExchange(auth -> auth.anyExchange().authenticated())
                .oauth2Login(withDefaults())
                .oauth2ResourceServer((oauth2) -> oauth2.jwt(Customizer.withDefaults()));
        http.csrf(ServerHttpSecurity.CsrfSpec::disable);
        return http.build();
    }

}

That’s not all. We also need to provide several configuration settings with the spring.security.oauth2 prefix. The Spring OAuth2 Resource Server module will use the Keycloak JWKS endpoint to verify incoming JWT tokens. In the Spring OAuth2 Client section, we need to provide the address of the Keycloak issuer realm. Of course, we also need to provide the Keycloak client credentials, choose the authorization grant type and scope.

spring.security.oauth2:
  resourceserver:
    jwt:
      jwk-set-uri: http://localhost:8080/realms/demo/protocol/openid-connect/certs
  client:
    provider:
      keycloak:
        issuer-uri: http://localhost:8080/realms/demo
    registration:
      spring-with-test-scope:
        provider: keycloak
        client-id: spring-with-test-scope
        client-secret: IWLSnakHG8aNTWNaWuSj0a11UY4lzxd9
        authorization-grant-type: authorization_code
        scope: openid

The gateway exposes a single HTTP endpoint by itself. It uses OAuth2AuthorizedClient bean to return the current JWT access token.

@SpringBootApplication
@RestController
public class GatewayApplication {

   private static final Logger LOGGER = LoggerFactory
      .getLogger(GatewayApplication.class);

   public static void main(String[] args) {
      SpringApplication.run(GatewayApplication.class, args);
   }

   @GetMapping(value = "/token")
   public Mono<String> getHome(@RegisteredOAuth2AuthorizedClient OAuth2AuthorizedClient authorizedClient) {
      return Mono.just(authorizedClient.getAccessToken().getTokenValue());
   }

}

That’s all about OAuth2 configuration in that section. We also need to configure routing on the gateway in the Spring application.yml file. Spring Cloud Gateway can forward OAuth2 access tokens downstream to the services it is proxying using the TokenRelay GatewayFilter. It is possible to set it as a default filter for all incoming requests. Our gateway forwards traffic to both our callme and caller microservices. I’m not using any service discovery in that scenario. By default, the callme app listens on the 8040 port, while the caller app on the 8020 port.

spring:
  application:
    name: gateway
  cloud:
    gateway:
      default-filters:
        - TokenRelay=
      routes:
        - id: callme-service
          uri: http://localhost:8040
          predicates:
            - Path=/callme/**
        - id: caller-service
          uri: http://localhost:8020
          predicates:
            - Path=/caller/**

Verify Tokens in Microservices with OAuth2 Resource Server

The list of dependencies for the callme and caller is pretty similar. They are exposing HTTP endpoints using the Spring Web module. Since the caller app uses the WebClient bean we also need to include the Spring WebFlux dependency. Once again, we need to include the Spring OAuth2 Resource Server module and the spring-security-oauth2-jose dependency for decoding JWT tokens.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-resource-server</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.security</groupId>
  <artifactId>spring-security-oauth2-jose</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework</groupId>
  <artifactId>spring-webflux</artifactId>
</dependency>

Here’s the configuration of the app security. This time we need to use the @EnableWebSecurity annotation since we have a Spring Web module. The oauth2ResourceServer() method verifies an access token with the Keyclock JWKS endpoint.

@Configuration
@EnableWebSecurity
public class SecurityConfig {

    @Bean
    public SecurityFilterChain filterChain(HttpSecurity http) throws Exception {
        http.authorizeHttpRequests(authorize -> authorize.anyRequest().authenticated())
                .oauth2ResourceServer((oauth2) -> oauth2.jwt(Customizer.withDefaults()));
        return http.build();
    }
}

Here’s the OAuth2 Resource Server configuration for Keycloak in the Spring application.yml file:

spring:
  security:
    oauth2:
      resourceserver:
        jwt:
          jwk-set-uri: http://localhost:8080/realms/demo/protocol/openid-connect/certs

Let’s take a look at the implementation of the REST controller class. It is a single ping method. That method may be accessed only by the client with the TEST scope. It returns a list of assigned scopes taken from the Authentication bean.

@RestController
@RequestMapping("/callme")
public class CallmeController {

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        SecurityContext context = SecurityContextHolder.getContext();
        Authentication authentication = context.getAuthentication();
        return "Scopes: " + authentication.getAuthorities();
    }
}

This method can be invoked directly by the external client through the API gateway. However, also the caller app calls that endpoint inside its own “ping” endpoint implementation.

@RestController
@RequestMapping("/caller")
public class CallerController {

    private WebClient webClient;

    public CallerController(WebClient webClient) {
        this.webClient = webClient;
    }

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        SecurityContext context = SecurityContextHolder.getContext();
        Authentication authentication = context.getAuthentication();

        String scopes = webClient
                .get()
                .uri("http://localhost:8040/callme/ping")
                .retrieve()
                .bodyToMono(String.class)
                .block();
        return "Callme scopes: " + scopes;
    }
}

If the WebClient calls the endpoint exposed by the second microservice, it also has to propagate the bearer token. We can easily achieve it with the ServletBearerExchangeFilterFunction as shown below. Thanks to that Spring Security will look up the current Authentication and extract the AbstractOAuth2Token credential. Then, it will propagate that token in the Authorization header automatically.

@SpringBootApplication
public class CallerApplication {

    public static void main(String[] args) {
        SpringApplication.run(CallerApplication.class, args);
    }

    @Bean
    public WebClient webClient() {
        return WebClient.builder()
                .filter(new ServletBearerExchangeFilterFunction())
                .build();
    }
    
}

Testing with Running Applications

We can run all three Spring Boot apps using the same Maven command. Let’s begin with the gateway app:

$ cd oauth/gateway
$ mvn spring-boot:run

Once we run the first app, we can check out the logs if everything works fine. Here are the logs generated by the gateway app. As you see, it listens on the 8060 port.

After that, we can run e.g. the caller app.

$ cd oauth/caller
$ mvn spring-boot:run

It listens on the 8020 port.

Of course, the order of starting apps doesn’t matter. As the last one, we can run the callme app.

$ cd oauth/callme
$ mvn spring-boot:run

Now, let’s call the caller app endpoint through the gateway. In that case, we need to go to the http://localhost:8060/caller/ping URL. The gateway app will redirect us to the Keycloak login page. We need to sign in there with the spring user and Spring_123 password.

After we sign in, everything happens automatically. Spring Cloud Gateway obtains the access token from Keycloak and then sends it to the downstream service. Once the caller app receives the request, it invokes the callme app using the WebClient instance. Here’s the result:

We can easily get the access token using the endpoint GET /token exposed by the gateway app.

Now, we can perform a similar call as before, but with the curl command. We need to copy the token string and put it inside the Authorization header as a bearer token.

$ curl http://localhost:8060/callme/ping \
    -H "Authorization: Bearer <TOKEN>" -v

Here’s my result:

Now, let’s do a similar thing, but in a fully automated way with JUnit and Testcontainers.

Spring OAuth2 with Keycloak Testcontainers

We need to switch to the gateway module once again. We will implement tests that run the API gateway app, connect it to the Keycloak instance, and route the authorized traffic in the target endpoint. Here’s the @RestController in the src/test/java directory that simulates the callme app endpoint:

@RestController
@RequestMapping("/callme")
public class CallmeController {

    @PreAuthorize("hasAuthority('SCOPE_TEST')")
    @GetMapping("/ping")
    public String ping() {
        return "Hello!";
    }
}

Here’s the required configuration to run the tests. We are starting the gateway app on the 8060 port and using the WebTestClient instance for calling it. In order to automatically configure Keycloak we will import the demo realm configuration stored in the realm-export.json. Since Testcontainers use random port numbers we need to override some Spring OAuth2 configuration settings. We also override the Spring Cloud Gateway route, to forward the traffic to our test implementation of the callme app controller instead of the real service. That’s all. We can proceed to the tests implementation.

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class GatewayApplicationTests {

   static String accessToken;

   @Autowired
   WebTestClient webTestClient;

   @Container
   static KeycloakContainer keycloak = new KeycloakContainer()
            .withRealmImportFile("realm-export.json")
            .withExposedPorts(8080);

   @DynamicPropertySource
   static void registerResourceServerIssuerProperty(DynamicPropertyRegistry registry) {
      registry.add("spring.security.oauth2.client.provider.keycloak.issuer-uri",
                () -> keycloak.getAuthServerUrl() + "/realms/demo");
      registry.add("spring.security.oauth2.resourceserver.jwt.jwk-set-uri",
                () -> keycloak.getAuthServerUrl() + "/realms/demo/protocol/openid-connect/certs");
      registry.add("spring.cloud.gateway.routes[0].uri",
                () -> "http://localhost:8060");
      registry.add("spring.cloud.gateway.routes[0].id", () -> "callme-service");
      registry.add("spring.cloud.gateway.routes[0].predicates[0]", () -> "Path=/callme/**");
   }

   // TEST IMPLEMENTATION ...

}

Here’s our first test. Since it doesn’t contain any token it should be redirected into the Keycloak authorization mechanism.

@Test
@Order(1)
void shouldBeRedirectedToLoginPage() {
   webTestClient.get().uri("/callme/ping")
             .exchange()
             .expectStatus().is3xxRedirection();
}

In the second test, we use the WebClient instance to interact with the Keycloak container. We need to authenticate against Kecloak with the spring user and the spring-with-test-scope client. Keycloak will generate and return an access token. We will save its value for the next test.

@Test
@Order(2)
void shouldObtainAccessToken() throws URISyntaxException {
   URI authorizationURI = new URIBuilder(keycloak.getAuthServerUrl() + "/realms/demo/protocol/openid-connect/token").build();
   WebClient webclient = WebClient.builder().build();
   MultiValueMap<String, String> formData = new LinkedMultiValueMap<>();
   formData.put("grant_type", Collections.singletonList("password"));
   formData.put("client_id", Collections.singletonList("spring-with-test-scope"));
   formData.put("username", Collections.singletonList("spring"));
   formData.put("password", Collections.singletonList("Spring_123"));

   String result = webclient.post()
                .uri(authorizationURI)
                .contentType(MediaType.APPLICATION_FORM_URLENCODED)
                .body(BodyInserters.fromFormData(formData))
                .retrieve()
                .bodyToMono(String.class)
                .block();
   JacksonJsonParser jsonParser = new JacksonJsonParser();
   accessToken = jsonParser.parseMap(result)
                .get("access_token")
                .toString();
   assertNotNull(accessToken);
}

Finally, we run a similar test as in the first step. However, this time, we provide an access token inside the Authorization header. The expected response is 200 OK and the “Hello!” payload, which is returned by the test instance of the CallmeController bean.

@Test
@Order(3)
void shouldReturnToken() {
   webTestClient.get().uri("/callme/ping")
                .header("Authorization", "Bearer " + accessToken)
                .exchange()
                .expectStatus().is2xxSuccessful()
                .expectBody(String.class).isEqualTo("Hello!");
}

Let’s run all the tests locally. As you see, they are all successfully finished.

Final Thoughts

After publishing my previous article about Spring Cloud Gateway and Keycloak I received a lot of comments and questions with a request for some clarifications. I hope that this article answers some of them. We focused more on automation and service-to-service communication than just on the OAuth2 support in the Spring Cloud Gateway. We considered a case where a gateway acts as the OAuth2 client and resource server at the same time. Finally, we used Testcontainers to verify our scenario with Spring Cloud Gateway and Keycloak.

The post Microservices with Spring Cloud Gateway, OAuth2 and Keycloak appeared first on Piotr's TechBlog.

In order to generate SSL certificates on Kubernetes we will use cert-manager. “Cert-manager” can rotate certificates after a specified period and save them as Kubernetes Secrets. I have already described how to implement a similar scenario with an automatic restart of a pod on a secret update in the article here. We were using the Stakater Reloader tool to restart the pod automatically on a new version of Secret. However, this time we use Spring Boot features to avoid having to restart an app (pod).

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

How It Works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need to design a solution for enabling SSL/TLS communication between the services running on Kubernetes. This solution must take into account a scenario of certificates reloading. Moreover, it must happen at the same time for the both server and client sides to avoid errors in the communication. On the server side, we use an embedded Tomcat server. In the client-side app, we use the Spring RestTemplate object.

“Cert-manager” can generate certificates automatically, based on the provided CRD object. It ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It serves all the required staff as the Kubernetes Secret. Such a secret is then mounted as a volume into the app pod. Thanks to that we don’t need to restart a pod, to see the latest certificates or “keystores” inside the pod. Here is the visualization of the described architecture.

Install cert-manager on Kubernetes

In order to install both “cert-manager” on Kubernetes we will use its Helm chart. We don’t need any specific settings. Before installing the chart we have to add CRD resources for the latest version 1.14.2:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.14.2/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart in the cert-manager namespace using the following command:

$ helm install my-release cert-manager jetstack/cert-manager \
    -n cert-manager

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0 

That’s all. Now we can proceed to the implementation.

Spring Boot SSL Hot Reload on the Embedded Server

Sample App Implementation

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our main goal is to enable HTTPS for that app and make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure SSL trust material for the web server (3). There are two types of trusted material it can support. In order to configure bundles using Java keystore files, we have to use the spring.ssl.bundle.jks group. On the other hand, it is possible to configure bundles using PEM-encoded text files with the spring.ssl.bundle.pem properties group.

In the exercise, we will use the Java keystore files (JKS). We define a single SSL bundle under the server name. It contains both keystore and truststore locations. With the reload-on-update property, we can instruct Spring Boot to watch the files in the background and trigger a web server reload if they change. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property (2). Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property. Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl:
  client-auth: NEED
  bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Generate Certificates with Cert-manager

Before we deploy the callme-secure-bundle app on Kubernetes, we need to configure “cert-manager” and generate the required certificates. Firstly, we need to define the CRD object responsible for issuing certificates. Here’s the ClusterIssuer object that generates self-signed certificates.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-cluster-issuer
spec:
  selfSigned: {}

Here’s the Kubernetes Secret with the password used for securing generated keystores:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

After that, we can generate certificates. Here’s the Certificate object for the app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer, which has been created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme-bundle, so the cert needs to have that name as CN. In order to enable certificate rotation, we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration “cert-manager” will automatically renew a certificate (5). However, it won’t rotate the private key.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme-bundle
    - localhost
  secretName: secure-callme-cert
  commonName: secure-callme-bundle
  duration: 1h
  renewBefore: 5m

Deploy on Kubernetes

After creating a certificate we can proceed to the secure-callme-bundle app deployment. It mounts the Secret containing certificates and keystores as a volume. The name of the output Secret is determined by the value of the spec.secretName defined in the Certificate object. We need to inject some environment variables into the Spring Boot app. It requires the password to the keystores (PASSWORD), the location of the mounted trusted material inside the pod (CERT_PATH), and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert

Here’s the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Firstly, make sure you are inside the secure-callme-bundle directory. Let’s build and run the app on Kubernetes with Skaffold and enable “port-forwarding” under 8443 port:

$ skaffold dev --port-forward

Skaffold will not only run the app but also apply all required Kubernetes objects defined in the app k8s directory. It applies also to the “cert-manager” Certificate object. Once the skaffold dev command finishes successfully, we access our HTTP endpoint under the http://127.0.0.1:8443 address.

Let’s call the GET /callme endpoint. Although, we enabled the --insecure option the request failed since the web server requires client authentication. To avoid it, we should include both key and certificate files in the curl command. However,

$ curl https://localhost:8443/callme --insecure -v
*   Trying [::1]:8443...
* Connected to localhost (::1) port 8443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-callme-bundle
*  start date: Feb 18 20:13:00 2024 GMT
*  expire date: Feb 18 21:13:00 2024 GMT
*  issuer: CN=secure-callme-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /callme HTTP/1.1
> Host: localhost:8443
> User-Agent: curl/8.4.0
> Accept: */*
>
* LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0
* Closing connection
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C412:SSL routines:ST_OK:sslv3 alert bad certificate, errno 0

Spring Boot SSL Hot Reload with RestTemplate

Sample App Implementation

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the keystore and truststore taken from Secret generated for the server-side app. With those files, the RestTemplate bean can authenticate against the secure-callme-bundle app. Of course, we also need to automatically reload the SslBundle bean after a certificate rotation.

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.jks:
  server:
    reload-on-update: true
    keystore:
      location: ${CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
  client:
    reload-on-update: true
    keystore:
      location: ${CLIENT_CERT_PATH}/keystore.jks
      password: ${PASSWORD}
      type: JKS
    truststore:
      location: ${CLIENT_CERT_PATH}/truststore.jks
      password: ${PASSWORD}
      type: JKS

Spring Boot 3.1 with the bundles’ concept extremely simplifies SSL context configuration for Spring REST clients like RestTemplate or WebClient. However, currently (Spring Boot 3.2.2) there is no built-in implementation for reloading e.g. Spring RestTemplate on the SslBundle update. Therefore we need to add a portion of code to achieve that. Fortunately, SslBundles allows us to define a custom handler that fires on the bundle update event. We need to define the handler for the client bundle. Once it receives a rotated version of SslBundle, it replaces the existing RestTemplate bean in the context with a new one using RestTemplateBuilder.

@SpringBootApplication
public class SecureCallerBundle {

   private static final Logger LOG = LoggerFactory
      .getLogger(SecureCallerBundle.class);

   public static void main(String[] args) {
      SpringApplication.run(SecureCallerBundle.class, args);
   }

   @Autowired
   ApplicationContext context;

   @Bean("restTemplate")
   RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
      sslBundles.addBundleUpdateHandler("client", sslBundle -> {
         try {
            LOG.info("Bundle updated: " + sslBundle.getStores().getKeyStore().getCertificate("certificate"));
         } catch (KeyStoreException e) {
            LOG.error("Error on getting certificate", e);
         }
         DefaultSingletonBeanRegistry registry = (DefaultSingletonBeanRegistry) context
            .getAutowireCapableBeanFactory();
         registry.destroySingleton("restTemplate");
         registry.registerSingleton("restTemplate", 
            builder.setSslBundle(sslBundle).build());
      });
      return builder.setSslBundle(sslBundles.getBundle("client")).build();
   }
}

Deploy on Kubernetes

Let’s take a look at the Kubernetes Deployment manifest for the current app. This time, we are mounting two secrets as volumes. The first one is generated for the current app web server, while the second one is generated for the secure-callme-bundle app and is used by the RestTemplate in establishing secure communication. We also set the address of the target service to inject it into the app (HOST) and activate the prod profile (SPRING_PROFILES_ACTIVE).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme-bundle
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Let’s deploy the app with the skaffold dev --port-forward command. Once again, it will deploy all the required staff on Kubernetes. Since we already exposed the secure-callme-bundle app with the “port-forward” option, the current app is exposed under the 8444 port.

Let’s try to call the GET /caller endpoint. Under the hood, it calls the endpoint exposed by the secure-callme-bundle app with RestTemplate. As you see, the secure communication is successfully established.

curl https://localhost:8444/caller --insecure -v
*   Trying [::1]:8444...
* Connected to localhost (::1) port 8444
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server did not agree on a protocol. Uses default.
* Server certificate:
*  subject: CN=secure-caller-bundle
*  start date: Feb 18 20:40:11 2024 GMT
*  expire date: Feb 18 21:40:11 2024 GMT
*  issuer: CN=secure-caller-bundle
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* using HTTP/1.x
> GET /caller HTTP/1.1
> Host: localhost:8444
> User-Agent: curl/8.4.0
> Accept: */*
>
< HTTP/1.1 200
< Content-Type: text/plain;charset=UTF-8
< Content-Length: 57
< Date: Sun, 18 Feb 2024 21:26:42 GMT
<
* Connection #0 to host localhost left intact
I'm `secure-caller`! calling... I'm secure-callme-bundle!

Now, we can wait one hour until the “cert-manager” rotates the certificate inside the secure-callme-cert Secret. However, we can also remove the secret, since “cert-manager” will regenerate it based on the Certificate object. Here’s the secret with certificates and keystores used to establish secure communication between both our sample Spring Boot apps.

No matter if you wait until the 1h rotation occurs or do it manually by removing the secret, you should see the following log inside the secure-callme-bundle app pod. It means that Spring Boot has received the SslBundle update event and then reloaded a Tomcat server.

The SslBundle event is also handled on the secure-caller-bundle app side. It refreshes the RestTemplate bean and prints information with the latest certificate in the logs.

Final Thoughts

The latest releases of Spring Boot simplify the management of SSL certificates on the both server and client sides a lot. Thanks to SslBundles we can easily handle the certificate rotation process without restarting the pod on Kubernetes. There are still some other things to consider don’t covered by this article. It includes the mechanism of distributing the trust bundles across the apps. However, for example, to manage trust bundles in the Kubernetes environment we can use the “cert-manager” trust-manager feature.

The post Spring Boot SSL Hot Reload on Kubernetes appeared first on Piotr's TechBlog.

The solution presented in this article is not the only way how we can deal with HashiCorp Vault on Kubernetes. If you are interested in other approaches you may refer to some of my previous articles. In that article, you can find a guide on how to integrate Vault secrets with Argo CD through the plugin. If you are running Spring Boot apps on Kubernetes you can also be interested in Spring Cloud Vault support. In that case please refer to the following article.

How it works

I guess you may not be very familiar with the Container Storage Interface (CSI) pattern. At the high-level CSI is a standard for exposing block or file storage to the containers. It is implemented by different storage providers.

The Secrets Store CSI Driver is running on Kubernetes as a DeamonSet. It is interacting with every instance of Kubelet on the Kubernetes nodes. Once the pod is starting, the Secrets Store CSI Driver communicates with the external secrets provider to retrieve the secret content. The following diagram illustrates how Secrets Store CSI Driver works on Kubernetes.

It provides the SecretProviderClass CRD to manage that process. In this provider class, we need to set the secure vault address and the location of the secret keys. Here’s the SecretProviderClass for our scenario. We will use HashiCorp Vault running on Kubernetes.

apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: vault-database
  namespace: default
spec:
  parameters:
    objects: |-
      - objectName: "db-password"
        secretPath: "secret/data/db-pass"
        secretKey: "password"
    roleName: webapp
    vaultAddress: 'http://vault.vault.svc:8200'
  provider: vault

Here’s the location of our secret in the HashiCorp Vault. As you see the current value of the password entry is test1.

Source Code

As usual – if you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my two GitHub repositories. The first of them contains Terraform scripts for installing Vault and Secrets Store CSI. After cloning it you go to the vault-ocp directory. The second repository contains a simple Spring Boot app for a test scenario. Once you clone it, go to the spring-util-app directory. Then you should just follow my instructions

Install Vault and Secrets Store CSI Driver with Terraform

As I mentioned before, we will use Terraform to set up almost the whole test scenario today. We will just leverage Skaffold, in the last step, to deploy the Spring Boot app on the Kubernetes cluster.

In order to install both Vault and Secrets Store CSI Driver we will use Helm charts. To do that part of the exercise, we need kubernetes (1) and helm (2) as the Terraform providers. The third step (3) is required only if you run the scenario on OpenShift. It changes the default service account access restrictions and security context constraints (SCCs) to ensure that a pod has sufficient permissions to start on OpenShift. Then we may proceed to the Helm charts installation. For the Secrets Store CSI Driver chart (4), it is important to enable secrets rotation since we will test this feature at the end of the article. Finally, we can install the HashiCorp Vault chart (5).

# (1)
provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

# (2)
provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

# (3)
resource "kubernetes_cluster_role_binding" "privileged" {
  metadata {
    name = "system:openshift:scc:privileged"
  }
  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = "system:openshift:scc:privileged"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "secrets-store-csi-driver"
    namespace = "k8s-secrets-store-csi"
  }
  subject {
    kind      = "ServiceAccount"
    name      = "vault-csi-provider"
    namespace = "vault"
  }
}

resource "kubernetes_namespace" "vault" {
  metadata {
    name = "vault"
  }
}

resource "kubernetes_service_account" "vault-sa" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name      = "vault"
    namespace = "vault"
  }
}

resource "kubernetes_secret_v1" "vault-secret" {
  depends_on = [kubernetes_namespace.vault]
  metadata {
    name = "vault-token"
    namespace = "vault"
    annotations = {
      "kubernetes.io/service-account.name" = "vault"
    }
  }

  type = "kubernetes.io/service-account-token"
}

# (4)
resource "helm_release" "secrets-store-csi-driver" {
  chart            = "secrets-store-csi-driver"
  name             = "csi-secrets-store"
  namespace        = "k8s-secrets-store-csi"
  create_namespace = true
  repository       = "https://kubernetes-sigs.github.io/secrets-store-csi-driver/charts"

  set {
    name  = "linux.providersDir"
    value = "/var/run/secrets-store-csi-providers"
  }

  set {
    name  = "syncSecret.enabled"
    value = "true"
  }

  set {
    name  = "enableSecretRotation"
    value = "true"
  }
}

# (5)
resource "helm_release" "vault" {
  chart            = "vault"
  name             = "vault"
  namespace        = "vault"
  create_namespace = true
  repository       = "https://helm.releases.hashicorp.com"

  values = [
    file("values.yaml")
  ]
}

The Helm values.yaml file used by the Vault chart is visible below. In order to simplify deployment, we will enable the development mode (1). It generates a root token automatically and runs a single instance of Vault. We can enable Route for OpenShift (2) and use the image supported by Red Hat (3). Of course, we also need to enable global OpenShift configuration (4). You can omit all three steps (2) (3) (4) when running the scenario on vanilla Kubernetes. Finally, we need to enable CSI support (5) and disable the Vault sidecar injector which is not needed in that exercise (7). The path in the csi.deamonSet.providersDir property should be the same as the linux.providersDir in the Halm chart params (6).

server:
  dev:
    enabled: true # (1)
  route: # (2)
    enabled: true
    host: ""
    tls: null
  image: # (3)
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.12.4-ubi"
  serviceAccount:
    name: vault
    create: false
global: # (4)
  openshift: true
csi: 
  debug: true
  enabled: true # (5)
  daemonSet:
    providersDir: /var/run/secrets-store-csi-providers # (6)
    securityContext:
      container:
        privileged: true
injector: # (7)
  enabled: false

Finally, let’s apply the configuration to the target cluster.

$ terraform apply -auto-approve -compact-warnings

Here’s the result of the terraform apply command for my cluster:

In order to verify if everything has been installed successfully we can display the details of the vault-csi-provider DaemonSet in the vault namespace.

$ kubectl describe ds vault-csi-provider -n vault

Then, we can do a very similar thing for the Secrets Store CSI driver. We need to display the details of the csi-secrets-store-secrets-store-csi-driver DeamonSet.

$ kubectl describe ds csi-secrets-store-secrets-store-csi-driver \
  -n k8s-secrets-store-csi

Configure Vault with Terraform

The big advantage of using Terraform in our scenario is its integration with Vault. There is a dedicated Terraform provider for interacting with HashiCorp Vault. In order to set up the provider, we need to pass the Vault token (root in dev mode) and address. We will still need the kubernetes provider in that part of the exercise.

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "vault" {
  token = "root"
  address = var.vault-addr
}

We need to set the Kubernetes context path name and Vault API address in the variables.tf file. Here’s my Terraform variables.tf file:

variable "cluster-context" {
  type    = string
  default = "default/api-cluster-6sccr-6sccr-sandbox1544-opentlc-com:6443/opentlc-mgr"
}

variable "vault-addr" {
  type = string
  default = "http://vault-vault.apps.cluster-6sccr.6sccr.sandbox1544.opentlc.com"
}

The Terraform script responsible for configuring Vault is visible below. There are several things we need to do before deploying a sample Spring Boot app. Here’s a list of the required steps:

1. We need to enable Kubernetes authentication in Vault. The Secrets Store CSI Driver will use it to authenticate against the instance of Vault
2. In the second step, we will create a test secret. Its name is password and the value is test1. It is stored in Vault under the /secret/data/db-pass path.
3. Then, we have to Configure Kubernetes authenticate method.
4. In the fourth step, we are creating the policy for our app. It has read access to the secret created in step 2
5. We are creating the ServicerAccount for our sample Spring Boot app in the default namespace. The name of the ServiceAccount object is webapp-sa.
6. Finally, we can proceed to the last step in the Vault configuration – the authentication role required to access the secret. The name of the role is webapp and is then used by the Secrets Store CSI SecretProviderClass CR. The authentication role refers to the already created policy and ServiceAccount webapp-sa in the default namespace.
7. Once the Vault backend is configured properly we create the Secrets Store CSI SecretProviderClass CR.

# (1)
resource "vault_auth_backend" "kubernetes" {
  type = "kubernetes"
}

# (2)
resource "vault_kv_secret_v2" "secret" {
  mount = "secret"
  name = "db-pass"
  data_json = jsonencode(
    {
      password = "test1"
    }
  )
}

data "kubernetes_secret" "vault-token" {
  metadata {
    name      = "vault-token"
    namespace = "vault"
  }
}

# (3)
resource "vault_kubernetes_auth_backend_config" "example" {
  backend                = vault_auth_backend.kubernetes.path
  kubernetes_host        = "https://172.30.0.1:443"
  kubernetes_ca_cert     = data.kubernetes_secret.vault-token.data["ca.crt"]
  token_reviewer_jwt     = data.kubernetes_secret.vault-token.data.token
}

# (4)
resource "vault_policy" "internal-app" {
  name = "internal-app"

  policy = <<EOT
path "secret/data/db-pass" {
  capabilities = ["read"]
}
EOT
}

# (5)
resource "kubernetes_service_account" "webapp-sa" {
  metadata {
    name      = "webapp-sa"
    namespace = "default"
  }
}

# (6)
resource "vault_kubernetes_auth_backend_role" "internal-role" {
  backend                          = vault_auth_backend.kubernetes.path
  role_name                        = "webapp"
  bound_service_account_names      = ["webapp-sa"]
  bound_service_account_namespaces = ["default"]
  token_ttl                        = 3600
  token_policies                   = ["internal-app"]
}

# (7)
resource "kubernetes_manifest" "vault-database" {
  manifest = {
    "apiVersion" = "secrets-store.csi.x-k8s.io/v1alpha1"
    "kind"       = "SecretProviderClass"
    "metadata" = {
      "name"      = "vault-database"
      "namespace" = "default"
    }
    "spec" = {
      "provider"   = "vault"
      "parameters" = {
        "vaultAddress" = "http://vault.vault.svc:8200"
        "roleName"     = "webapp"
        "objects"      = "- objectName: \"db-password\"\n  secretPath: \"secret/data/db-pass\"\n  secretKey: \"password\""
      }
    }
  }
}

Once again, to apply the configuration we need to execute the terraform apply command.

Of course, we could apply the whole configuration visible above using Vault CLI or UI. However, we can verify it with Vault UI. In order to log in there we must use the root token. After login, we need to go to the Access tab and then to the Auth Methods menu. As you see, there is a webapp method defined in Terraform scripts.

Let’s switch to the Policies tab. Then we can check out if the internal-app policy exists.

Run the App with Mounted Secrets

Once we applied the whole configuration with Terraform we may proceed to the sample Spring Boot app. The idea is pretty simple. Our app just reads the data from the file and exposes it as the REST endpoint. Here’s the REST @Controller implementation:

@RestController
@RequestMapping("/api")
class SampleUtilController {

    @GetMapping("/db-password")
    fun resourceString(): String {
        val file = File("/mnt/secrets-store/db-password")
        return if(file.exists()) file.readText()
        else "none"
    }
}

Here’s the app Deployment manifest. As you see we are using the secrets-store.csi.k8s.io implementation of the CSI driver for mounted volume. It refers the vault-database SecretProviderClass object created with the Terraform script. The volume is containing the file with the value of our secret. We are mounting it under the path /mnt/secrets-store, which is accessed by the Spring Boot application.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-util-app
spec:
  selector:
    matchLabels:
      app: sample-util-app
  template:
    metadata:
      labels:
        app: sample-util-app
    spec:
      serviceAccountName: webapp-sa
      containers:
        - name: sample-util-app
          image: piomin/sample-util-app
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /mnt/secrets-store
              name: secrets-store
              readOnly: true
      volumes:
        - name: secrets-store
          csi:
            driver: secrets-store.csi.k8s.io
            readOnly: true
            volumeAttributes:
              secretProviderClass: "vault-database"

Here’s the Kubernetes Service manifest:

apiVersion: v1
kind: Service
metadata:
  name: sample-util-app
spec:
  type: ClusterIP
  selector:
    app: sample-util-app
  ports:
    - port: 8080
      targetPort: 8080

We can easily build and deploy the app with Skaffold. It also allows exposing a port outside the cluster as a local port with the port-forward option.

$ skaffold dev --port-forward

Finally, we can our test endpoint GET /api/db-password. It returns the value we have already set in Vault for the db-pass/password secret.

$ curl http://localhost:8080/api/db-password
test2

Now, let’s test the secret rotation feature. In order to do that we need to change the value of the db-pass/password secret. We can do it using Vault UI. We can set the test2 value:

Secrets Store CSI Driver periodically queries managed secrets to detect changes. So, after the change, we would probably wait a moment until our app refreshes that value. The default poll interval is 2 minutes. We can override it using the rotation-poll-interval parameter (e.g. on the Helm chart). However, the most important thing is, that everything happens without restarting the pod. The only trace of change is in the events:

Now, let’s query for the latest value of the key used by the app. As you see the value has been refreshed.

Final Thoughts

If you are looking for a solution that injects Vault secrets into your app without creating Kubernetes Secret, Secrets Store CSI Driver is the solution for you. It is able to refresh the value of the secret in your app without restarting the container. In this article, I’m presenting how to install and configure it with Terraform to simplify the installation and configuration process.

The post Vault with Secrets Store CSI Driver on Kubernetes appeared first on Piotr's TechBlog.

How it works

Let’s discuss our architecture in greater detail. In the first step, we are installing ArgoCD and Sealed Secrets on Kubernetes with Terraform. In order to install both these tools, we will leverage Terraform support for Helm charts. During ArgoCD installation we will also create the Application that refers to the Git repository with configuration (1). This repository will contain YAML manifests including an encrypted version of our Kubernetes Secret. When Terraform installs Sealed Secrets it sets the private key for secrets decryption and the public key for encryption (2).

Once we successfully install Sealed Secrets we can interact with its controller running on the cluster with kubeseal CLI. With the kubeseal command we can get an encrypted version of the input Kubernetes Secret (3). Then we place an encrypted secret inside the repository with the app deployment manifests (4). Argo CD will automatically apply the latest configuration to the Kubernetes cluster (5). Once the new encrypted secret appears Sealed Secrets detects it and tries to decrypt using a previously set private key (6). As a result, a new Secret object is created and then injected into our sample app (7). That’s the last step of our exercise. We will test the result using the HTTP endpoint exposed by the app.

Prerequisites

To proceed with the exercise you need to have a running instance of Kubernetes. It can be a local or a cloud instance – it doesn’t matter. Additionally, you also need to install two CLI tools on your laptop:

1. kubeseal – the client-side part of Sealed Secrets. You will installation instructions here.
2. terraform – to run Terraform HCL scripts you need to have CLI. You will installation instructions here.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that you need to clone my GitHub repository. This repository contains sample Terraform scripts for initializing Kubernetes. You should go to the sealedsecrets directory. Then just follow my instructions.

Install ArgoCD and Sealed Secrets with Terraform

Assuming you already have the terraform CLI installed the first thing you need to do is to define the Helm provider with the path to your Kube config and the name of the Kube context. Here’s our providers.tf file:

provider "kubernetes" {
  config_path = "~/.kube/config"
  config_context = var.cluster-context
}

provider "helm" {
  kubernetes {
    config_path = "~/.kube/config"
    config_context = var.cluster-context
  }
}

Since I’m using Kubernetes on the Docker Desktop the name of my context is docker-desktop. Here’s the variables.tf file:

variable "cluster-context" {
  type    = string
  default = "docker-desktop"
}

Here’s the Terraform script for installing ArgoCD and Sealed Secrets. For Sealed Secrets, we need to set keys for encryption and decryption. By default, the Sealed Secrets chart detects an existing TLS secret with the name sealed-secrets-key inside the target namespace. If it does not exist the chart creates a new one containing generated keys. In order to define the secret with the predefined TLS keys, we first need to create the namespace (1). Then we create the Secret sealed-secrets-key that contains our tls.crt and tls.key (2). After that we may install Sealed Secrets in the sealed-secrets namespace using Helm chart (3).

At the same time, we are installing ArgoCD in the argocd namespace also using Helm chart (4). The chart automatically creates the namespace thanks to the create_namespace parameter. Once we install ArgoCD we can create the Application object responsible for synchronization between the Git repository and Kubernetes cluster. We can also do it using the same Terraform script thanks to the argocd-apps Helm chart (5). It allows us to define a list of ArgoCD Applications inside the Helm values file (6).

# Sealed Secrets Installation

# (1)
resource "kubernetes_namespace" "sealed-secrets-ns" {
  metadata {
    name = "sealed-secrets"
  }
}

# (2)
resource "kubernetes_secret" "sealed-secrets-key" {
  depends_on = [kubernetes_namespace.sealed-secrets-ns]
  metadata {
    name      = "sealed-secrets-key"
    namespace = "sealed-secrets"
  }
  data = {
    "tls.crt" = file("keys/tls.crt")
    "tls.key" = file("keys/tls.key")
  }
  type = "kubernetes.io/tls"
}

# (3)
resource "helm_release" "sealed-secrets" {
  depends_on = [kubernetes_secret.sealed-secrets-key]
  chart      = "sealed-secrets"
  name       = "sealed-secrets"
  namespace  = "sealed-secrets"
  repository = "https://bitnami-labs.github.io/sealed-secrets"
}

# ArgoCD Installation

# (4)
resource "helm_release" "argocd" {
  chart            = "argo-cd"
  name             = "argocd"
  namespace        = "argocd"
  repository       = "https://argoproj.github.io/argo-helm"
  create_namespace = true
}

# (5)
resource "helm_release" "argocd-apps" {
  depends_on = [helm_release.argocd]
  chart      = "argocd-apps"
  name       = "argocd-apps"
  namespace  = "argocd"
  repository = "https://argoproj.github.io/argo-helm"

  # (6)
  values = [
    file("argocd/applications.yaml")
  ]
}

We store Helm values inside the argocd/applications.yml file. In fact, we are going to apply the same set of YAML manifests into two different namespaces: demo-1 and demo-2. The namespace is automatically created during the synchronization.

applications:
 - name: sample-app-1
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-1
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
      - CreateNamespace=true
 - name: sample-app-2
   namespace: argocd
   project: default
   source:
     repoURL: https://github.com/piomin/openshift-cluster-config.git
     targetRevision: HEAD
     path: apps/simple
   destination:
     server: https://kubernetes.default.svc
     namespace: demo-2
   syncPolicy:
     automated:
       prune: false
       selfHeal: false
     syncOptions:
       - CreateNamespace=true

Now, the only thing is to apply the configuration to the Kubernetes cluster. Before we do that we need to initialize Terraform working directory with the following command:

$ cd sealed-secrets
$ terraform init

Finally, we can apply the configuration:

$ terraform apply

Here’s the output of the terraform apply command:

Encrypt Secret with Kubeseal

Assuming you have already installed Sealed Secrets with Terraform on your Kubernetes cluster and kubeseal CLI on your laptop you can encrypt your secret for the first time. Here’s our Kubernetes Secret. It contains just the single field password with the base64-encoded value 123456. We are going to create the SealedSecret object from that Secret using the kubeseal command.

apiVersion: v1
kind: Secret
metadata:
  name: sample-secret
type: Opaque
data:
  password: MTIzNDU2

By default, kubeseal tries to find the Sealed Secrets controller under the sealed-secrets-controller name inside the kube-system namespace. As you see we have already installed it in the sealed-secrets namespace under the sealed-secrets name.

We need to override both the controller name and namespace in the kubeseal command with the --controller-name and --controller-namespace parameters. Here’s our command:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets

The result may be quite surprising. Sealed Secrets doesn’t allow encrypting secrets without a namespace set in the YAML manifest. That’s because, by default, it uses a strict scope. With that scope, the secret must be sealed with exactly the same name and namespace. These attributes become part of the encrypted data. For me, it adds the default namespace as shown below.

Therefore it won’t be possible to decrypt the secret in a different namespace than the namespace set for the input Kubernetes Secret. On the other hand, we want to apply the same configuration in two different namespaces demo-1 and demo-2. In that case, we have to change the default scope to cluster-wide. With that kubeseal parameter the secret can be unsealed in any namespace and can be given any name. Here’s the command we should use to generate the SealedSecret object:

$ kubeseal -f sample-secret.yaml -w sample-sealed-secret.yaml \
   --controller-name sealed-secrets \
   --controller-namespace sealed-secrets \
   --scope cluster-wide

The output file of the command visible above contains the encrypted secret inside the SealedSecret object. Now, we should just add that YAML manifest to our Git repository.

Apply Sealed Secret with ArgoCD

Our sample Git repository with configuration for ArgoCD is available here. You should go to the apps/simple-with-secret directory. You will find there a Deployment, Service and SealedSecret objects. What’s important they don’t have any namespace set. Here’s our SealedSecret object:

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  annotations:
    sealedsecrets.bitnami.com/cluster-wide: "true"
  creationTimestamp: null
  name: sample-secret
spec:
  encryptedData:
    password: AgCW2Nf1gZzn42QQai/zr0VAtb5ZFyOjxMC8ghYcp5bu4EiYmJupX726zTx4XHQThrPgi/jHvzJoymToYJMIYuMegfKmZGcyMMZxJavYFTtlF9CIegPCkD3kjrJMCWcOadyDkBNIIfFAO6ljPwMD+stpsoBZ6WT8fGokxSwE/poKPpWFozC5RImf7HsYjGYVd8onxCySmcJZFYERi2G0qSWBlFDUsJ/ao5vyxIeiS25DBV1Bn475Lgyv6uTfvY6mesvrxw7OWjJmve2xRD/hS87Wp7cdBE264M/NMk1z24VysQr/ezSSI6S14NgzbcWo/hsKwWLmy6u259o8Xot5nVYpo2EhKFm/r62rko0eC2XMkjXhntMLKLpML3mTdadIFK50OauJvyVZS21sgeTlIMeSq6A6trekYyZvBtQaVixIthGHa/ymJXlIBZVJRL7/SJXquaX+J75AXUzPD3Hag8Kt5R5F6TVY2ox8RkMCpAVVAsiztMbyfRgzel6cAfDyj6l5f8GWI2T7gu5uHXgZFwVeyESn3aTO8qqws6NpLlwrtnjLwoCiXXC1Qo39wXaSJoH7fdJwihvOyiwbfaHkjhQwavNHpBoMEbKYQTV6DXSOTN8eeT1ZPoTN8AM+DtMdS2IpvMxZRsgaanh3O7gf5L02nGEq2WyP75s5sLoa7F8dQ27ZUeznqxIrNzrLqNM4dJuqZTbL4AM=
  template:
    metadata:
      annotations:
        sealedsecrets.bitnami.com/cluster-wide: "true"
      creationTimestamp: null
      name: sample-secret
    type: Opaque

Once it will be applied to the cluster, the Sealed Secrets controller will decrypt it and create the Kubernetes Secret object. Our sample app just takes the Kubernetes Secret and prints the value of the password key.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-app
spec:
  selector:
    matchLabels:
      app: sample-app
  template:
    metadata:
      labels:
        app: sample-app
    spec:
      containers:
      - name: sample-app
        image: quay.io/pminkows/sample-kotlin-spring:1.4.30
        ports:
        - containerPort: 8080
          name: http
        env:
          - name: PASS
            valueFrom:
              secretKeyRef:
                key: password
                name: sample-secret

We will test the app’s HTTP endpoint through the Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: sample-app
spec:
  type: ClusterIP
  selector:
    app: sample-app
  ports:
  - port: 8080
    name: http

Let’s verify what happened. Go to the ArgoCD dashboard. As you see there are two applications created by the Terraform script during installation. Both of them automatically applied the configuration from the Git repository to the cluster.

Let’s display details of the selected ArgoCD Application. As you see the sample-secret Secret object has already been created from the sample-secret SealedSecret object.

Now, let’s enable port-forward for the Kubernetes Service on port 8080:

$ kubectl port-forward svc/sample-app 8080:8080 -n demo-1

The app is able to display a list of environment variables. We can also display just a selected variable by calling the following endpoint:

$ curl http://localhost:8080/actuator/env/PASS

Final Thoughts

In general, there are two popular approaches to managing secrets on Kubernetes in GitOps style. In the first of them, we store the encrypted value of the secret in the Git repository. Then the software running on the cluster decrypts the value and creates Kubernetes Secret. That solution is represented by the Sealed Secrets and has been described today. In the second of them, we store just a reference to the secret, not the value of the secret in the Git repository. The value of the secret is stored in the third-party tool. Based on the key software running on the cluster retrieves the value.

The most popular example of such a third-party tool is HashiCorp Vault. You can read more about managing secrets with Vault and ArgoCD in the following article. There is also another very promising project in that area – External Secrets. You can expect my article about it soon

The post Sealed Secrets on Kubernetes with ArgoCD and Terraform appeared first on Piotr's TechBlog.

Before we start, it is worth reading the following article. It shows how to use cert-manager together with Istio to create secure gateways on Kubernetes.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the ssl directory. You will find two Spring Boot apps: secure-callme and secure-caller. After that, you should just follow my instructions. Let’s begin.

How it works

Before we go into the technical details, let me write a little bit more about the architecture of our solution. Our challenge is pretty common. We need secure SSL/TLS communication between the services running on Kubernetes. Instead of manually generating and replacing certs inside the apps, we need an automatic approach.

Here come the cert-manager and the Stakater Reloader. Cert Manager is able to generate certificates automatically, based on the provided CRD object. It also ensures the certificates are valid and up-to-date and will attempt to renew certificates before expiration. It puts all the required inside Kubernetes Secret. On the other hand, Stakater Reloader is able to watch if any changes happen in ConfigMap or Secret. Then it performs a rolling upgrade on pods, which use the particular ConfigMap or Secret. Here is the visualization of the described architecture.

Prerequisites

Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube, kind, or a cloud-hosted instance. No matter which distribution you choose you also need to have:

1. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
2. Helm – used to install additional tools on Kubernetes like Stakater Reloader or cert-manager

Install Cert Manager and Stakater Reloader

In order to install both cert-manager and Reloader on Kubernetes we will use Helm charts. We don’t need any specific settings just defaults. Let’s begin with the cert-manager. Before installing the chart we have to add CRD resources for the latest version 1.10.1:

$ kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.10.1/cert-manager.crds.yaml

Then, we need to add the jetstack chart repository:

$ helm repo add jetstack https://charts.jetstack.io

After that, we can install the chart using the following command:

$ helm install my-release cert-manager jetstack/cert-manager

The same with the Stakater Reloader – first we need to add the stakater charts repository:

$ helm repo add stakater https://stakater.github.io/stakater-charts

Then, we can install the latest version of the chart:

$ helm install my-reloader stakater/reloader

In order to verify that the installation finished successfully we can display a list of running pods:

$ kubectl get po
NAME                                          READY   STATUS    RESTARTS   AGE
my-cert-manager-578884c6cf-f9ppt              1/1     Running   0          1m
my-cert-manager-cainjector-55d4cd4bb6-6mgjd   1/1     Running   0          1m
my-cert-manager-webhook-5c68bf9c8d-nz7sd      1/1     Running   0          1m
my-reloader-reloader-7566fdc68c-qj9l4         1/1     Running   0          1m

That’s all. Now we can proceed to the implementation.

HTTPS with Spring Boot

Our first app secure-callme exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now our goal is to enable HTTPS for that app, and of course, make it work properly on Kubernetes. First, we should change the default server port for the Spring Boot app to 8443. Then we have to enable SSL and provide locations of key stores. Additionally, we will force verification of the client’s certificate with the server.ssl.client-auth property. Here’s the configuration for our Spring Boot inside the application.yml file.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

We will set the values of CERT_PATH and PASSWORD at the level of Kubernetes Deployment. Now, let’s switch to the secure-caller implementation. We have to configure SSL on the REST client side. Since we use Spring RestTemplate for calling services, we need to add customize its default behavior. Firstly, let’s include the Apache HttpClient dependency.

<dependency>
  <groupId>org.apache.httpcomponents.client5</groupId>
  <artifactId>httpclient5</artifactId>
</dependency>

Now, we will use Apache HttpClient as a low-level client for the Spring RestTemplate. We need to define a key store and trust store for the client since a server-side requires and verifies client cert. In order to create RestTempate @Bean we use RestTemplateBuilder.

@SpringBootApplication
public class SecureCaller {

   public static void main(String[] args) {
      SpringApplication.run(SecureCaller.class, args);
   }

   @Autowired
   ClientSSLProperties clientSSLProperties;

   @Bean
   RestTemplate builder(RestTemplateBuilder builder) throws 
      GeneralSecurityException, IOException {

      final SSLContext sslContext = new SSLContextBuilder()
         .loadTrustMaterial(
            new File(clientSSLProperties.getTrustStore()),
            clientSSLProperties.getTrustStorePassword().toCharArray())
         .loadKeyMaterial(
            new File(clientSSLProperties.getKeyStore()),
            clientSSLProperties.getKeyStorePassword().toCharArray(),
            clientSSLProperties.getKeyStorePassword().toCharArray()
         )
         .build();

      final SSLConnectionSocketFactory sslSocketFactory = 
         SSLConnectionSocketFactoryBuilder.create()
                .setSslContext(sslContext)
                .build();

      final HttpClientConnectionManager cm = 
         PoolingHttpClientConnectionManagerBuilder.create()
                .setSSLSocketFactory(sslSocketFactory)
                .build();

      final HttpClient httpClient = HttpClients.custom()
         .setConnectionManager(cm)
         .evictExpiredConnections()
         .build();

      return builder
         .requestFactory(() -> 
            new HttpComponentsClientHttpRequestFactory(httpClient))
         .build();
   }
}

The client credentials are taken from configuration settings under the client.ssl key. Here is the @ConfigurationProperties class used the RestTemplateBuilder in the previous step.

@Configuration
@ConfigurationProperties("client.ssl")
public class ClientSSLProperties {

   private String keyStore;
   private String keyStorePassword;
   private String trustStore;
   private String trustStorePassword;

   // GETTERS AND SETTERS ...

}

Here’s the configuration for the secure-caller inside application.yml file. The same as for the secure-callme we expose the REST endpoint over HTTPS.

server.port: 8443
server.ssl:
  enabled: true
  key-store: ${CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}
  client-auth: NEED

client.url: https://${HOST}:8443/callme
client.ssl:
  key-store: ${CLIENT_CERT_PATH}/keystore.jks
  key-store-password: ${PASSWORD}
  trust-store: ${CLIENT_CERT_PATH}/truststore.jks
  trust-store-password: ${PASSWORD}

The secure-caller app calls GET /callme exposed by the secure-callme app using customized RestTemplate.

@RestController
public class SecureCallerController {

   RestTemplate restTemplate;

   @Value("${client.url}")
   String clientUrl;

   public SecureCallerController(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @GetMapping("/caller")
   public String call() {
      return "I'm `secure-caller`! calling... " +
             restTemplate.getForObject(clientUrl, String.class);
   }

}

Generate and Renew Certificates on Kubernetes with Cert Manager

With cert-manager, we can automatically generate and renew certificates on Kubernetes. Of course, we could generate TLS/SSL certs using e.g. openssl as well and then apply them on Kubernetes. However, Cert Manager simplifies that process. It allows us to declaratively define the rules for the certs generation process. Let’s see how it works. Firstly, we need the issuer object. We can create a global issuer for the whole as shown. It uses the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: ss-clusterissuer
spec:
  selfSigned: {}

After that, we can generate certificates. Here’s the cert-manager Certificate object for the secure-callme app. There are some important things here. First of all, we can generate key stores together with a certificate and private key (1). The object refers to the ClusterIssuer created in the previous step (2). The name of Kubernetes Service used during communication is secure-callme, so the cert needs to have that name as CN. In order to enable certificate rotation we need to set validity time. The lowest possible value is 1 hour (4). So each time 5 minutes before expiration cert-manager will automatically renew a certificate (5). It won’t rotate the private key. In order to enable it we should set the parameter rotationPolicy to Always.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-callme-cert
spec:
  keystores: # (1)
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef: # (2)
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - secure-callme
  secretName: secure-callme-cert
  commonName: secure-callme # (3)
  duration: 1h # (4)
  renewBefore: 5m  # (5)

The Certificate object for secure-caller is very similar. The only difference is in the CN field. We will use the port-forward option during the test, so I’ll set the domain name to localhost (1).

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: secure-caller-cert
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: ss-clusterissuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - localhost
    - secure-caller
  secretName: secure-caller-cert
  commonName: localhost # (1)
  duration: 1h
  renewBefore: 5m

After applying both manifests we can display a list of Certificates. Each of them is related to the Secret with the same name:

$ kubectl get certificate
NAME                 READY   SECRET               AGE
secure-caller-cert   True    secure-caller-cert   1m
secure-callme-cert   True    secure-callme-cert   1m

Here are the details of the secure-callme-cert Secret. It contains the key store and trust store in the JKS format. We will use both of them in the Spring Boot SSL configuration (server.ssl.trust-store and server.ssl.key-store properties). There is also a certificate (tls.crt), a private key (tls.key), and CA (ca.crt).

$ kubectl describe secret secure-callme-cert
Name:         secure-callme-cert
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: secure-callme
              cert-manager.io/certificate-name: secure-callme-cert
              cert-manager.io/common-name: secure-callme
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: ss-clusterissuer
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
ca.crt:          550 bytes
keystore.jks:    1029 bytes
tls.crt:         550 bytes
tls.key:         227 bytes
truststore.jks:  422 bytes

Deploy and Reload Apps on Kubernetes

Since we have already prepared all the required components and objects, we may proceed with the deployment of our apps. As I mentioned in the “Prerequisites” section we will use Skaffold for building and deploying apps on the local cluster. Let’s begin with the secure-callme app.

First of all, we need to reload the app each time the secure-callme-cert changes. It occurs once per hour when the cert-manager renews the TLS certificate. In order to enable the automatic restart of the pod with Stakater Reloader we need to annotate the Deployment with secret.reloader.stakater.com/reload (1). The annotation should contain the name of Secret, which triggers the app reload. Of course, we also need to mount key store and trust store files (3) and set the mount path for the Spring Boot app available under the CERT_PATH env variable (2). We are mounting the whole secure-callme-cert Secret.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme
  annotations:
    # (1)
    secret.reloader.stakater.com/reload: "secure-callme-cert" 
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme
    spec:
      containers:
        - image: piomin/secure-callme
          name: secure-callme
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            - name: CERT_PATH # (2)
              value: /opt/secret
          volumeMounts:
            - mountPath: /opt/secret # (3)
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-cert # (4)

The password to the key store files is available inside the jks-password-secret:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
data:
  password: MTIzNDU2
type: Opaque

There is also the Kubernetes Service related to the app:

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme
  name: secure-callme
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme
  type: ClusterIP

Now, go to the secure-callme directory and just run the following command:

$ skaffold run

The Deployment manifest of the secure-caller app is a little bit more complicated. The same as before we need to reload the app on Secret change (1). However, this app uses two secrets. The first of them contains server certs (secure-caller-cert), while the second contains certs for communication with secure-callme. Consequently, we are mounting two secrets (5) and we are setting the path with server key stores (2) and client key stores (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller
  # (1)
  annotations:
    secret.reloader.stakater.com/reload: "secure-caller-cert,secure-callme-cert"
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller
    spec:
      containers:
        - image: piomin/secure-caller
          name: secure-caller
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: PASSWORD
              valueFrom:
                secretKeyRef:
                  key: password
                  name: jks-password-secret
            # (2)
            - name: CERT_PATH
              value: /opt/secret
            # (3)
            - name: CLIENT_CERT_PATH
              value: /opt/client-secret
            - name: HOST
              value: secure-callme
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
            - mountPath: /opt/client-secret
              name: client-cert
      volumes:
        # (5)
        - name: cert
          secret:
            secretName: secure-caller-cert
        - name: client-cert
          secret:
            secretName: secure-callme-cert

Then, go to the secure-caller directory and deploy the app. This time we enable port-forward to easily test the app locally.

$ skaffold run --port-forward

Let’s display a final list of all running apps. We have cert-manager components, the Stakater reloader, and our two sample Spring Boot apps.

$ kubectl get deploy 
NAME                         READY   UP-TO-DATE   AVAILABLE   AGE
my-cert-manager              1/1     1            1           1h
my-cert-manager-cainjector   1/1     1            1           1h
my-cert-manager-webhook      1/1     1            1           1h
my-reloader-reloader         1/1     1            1           1h
secure-caller                1/1     1            1           1h
secure-callme                1/1     1            1           1h

Testing Renew of Certificates on Kubernetes

The secure-caller app is available on the 8443 port locally, while the secure-callme app is available inside the cluster under the service name secure-callme. Let’s make a test call. Firstly, we need to download certificates and private keys stored on Kubernetes:

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.key}' | base64 --decode > tls.key

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.tls\.crt}' | base64 --decode > tls.crt

$ kubectl get secret secure-caller-cert \
  -o jsonpath \
  --template '{.data.ca\.crt}' | base64 --decode > ca.crt

Now, we can call the GET /caller endpoint using the following curl command. Under the hood, the secure-caller calls the endpoint GET /callme exposed by the secure-callme also over HTTPS. If you did everything according to the instruction you should have the same result as below.

$ curl https://localhost:8443/caller \
  --key tls.key \
  --cert tls.crt \
  --cacert ca.crt 
I'm `secure-caller`! calling... I'm `secure-callme`!

Our certificate is valid for one hour.

Let’s see what happens after one hour. A new certificate has already been generated and both our apps have been reloaded. Now, if try to call the same endpoint as before using old certificates you should see the following error.

Now, if you repeat the first step in that section it should work properly. We just need to download the certs to make a test. The internal communication over SSL works automatically after a reload of apps.

Final Thoughts

Of course, there are some other ways for achieving the same result as in our exercise. For example, you can a service mesh tool like Istio and enable mutual TLS for internal communication. You will still need to handle the automatic renewal of certificates somehow in that scenario. Cert Manager may be replaced with some other tools like HashiCorp Vault, which provides features for generating SSL/TLS certificates. You can as well use Spring Cloud Kubernetes with Spring Boot for watching for changes in secrets and reloading them without restarting the app. However, the solution used to renew certificates on Kubernetes presented in that article is simple and will work for any type of app.

The post Renew Certificates on Kubernetes with Cert Manager and Reloader appeared first on Piotr's TechBlog.

You can use Vault in several different ways on Kubernetes. For example, you may integrate it directly with your Spring Boot app using the Spring Cloud Vault project. To read more about it please refer to that post on my blog.

Prerequisites

In this exercise, we are going to use Helm a lot. Our sample app Deployment is based on the Helm template. We also use Helm to install Vault and ArgoCD on Kubernetes. Finally, we need to customize the ArgoCD Helm chart parameters to enable and configure ArgoCD Vault Plugin. So, before proceeding please ensure you have basic knowledge about Helm. Of course, you should also install it on your laptop. For me, it is possible with homebrew:

$ brew install helm

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

Run and Configure Vault on Kubernetes

In the first step, we are going to install Vault on Kubernetes. We can easily do it using its official Helm chart. In order to simplify our exercise, we run it in development mode with a single server instance. Normally, you would configure it in HA mode. Let’s add the Hashicorp helm repository:

$ helm repo add hashicorp https://helm.releases.hashicorp.com

In order to enable development mode the parameter server.dev.enabled should have the value true. We don’t need to override any other default values:

$ helm install vault hashicorp/vault \
    --set "server.dev.enabled=true"

To check if Vault is successfully installed on the Kubernetes cluster we can display a list of running pods:

$ kubectl get pod 
NAME                                   READY   STATUS    RESTARTS   AGE
vault-0                                1/1     Running   0          25s
vault-agent-injector-9456c6d55-hx2fd   1/1     Running   0          21s

We can configure Vault in several different ways. One of the options is through the UI. To login there we may use the root token generated only in development mode. Vault also exposes the HTTP API. The last available option is with the CLI. CLI is available inside the Vault pod, but as well we can install it locally. For me, it is possible using the brew install vault command. Then we need to enable port-forwarding and export Vault local address as the VAULT_ADDR environment variable:

$ kubectl port-forward vault-0 8200
$ export VAULT_ADDR=http://127.0.0.1:8200
$ vault status

Then just login to the Vault server using the root token:

Enable Kubernetes Authentication on Vault

There are several authentication methods on Vault. However, since we run it on Kubernetes we should the method dedicated to that platform. What is important, this method is also supported by the ArgoCD Vault Plugin. Firstly, let’s enable the Kubernetes Auth method:

$ vault auth enable kubernetes

Then, we need to configure our authentication method. There are three required parameters: the URL of the Kubernetes API server, Kubernetes CA certificate, and a token reviewer JWT.

$ vault write auth/kubernetes/config \
    token_reviewer_jwt="<your reviewer service account JWT>" \
    kubernetes_host=<your Kubernetes API address> \
    kubernetes_ca_cert=@ca.crt

In order to easily obtain all those parameters, you can the following three commands. Then you can set them also e.g. using the Vault UI.

$ echo "https://$( kubectl exec vault-0 -- env | grep KUBERNETES_PORT_443_TCP_ADDR| cut -f2 -d'='):443"
$ kubectl exec vault-0 \
  -- cat /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
$ echo $(kubectl exec vault-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token)

Once we pass all the required parameters, we may proceed to the named role creation. ArgoCD Vault Plugin will use that role to authenticate against the Vault server. We need to provide the namespace with ArgoCD and the name of the Kubernetes service account used by the ArgoCD Repo Server. Our token expires after 24 hours.

$ vault write auth/kubernetes/role/argocd \
  bound_service_account_names=argocd-repo-server \
  bound_service_account_namespaces=argocd \
  policies=argocd \
  ttl=24h

That’s all for now. We will also need to create a test secret on Vault and configure a policy for the argocd role. Before that, let’s take a look at our sample Spring Boot app and its Helm template.

Helm Template for Spring Boot App

Our app is very simple. It just exposes a single HTTP endpoint that returns the value of the environment variable inside a container. Here’s the REST controller class written in Kotlin.

@RestController
@RequestMapping("/persons")
class PersonController {

    @Value("\${PASS:none}")
    lateinit var pass: String

    @GetMapping("/pass")
    fun printPass() = pass

}

We will use a generic Helm chart to deploy our app on Kubernetes. Our Deployment template contains a list of environment variables defined for the container.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: {{ .Values.app.name }}
spec:
  replicas: {{ .Values.app.replicas }}
  selector:
    matchLabels:
      app: {{ .Values.app.name }}
  template:
    metadata:
      labels:
        app: {{ .Values.app.name }}
    spec:
      containers:
        - name: {{ .Values.app.name }}
          image: {{ .Values.image.registry }}/{{ .Values.image.repository }}:{{ .Values.image.tag }}
          ports:
          {{- range .Values.app.ports }}
            - containerPort: {{ .value }}
              name: {{ .name }}
          {{- end }}
          {{- if .Values.app.envs }}
          env:
          {{- range .Values.app.envs }}
            - name: {{ .name }}
              value: {{ .value }}
          {{- end }}
          {{- end }}

There is also another file in the templates directory. It contains a definition of the Kubernetes Service.

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.app.name }}
spec:
  type: ClusterIP
  selector:
    app: {{ .Values.app.name }}
  ports:
  {{- range .Values.app.ports }}
  - port: {{ .value }}
    name: {{ .name }}
  {{- end }}

Finally, let’s take a look at the Chart.yaml file.

apiVersion: v2
name: sample-with-envs
description: A Helm chart for Kubernetes
type: application
version: 1.0
appVersion: "1.0"

Our goal is to use this Helm chart to deploy the sample Spring Boot app on Kubernetes with ArgoCD and Vault. Of course, before we do it we need to install ArgoCD.

Install ArgoCD with Vault Plugin on Kubernetes

Normally, it would be a very simple installation. But this time we need to customize the ArgoCD template to install it with the Vault plugin. Or more precisely, we have to customize the configuration of the ArgoCD Repository Server. It is one of the ArgoCD internal services. It maintains the local cache of the Git repository and generates Kubernetes manifests.

There are some different options for installing Vault plugin on ArgoCD. The full list of options is available here. Starting with the version 2.4.0 of ArgoCD it is possible to install it via a sidecar container. We will choose the option based on sidecar and initContainer. You may read more about it here. However, our case would be different a little since we Helm instead of Kustomize for installing ArgoCD. To clarify, we need to do three things to install the Vault plugin on ArgoCD. Let’s analyze those steps:

• define initContainer on the ArgoCD Repository Server Deployment to download argocd-vault-plugin binaries and mount them on the volume
• define the ConfigMap containing the ConfigManagementPlugin CRD overriding a default behavior of Helm on ArgoCD
• customize argocd-repo-server Deployment to mount the volume with argocd-vault-plugin and the ConfigMap created in the previous step

After those steps, we would have to integrate the plugin with the running instance of the Vault server. We will use a previously create Vault argocd role.

Firstly, let’s create the ConfigMap to customize the default behavior of Helm on ArgoCD. After running the helm template command we will also run the argocd-vault-plugin generate command to replace all inline placeholders with the secrets defined in Vault. The address and auth configuration of Vault are defined in the vault-configuration secret.

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  plugin.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-helm
    spec:
      allowConcurrency: true
      discover:
        find:
          command:
            - sh
            - "-c"
            - "find . -name 'Chart.yaml' && find . -name 'values.yaml'"
      generate:
        command:
          - bash
          - "-c"
          - |
            helm template $ARGOCD_APP_NAME -n $ARGOCD_APP_NAMESPACE -f <(echo "$ARGOCD_ENV_HELM_VALUES") . |
            argocd-vault-plugin generate -s vault-configuration -
      lockRepo: false

Here’s the vault-configuration Secret:

apiVersion: v1
kind: Secret
metadata:
  name: vault-configuration
  namespace: argocd 
data:
  AVP_AUTH_TYPE: azhz
  AVP_K8S_ROLE: YXJnb2Nk
  AVP_TYPE: dmF1bHQ=
  VAULT_ADDR: aHR0cDovL3ZhdWx0LmRlZmF1bHQ6ODIwMA==
type: Opaque

To see the values let’s display the Secret in Lens. Vault is running in the default namespace, so its address is http://vault.default:8200. The name of our role in Vault is argocd. We also need to set the auth type as k8s.

Finally, we need to customize the ArgoCD Helm installation. To achieve that let’s define the Helm values.yaml file. It contains the definition of initContainer and sidecar for argocd-repo-server. We also mount the cmp-plugin ConfigMap to the Deployment, and add additional privileges to the argocd-repo-service ServiceAccount to allow reading Secrets.

repoServer:
  rbac:
    - verbs:
        - get
        - list
        - watch
      apiGroups:
        - ''
      resources:
        - secrets
        - configmaps
  initContainers:
    - name: download-tools
      image: registry.access.redhat.com/ubi8
      env:
        - name: AVP_VERSION
          value: 1.11.0
      command: [sh, -c]
      args:
        - >-
          curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
          chmod +x argocd-vault-plugin &&
          mv argocd-vault-plugin /custom-tools/
      volumeMounts:
        - mountPath: /custom-tools
          name: custom-tools

  extraContainers:
    - name: avp-helm
      command: [/var/run/argocd/argocd-cmp-server]
      image: quay.io/argoproj/argocd:v2.4.8
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
      volumeMounts:
        - mountPath: /var/run/argocd
          name: var-files
        - mountPath: /home/argocd/cmp-server/plugins
          name: plugins
        - mountPath: /tmp
          name: tmp-dir
        - mountPath: /home/argocd/cmp-server/config
          name: cmp-plugin
        - name: custom-tools
          subPath: argocd-vault-plugin
          mountPath: /usr/local/bin/argocd-vault-plugin

  volumes:
    - configMap:
        name: cmp-plugin
      name: cmp-plugin
    - name: custom-tools
      emptyDir: {}

In order to install ArgoCD on Kubernetes add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm

Let’s install it in the argocd namespace using customized parameters in the values.yaml file:

$ kubectl create ns argocd
$ helm install argocd argo/argo-cd -n argocd -f values.yaml

Sync Vault Secrets with ArgoCD

Once we deployed Vault and ArgoCD on Kubernetes we may proceed to the next step. Now, we are going to create a secret on Vault. Firstly, let’s enable the KV engine:

$ vault secrets enable kv-v2

Then, we can create a sample secret with the argocd name and a single password key:

$ vault kv put kv-v2/argocd password="123456"

ArgoCD Vault Plugin uses the argocd policy to read secrets. So, in the next step, we need to create the following policy to enable reading the previously created secret:

$ vault policy write argocd - <<EOF
path "kv-v2/data/argocd" {
  capabilities = ["read"]
}
EOF

Then, we may define the ArgoCD Application for deploying our Spring Boot app on Kubernetes. The Helm template for Kubernetes manifests is available on the GitHub repository under the simple-with-envs directory (1). As a tool for creating manifests we choose plugin (2). However, we won’t set its name since we use a sidecar container with argocd-vault-plugin. ArgoCD Vault plugin allows passing inline values in the application manifest. It reads the content defined inside the HELM_VALUES environment variable (3) (depending on the environment variable name set inside cmp-plugin ConfigMap). And finally, the most important thing. ArgoCD Vault Plugin is looking for placeholders inside the <> brackets. For inline values, it should have the following structure: <path:path_to_the_vault_secret#name_of_the_key> (4). In our case, we define the environment variable PASS that uses the argocd secret and the password key stored inside the KV engine.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: simple-helm
  namespace: argocd
spec:
  destination:
    name: ''
    namespace: default
    server: https://kubernetes.default.svc
  project: default
  source:
    path: simple-with-envs # (1)
    repoURL: https://github.com/piomin/sample-generic-helm-charts.git 
    targetRevision: HEAD
    plugin: # (2)
      env:
        - name: HELM_VALUES # (3)
          value: |
            image:
              registry: quay.io
              repository: pminkows/sample-kotlin-spring
              tag: "1.4.30"

            app:
              name: sample-spring-boot-kotlin
              replicas: 1
              ports:
                - name: http
                  value: 8080
              envs:
                - name: PASS
                  value: <path:kv-v2/data/argocd#password> # (4)

Finally, we can create the ArgoCD Application. It should have the OutOfSync status:

Let’s synchronize its state with the Git repository. We can do it using e.g. ArgoCD UI. If everything works fine you should see the green tile with your application name.

Then, let’s just verify the structure of our app Deployment. You should see the value 123456 instead of the placeholder defined inside the ArgoCD Application.

It is just a formality, but in the end, you can test the endpoint GET /persons/pass exposed by our Spring Boot app. It prints the value of the PASS environment variable. To do that you should also enable port-forwarding for the app.

$ kubectl port-forward svc/simple-helm 8080:8080
$ curl http://localhost:8080/persons/pass

Final Thoughts

GitOps approach becomes very popular in a Kubernetes-based environment. As always, one of the greatest challenges with that approach is security. Hashicorp Vault is one of the best tools for managing and protecting sensitive data. It can be easily installed on Kubernetes and included in your GitOps process. In this article, I showed how to use it together with other very popular solutions for deploying apps: ArgoCD and Helm.

The post Manage Secrets on Kubernetes with ArgoCD and Vault appeared first on Piotr's TechBlog.

There are several topics you should be familiar with before start reading the following article. At least, it is worth reading about the basics related to service mesh on Kubernetes with Istio and Spring Boot here. You may also be interested in the article about security best practices for Spring Boot apps.

Prerequisites

Before we begin we need to prepare a test cluster. Personally, I’m using a local OpenShift cluster that simplifies the installation of several tools we need. But you can as well use any other Kubernetes distribution like minikube or kind. No matter which version you choose, you need to install at least:

1. Istio – you can do it in several ways, here’s an instruction with Istio CLI
2. Cert Manager – the simplest way to install it with the kubectl
3. Skaffold (optionally) – a CLI tool to simplify deploying the Spring Boot app on Kubernetes and applying all the manifests in that exercise using a single command. You can find installation instructions here
4. Helm – used to install additional tools on Kubernetes (or OpenShift)

Since I’m using OpenShift I can install everything using the UI console and Operator Hub. Here’s the list of my tools:

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Then switch to the tls branch. After that, you should just follow my instructions. Let’s begin.

Generate Certificates with Cert Manager

You can as well generate TLS/SSL certs using e.g. openssl and then apply them on Kubernetes. However, Cert Manager simplifies that process. It automatically creates a Kubernetes Secret with all the required staff. Before we create a certificate we should configure a default ClusterIssuer. I’m using the simplest option – self-signed.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

In the next step, we are going to create the Certificate object. It is important to set the commonName parameter properly. It should be the same as the hostname of our gateway. Since I’m running the local instance of OpenShift the default domain suffix is apps-crc.testing. The name of the application is sample-spring-kotlin in that case. We will also need to generate a keystore and truststore for configuring 2-way SSL in the second scenario with the passthrough gateway. Finally, we should have the same Kubernetes Secret with a certificate placed in the namespace with the app (demo-apps) and the namespace with Istio workloads (istio-system in our case). We can sync secrets across namespaces in several different ways. I’ll use the built-in feature of Cert Manager based on the secretTemplate field and a tool called reflector. Here’s the final version of the Certificate object:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: sample-spring-boot-cert
  namespace: demo-apps
spec:
  keystores:
    jks:
      passwordSecretRef:
        name: jks-password-secret
        key: password
      create: true
  issuerRef:
    name: selfsigned-cluster-issuer
    group: cert-manager.io
    kind: ClusterIssuer
  privateKey:
    algorithm: ECDSA
    size: 256
  dnsNames:
    - sample-spring-kotlin.apps-crc.testing
  secretName: sample-spring-boot-cert
  commonName: sample-spring-kotlin.apps-crc.testing
  secretTemplate:
    annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed: "true"  
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "istio-system"
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "istio-system"

Before we apply the Certificate object we need to create the Secret with a keystore password:

kind: Secret
apiVersion: v1
metadata:
  name: jks-password-secret
  namespace: demo-apps
data:
  password: MTIzNDU2
type: Opaque

Of course, we also need to install the reflector tool on our Kubernetes cluster. We can easily do it using the following Helm commands:

$ helm repo add emberstack https://emberstack.github.io/helm-charts
$ helm repo update
$ helm upgrade --install reflector emberstack/reflector

Here’s the final result. There is the Secret sample-spring-kotlin-cert in the demo-apps namespace. It contains a TLS certificate, private key, CA, JKS keystore, and truststore. You can also verify that the same Secret is available in the istio-system namespace.

Create Istio Gateway with Mutual TLS

Let’s begin with our first scenario. In order to create a gateway with mTLS, we should set MUTUAL as a mode and set the name of the Secret containing the certificate and private key. The name of the Istio Gateway host is sample-spring-kotlin.apps-crc.testing. Gateway is available on Kubernetes under the default HTTPS port.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: MUTUAL
        credentialName: sample-spring-boot-cert

The Spring Boot application is available under the HTTP port. Therefore we should create a standard Istio VirtualService that refers to the already created gateway.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  http:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8080

Finally, we can run the Spring Boot app on Kubernetes using skaffold. We need to activate the istio-mutual profile that exposes the HTTP (8080) port.

$ skaffold dev -p istio-mutual

OpenShift Service Mesh automatically exposes Istio Gateway as the Route object. Let’s just verify if everything works fine. Assuming that the app has started successfully we can display a list of Istio gateways in the demo-apps namespace:

$ kubectl get gw -n demo-apps
NAME                    AGE
microservices-gateway   3m4s

Then let’s display a list of Istio virtual services in the same namespace:

$ kubectl get vs -n demo-apps
NAME                           GATEWAYS                    HOSTS                                       AGE
sample-spring-boot-vs-via-gw   ["microservices-gateway"]   ["sample-spring-kotlin.apps-crc.testing"]   4m2s

If you are running it on OpenShift you should also check the Route object in the istio-system namespace:

$ oc get route -n istio-system | grep sample-spring-kotlin

If you test it on Kubernetes you just need to set the Host header on your request. Here’s the curl command for testing our secure gateway. Since we enabled mutual TLS auth we need to provide the client key and certificate. We can copy them to the local machine from Kubernetes Secret generated by the Cert Manager.

Let’s call the REST endpoint GET /persons exposed by our sample Spring Boot app:

$ curl -v https://sample-spring-kotlin.apps-crc.testing/persons \
    --key tls.key \
    --cert tls.crt

You will probably receive the following response:

Ok, we forgot to add our CA to the trusted certificates. Let’s do that. Alternatively, we can set the parameter --cacert on the curl command.

Now, we can run again the same curl command as before. The secure communication with our app should work perfectly fine. We can proceed to the second scenario.

Mutual Auth for Spring Boot and Passthrough Istio Gateway

Let’s proceed to the second scenario. Now, our sample Spring Boot is exposing the HTTPS port with the client cert verification. In order to enable it on the app side, we need to provide some configuration settings.

Here’s our application.yml file. Firstly, we need to enable SSL and set the default port to 8443. It is important to force client certificate authentication with the server.ssl.client-auth property. As a result, we also need to provide a truststore file location. Finally, we don’t want to expose Spring Boot Actuator endpoint over SSL, so we force to expose them under the default plain port.

server.port: 8443
server.ssl.enabled: true
server.ssl.key-store: /opt/secret/keystore.jks
server.ssl.key-store-password: ${PASSWORD}

server.ssl.trust-store: /opt/secret/truststore.jks
server.ssl.trust-store-password: ${PASSWORD}
server.ssl.client-auth: NEED

management.server.port: 8080
management.server.ssl.enabled: false

In the next step, we need to inject both keystore.jks and truststore.jks into the app container. Therefore we need to modify Deployment to mount a secret generated by the Cert Manager as a volume. Once again, the name of that Secret is sample-spring-boot-cert. The content of that Secret would be available for the app under the /opt/secret directory. Of course, we also need to expose the port 8443 outside and inject a secret with the keystone and truststore password.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin-microservice
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: sample-spring-kotlin-microservice
  template:
    metadata:
      annotations:
        sidecar.istio.io/inject: "true"
      labels:
        app.kubernetes.io/name: sample-spring-kotlin-microservice
    spec:
      containers:
      - image: piomin/sample-spring-kotlin
        name: sample-spring-kotlin-microservice
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8443
          name: https
        env:
          - name: PASSWORD
            valueFrom:
              secretKeyRef:
                key: password
                name: jks-password-secret
        volumeMounts:
          - mountPath: /opt/secret
            name: sample-spring-boot-cert
      volumes:
        - name: sample-spring-boot-cert
          secret:
            secretName: sample-spring-boot-cert

Here’s the definition of our Istio Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - sample-spring-kotlin.apps-crc.testing
      tls:
        mode: PASSTHROUGH

The definition of the Istio VirtualService is slightly different than before. It contains the TLS route with the name of the host to match SNI.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-spring-boot-vs-via-gw
spec:
  hosts:
    - sample-spring-kotlin.apps-crc.testing
  gateways:
    - microservices-gateway
  tls:
    - route:
        - destination:
            host: sample-spring-kotlin-microservice
            port:
              number: 8443
          weight: 100
      match:
        - port: 443
          sniHosts:
            - sample-spring-kotlin.apps-crc.testing

Finally, we can deploy the current version of the app using the following skaffold command to enable HTTPS port for the app running on Kubernetes:

$ skaffold dev -p istio-passthrough

Now, you can repeat the same steps as in the previous section to verify that the current configuration works fine.

Final Thoughts

In this article, I showed how to use some interesting tools that simplify the configuration of HTTPS for apps running on Kubernetes. You could see how Istio, Cert Manager, or reflector can work together. We considered two variants of making secure HTTPS Istio gateways for the Spring Boot application.

The post HTTPS on Kubernetes with Spring Boot, Istio and Cert Manager appeared first on Piotr's TechBlog.