openshift Archives - Piotr's TechBlog https://piotrminkowski.com/tag/openshift/ Java, Spring, Kotlin, microservices, Kubernetes, containers Mon, 08 Dec 2025 23:10:01 +0000 en-US hourly 1 https://wordpress.org/?v=6.9.1 https://i0.wp.com/piotrminkowski.com/wp-content/uploads/2020/08/cropped-me-2-tr-x-1.png?fit=32%2C32&ssl=1 openshift Archives - Piotr's TechBlog https://piotrminkowski.com/tag/openshift/ 32 32 181738725 A Book: Hands-On Java with Kubernetes https://piotrminkowski.com/2025/12/08/a-book-hands-on-java-with-kubernetes/ https://piotrminkowski.com/2025/12/08/a-book-hands-on-java-with-kubernetes/#respond Mon, 08 Dec 2025 16:05:58 +0000 https://piotrminkowski.com/?p=15892 My book about Java and Kubernetes has finally been published! The book “Hands-On Java with Kubernetes” is the result of several months of work and, in fact, a summary of my experiences over the last few years of research and development. In this post, I want to share my thoughts on this book, explain why […]

The post A Book: Hands-On Java with Kubernetes appeared first on Piotr's TechBlog.

]]> My book about Java and Kubernetes has finally been published! The book “Hands-On Java with Kubernetes” is the result of several months of work and, in fact, a summary of my experiences over the last few years of research and development. In this post, I want to share my thoughts on this book, explain why I chose to write and publish it, and briefly outline its content and concept. To purchase the latest version, go to this link.

Here is a brief overview of all my published books.

Motivation

I won’t hide that this post is mainly directed at my blog subscribers and people who enjoy reading it and value my writing style. As you know, all posts and content on my blog, along with sample application repositories on GitHub, are always accessible to you for free. Over the past eight years, I have worked to publish high-quality content on my blog, and I plan to keep doing so. It is a part of my life, a significant time commitment, but also a lot of fun and a hobby.

I want to explain why I decided to write this book, why now, and why in this way. But first, a bit of background. I wrote my last book first, then my first book, over seven years ago. It focused on topics I was mainly involved with at the time, specifically Spring Boot and Spring Cloud. Since then, a lot of time has passed, and much has changed – not only in the technology itself but also a little in my personal life. Today, I am more involved in Kubernetes and container topics than, for example, Spring Cloud. For years, I have been helping various organizations transition from traditional application architectures to cloud-native models based on Kubernetes. Of course, Java remains my main area of expertise. Besides Spring Boot, I also really like the Quarkus framework. You can read a lot about both in my book on Kubernetes.

Based on my experience over the past few years, involving development teams is a key factor in the success of the Kubernetes platform within an organization. Ultimately, it is the applications developed by these teams that are deployed there. For developers to be willing to use Kubernetes, it must be easy for them to do so. That is why I persuade organizations to remove barriers to using Kubernetes and to design it in a way that makes it easier for development teams. On my blog and in this book, I aim to demonstrate how to quickly and simply launch applications on Kubernetes using frameworks such as Spring Boot and Quarkus.

It’s an unusual time to publish a book. AI agents are producing more and more technical content online. More often than not, instead of grabbing a book, people turn to an AI chatbot for a quick answer, though not always the best one. Still, a book that thoroughly introduces a topic and offers a step-by-step guide remains highly valuable.

Content of the Book

This book demonstrates that Java is an excellent choice for building applications that run on Kubernetes. In the first chapter, I’ll show you how to quickly build your application, create its image, and run it on Kubernetes without writing a single line of YAML or Dockerfile. This chapter also covers the minimum Kubernetes architecture you must understand to manage applications effectively in this environment. The second chapter, on the other hand, demonstrates how to effectively organize your local development environment to work with a Kubernetes cluster. You’ll see several options for running a distribution of your cluster locally and learn about the essential set of tools you should have. The third chapter outlines best practices for building applications on the Kubernetes platform. Most of the presented requirements are supported by simple examples and explanations of the benefits of meeting them. The fourth chapter presents the most valuable tools for the inner development loop with Kubernetes. After reading the first four chapters, you will understand the main Kubernetes components related to application management, enabling you to navigate the platform efficiently. You’ll also learn to leverage Spring Boot and Quarkus features to adapt your application to Kubernetes requirements.

In the following chapters, I will focus on the benefits of migrating applications to Kubernetes. The first area to cover is security. Chapter five discusses mechanisms and tools for securing applications running in a cluster. Chapter six describes Spring and Quarkus projects that enable native integration with the Kubernetes API from within applications. In chapter seven, you’ll learn about the service mesh tool and the benefits of using it to manage HTTP traffic between microservices. Chapter eight addresses the performance and scalability of Java applications in a Kubernetes environment. Chapter Eight demonstrates how to design a CI/CD process that runs entirely within the cluster, leveraging Kubernetes-native tools for pipeline building and the GitOps approach. This book also covers AI. In the final, ninth chapter, you’ll learn how to run a simple Java application that integrates with an AI model deployed on Kubernetes.

Publication

I decided to publish my book on Leanpub. Leanpub is a platform for writing, publishing, and selling books, especially popular among technical content authors. I previously published a book with Packt, but honestly, I was alone during the writing process. Leanpub is similar but offers several key advantages over publishers like Packt. First, it allows you to update content collaboratively with readers and keep it current. Even though my book is finished, I don’t rule out adding more chapters, such as on AI on Kubernetes. I also look forward to your feedback and plan to improve the content and examples in the repository continuously. Overall, this has been another exciting experience related to publishing technical content.

And when you buy such a book, you can be sure that most of the royalties go to me as the author, unlike with other publishers, where most of the royalties go to them as promoters. So, I’m looking forward to improving my book with you!

Conclusion

My book aims to bring together all the most interesting elements surrounding Java application development on Kubernetes. It is intended not only for developers but also for architects and DevOps teams who want to move to the Kubernetes platform.

The post A Book: Hands-On Java with Kubernetes appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2025/12/08/a-book-hands-on-java-with-kubernetes/feed/ 0 15892 Running .NET Apps on OpenShift https://piotrminkowski.com/2025/11/17/running-net-apps-on-openshift/ https://piotrminkowski.com/2025/11/17/running-net-apps-on-openshift/#respond Mon, 17 Nov 2025 09:27:34 +0000 https://piotrminkowski.com/?p=15785 This article will guide you on running a .NET application on OpenShift using the Source-to-Image (S2I) tool. While .NET is not my primary area of expertise, I have been working with it quite extensively lately. In this article, we will examine more complex application cases, which may initially present some challenges. If you are interested […]

The post Running .NET Apps on OpenShift appeared first on Piotr's TechBlog.

]]> This article will guide you on running a .NET application on OpenShift using the Source-to-Image (S2I) tool. While .NET is not my primary area of expertise, I have been working with it quite extensively lately. In this article, we will examine more complex application cases, which may initially present some challenges.

If you are interested in developing applications for OpenShift, you may also want to read my article on deploying Java applications using the odo tool.

Why Source-to-Image?

That’s probably the first question that comes to mind. Let’s start with a brief definition. Source-to-Image (S2I) is a framework and tool that enables you to write images using the application’s source code as input, producing a new image. In other words, it provides a clean, repeatable, and developer-friendly way to build container images directly from source code – especially in OpenShift, where it’s a core built-in mechanism. With S2I, there is no need to create Dockerfiles, and you can trust that the images will be built to run seamlessly on OpenShift without any issues.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Prerequisite – OpenShift cluster

There are several ways in which you can run an OpenShift cluster. I’m using a cluster that runs in AWS. But you can run it locally using OpenShift Local. This article describes how to install it on your laptop. You can also take advantage of the 30-day free Developer Sandbox service. However, it is worth mentioning that its use requires creating an account with Red Hat. To provision an OpenShift cluster in the developer sandbox, go here. You can also download and install Podman Desktop, which will help you set up both OpenShift Local and connect to the Developer Sandbox. Generally speaking, there are many possibilities. I assume you simply have an OpenShift cluster at your disposal.

Create a .NET application

I have created a slightly more complex application in terms of its modules. It consists of two main projects and two projects with unit tests. The WebApi.Library project is simply a module to be included in the main application, which is WebApi.App. Below is the directory structure of our sample repository.

.
├── README.md
├── WebApi.sln
├── src
│   ├── WebApi.App
│   │   ├── Controllers
│   │   │   └── VersionController.cs
│   │   ├── Program.cs
│   │   ├── Startup.cs
│   │   ├── WebApi.App.csproj
│   │   └── appsettings.json
│   └── WebApi.Library
│       ├── VersionService.cs
│       └── WebApi.Library.csproj
└── tests
    ├── WebApi.App.Tests
    │   ├── VersionControllerTests.cs
    │   └── WebApi.App.Tests.csproj
    └── WebApi.Library.Tests
        ├── VersionServiceTests.cs
        └── WebApi.Library.Tests.csproj
Plaintext

Both the library and the application are elementary in nature. The library provides a single method in the VersionService class to return its version read from the .csproj file.

using System.Reflection;

namespace WebApi.Library;

public class VersionService
{
    private readonly Assembly _assembly;

    public VersionService(Assembly? assembly = null)
    {
        _assembly = assembly ?? Assembly.GetExecutingAssembly();
    }

    public string? GetVersion()
    {
        var informationalVersion = _assembly
            .GetCustomAttribute<AssemblyInformationalVersionAttribute>()?
            .InformationalVersion;

        return informationalVersion ?? _assembly.GetName().Version?.ToString();
    }
}
C#

The application includes the library and uses its VersionService class to read and return the library version in the GET /api/version endpoint. There is no story behind it.

using Microsoft.AspNetCore.Mvc;
using WebApi.Library;

namespace WebApi.App.Controllers
{
    [ApiController]
    [Route("api/version")]
    public class VersionController : ControllerBase
    {
        private readonly VersionService _versionService;
        private readonly ILogger<VersionController> _logger;

        public VersionController(ILogger<VersionController> logger)
        {
            _versionService = new VersionService();
            _logger = logger;
        }

        [HttpGet]
        public IActionResult GetVersion()
        {
            _logger.LogInformation("GetVersion");
            var version = _versionService.GetVersion();
            return Ok(new { version });
        }
    }
}
C#

The application itself utilizes several other libraries, including those for generating Swagger API documentation, Prometheus metrics, and Kubernetes health checks.

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.OpenApi.Models;
using HealthChecks.UI.Client;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Prometheus;
using System;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Diagnostics.HealthChecks;

namespace WebApi.App
{
    public class Startup
    {
        private readonly IConfiguration _configuration;

        public Startup(IConfiguration configuration)
        {
            _configuration = configuration;
        }

        public void ConfigureServices(IServiceCollection services)
        {
            // Enhanced Health Checks
            services.AddHealthChecks()
                .AddCheck("memory", () =>
                    HealthCheckResult.Healthy("Memory usage is normal"),
                    tags: new[] { "live" });

            services.AddControllers();

            services.AddSwaggerGen(c =>
            {
                c.SwaggerDoc("v1", new OpenApiInfo {Title = "WebApi.App", Version = "v1"});
            });
        }

        public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
        {
            // Enable prometheus metrics
            app.UseMetricServer();
            app.UseHttpMetrics();

            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }

            app.UseSwagger();
            app.UseSwaggerUI(c => c.SwaggerEndpoint("/swagger/v1/swagger.json", "person-service v1"));

            // Kubernetes probes
            app.UseHealthChecks("/health/live", new HealthCheckOptions
            {
                Predicate = reg => reg.Tags.Contains("live"),
                ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
            });

            app.UseHealthChecks("/health/ready", new HealthCheckOptions
            {
                Predicate = reg => reg.Tags.Contains("ready"),
                ResponseWriter = UIResponseWriter.WriteHealthCheckUIResponse
            });

            app.UseRouting();

            app.UseEndpoints(endpoints =>
            {
                endpoints.MapControllers();
                endpoints.MapMetrics();
            });

            using var scope = app.ApplicationServices.CreateScope();
        }
    }
}
C#

As you can see, the WebApi.Library project is included as an internal module, while other dependencies are simply added from the external NuGet repository.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <ItemGroup>
    <ProjectReference Include="..\WebApi.Library\WebApi.Library.csproj" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="9.0.4" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
    <PackageReference Include="AspNetCore.HealthChecks.NpgSql" Version="9.0.0" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.Client" Version="9.0.0" />
    <PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore" Version="9.0.8" />
  </ItemGroup>

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Version>1.0.3</Version>
    <IsPackable>true</IsPackable>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <PackageId>WebApi.App</PackageId>
    <Authors>piomin</Authors>
    <Description>WebApi</Description>
  </PropertyGroup>

</Project>
XML

Using OpenShift Source-to-Image for .NET

S2I Locally with CLI

Before testing a mechanism on OpenShift, you try Source-to-Image locally. On macOS, you can install s2i CLI using Homebrew:

brew install source-to-image
ShellSession

After installation, check its version:

$ s2i version
s2i v1.5.1
ShellSession

Then, go to the repository root directory. At this point, we need to parameterize our build because the repository contains several projects. Fortunately, S2I provides a parameter that allows us to set the main project in a multi-module structure easily. It must be set as an environment variable for the s2i command. The following command sets the DOTNET_STARTUP_PROJECT environment variable and uses the registry.access.redhat.com/ubi8/dotnet-90:latest as a builder image. 

s2i build . registry.access.redhat.com/ubi8/dotnet-90:latest webapi-app \
  -e DOTNET_STARTUP_PROJECT=src/WebApi.App
ShellSession

Of course, you must have Docker or Podman running on your laptop to use s2i. So, before using a builder image, pull it to your host.

podman pull registry.access.redhat.com/ubi8/dotnet-90:latest
ShellSession

Let’s take a look at the s2i build command output. As you can see, s2i restored and built two projects, but then created a runnable output for the WebApi.App project.

net-openshift-s2i-cli

What about our unit tests? To execute tests during the build, we must also set the DOTNET_TEST_PROJECTS environment variable.

s2i build . registry.access.redhat.com/ubi8/dotnet-90:latest webapi-app \
  -e DOTNET_STARTUP_PROJECT=src/WebApi.App \
  -e DOTNET_TEST_PROJECTS=tests/WebApi.App.Tests
ShellSession

Here’s the command output:

net-openshift-cli-2

The webapi-app image is ready.

$ podman images webapi-app
REPOSITORY                    TAG         IMAGE ID      CREATED        SIZE
docker.io/library/webapi-app  latest      e9d94f983ac1  5 seconds ago  732 MB
ShellSession

We can run it locally with Podman (or Docker):

S2I for .NET on OpenShift

Then, let’s switch to the OpenShift cluster. You need to log in to your cluster using the oc login command. After that, create a new project for testing purposes:

oc new-project dotnet
ShellSession

In OpenShift, a single command can handle everything necessary to build and deploy an application. We need to provide the address of the Git repository containing the source code, specify the branch name, and indicate the name of the builder image located within the cluster’s namespace. Additionally, we should include the same environment variables as we did previously. Since the version of source code we tested before is located in the dev branch, we must pass it together with the repository URL after #.

oc new-app openshift/dotnet:latest~https://github.com/piomin/web-api-2.git#dev --name webapi-app \
  --build-env DOTNET_STARTUP_PROJECT=src/WebApi.App \
  --build-env DOTNET_TEST_PROJECTS=tests/WebApi.App.Tests
ShellSession

Here’s the oc new-app command output:

Then, let’s expose the application outside the cluster using OpenShift Route.

oc expose service/webapi-app
ShellSession

Finally, we can verify the build and deployment status:

There is also a really nice command you can use here. Try yourself 🙂

oc get all
ShellSession

OpenShift Builds with Source-to-Image

Verify Build Status

Let’s verify what has happened after taking the steps from the previous section. Here’s the panel that summarizes the status of our application on the cluster. OpenShift automatically built the image from the .NET source code repository and then deployed it in the target namespace.

net-openshift-console

Here are the logs from the Pod with our application:

The build was entirely performed on the cluster. You can verify the logs from the build by accessing the Build object. After building, the image was pushed to the internal image registry in OpenShift.

net-openshift-build

Under the hood, the BuildConfig was created. This will be the starting point for the next example we will consider.

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  labels:
    app: webapi-app
    app.kubernetes.io/component: webapi-app
    app.kubernetes.io/instance: webapi-app
  name: webapi-app
  namespace: dotnet
spec:
  output:
    to:
      kind: ImageStreamTag
      name: webapi-app:latest
  source:
    git:
      ref: dev
      uri: https://github.com/piomin/web-api-2.git
    type: Git
  strategy:
    sourceStrategy:
      env:
      - name: DOTNET_STARTUP_PROJECT
        value: src/WebApi.App
      - name: DOTNET_TEST_PROJECTS
        value: tests/WebApi.App.Tests
      from:
        kind: ImageStreamTag
        name: dotnet:latest
        namespace: openshift
    type: Source
YAML

OpenShift with .NET and Azure Artifacts Proxy

Now let’s switch to the master branch in our Git repository. In this branch, the WebApi.Library library is no longer included as a path in the project, but as a separate dependency from an external repository. However, this library has not been published in the public NuGet repository, but in the internal Azure Artifacts repository. Therefore, the build process must take place via a proxy pointing to the address of our repository, or rather, a feed in Azure Artifacts.

<Project Sdk="Microsoft.NET.Sdk.Web">

  <ItemGroup>
    <PackageReference Include="WebApi.Library" Version="1.0.3" />
    <PackageReference Include="Swashbuckle.AspNetCore" Version="9.0.4" />
    <PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />
    <PackageReference Include="AspNetCore.HealthChecks.NpgSql" Version="9.0.0" />
    <PackageReference Include="AspNetCore.HealthChecks.UI.Client" Version="9.0.0" />
    <PackageReference Include="Microsoft.Extensions.Diagnostics.HealthChecks.EntityFrameworkCore" Version="9.0.8" />
  </ItemGroup>

  <PropertyGroup>
    <TargetFramework>net9.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <Version>1.0.3</Version>
    <IsPackable>true</IsPackable>
    <GeneratePackageOnBuild>true</GeneratePackageOnBuild>
    <PackageId>WebApi.App</PackageId>
    <Authors>piomin</Authors>
    <Description>WebApi</Description>
  </PropertyGroup>

</Project>
XML

This is how it looks in Azure Artifacts. The name of my feed is pminkows. To access the feed, I must be authenticated against Azure DevOps using a personal token. The full address of the NuGet registry exposed via my instance of Azure Artifacts is https://pkgs.dev.azure.com/pminkows/_packaging/pminkows/nuget/v3/index.json.

If you would like to build such an application locally using Azure Artifacts, you should create a nuget.config file with the configuration below. Then place it in the $HOME/.nuget/NuGet directory.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <clear />
    <add key="pminkows" value="https://pkgs.dev.azure.com/pminkows/_packaging/pminkows/nuget/v3/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <pminkows>
      <add key="Username" value="pminkows" />
      <add key="ClearTextPassword" value="<MY_PERSONAL_TOKEN>" />
    </pminkows>
  </packageSourceCredentials>
</configuration>
nuget.config

Our goal is to run this type of build on OpenShift instead of locally. To achieve this, we need to create a Kubernetes Secret containing the nuget.config file.

oc create secret generic nuget-config --from-file=nuget.config
ShellSession

Then, we must update the contents of the BuildConfig object. The changed lines in the object have been highlighted. The most important element is spec.source.secrets. The Kubernetes Secret containing the nuget.config file must be mounted in the HOME directory of the base image with .NET. We also change the branch in the repository to master and increase the logging level for the builder to detailed. 

apiVersion: build.openshift.io/v1
kind: BuildConfig
metadata:
  annotations:
    openshift.io/generated-by: OpenShiftNewApp
  labels:
    app: webapi-app
    app.kubernetes.io/component: webapi-app
    app.kubernetes.io/instance: webapi-app
  name: webapi-app
  namespace: dotnet
spec:
  output:
    to:
      kind: ImageStreamTag
      name: webapi-app:latest
  source:
    git:
      ref: master
      uri: https://github.com/piomin/web-api-2.git
    type: Git
    secrets:
      - secret:
          name: nuget-config
        destinationDir: /opt/app-root/src/
  strategy:
    sourceStrategy:
      env:
      - name: DOTNET_STARTUP_PROJECT
        value: src/WebApi.App
      - name: DOTNET_TEST_PROJECTS
        value: tests/WebApi.App.Tests
      - name: DOTNET_VERBOSITY
        value: d
      from:
        kind: ImageStreamTag
        name: dotnet:latest
        namespace: openshift
    type: Source
YAML

Next, we can run the build again, but this time with new parameters using the command below. With increased logging level, you can confirm that all dependencies are being retrieved via the Azure Artifacts instance.

oc start-build webapi-app --follow
ShellSession

Conclusion

This article covers different scenarios about building and deploying .NET applications in developer mode on OpenShift. It demonstrates how to use various parameters to customize image building according to the application’s needs. My goal was to demonstrate that deploying .NET applications on OpenShift is straightforward with the help of Source-to-Image.

The post Running .NET Apps on OpenShift appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2025/11/17/running-net-apps-on-openshift/feed/ 0 15785 The Art of Argo CD ApplicationSet Generators with Kubernetes https://piotrminkowski.com/2025/03/20/the-art-of-argo-cd-applicationset-generators-with-kubernetes/ https://piotrminkowski.com/2025/03/20/the-art-of-argo-cd-applicationset-generators-with-kubernetes/#comments Thu, 20 Mar 2025 09:40:46 +0000 https://piotrminkowski.com/?p=15624 This article will teach you how to use the Argo CD ApplicationSet generators to manage your Kubernetes cluster using a GitOps approach. An Argo CD ApplicationSet is a Kubernetes resource that allows us to manage and deploy multiple Argo CD Applications. It dynamically generates multiple Argo CD Applications based on a given template. As a […]

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

]]> This article will teach you how to use the Argo CD ApplicationSet generators to manage your Kubernetes cluster using a GitOps approach. An Argo CD ApplicationSet is a Kubernetes resource that allows us to manage and deploy multiple Argo CD Applications. It dynamically generates multiple Argo CD Applications based on a given template. As a result, we can deploy applications across multiple Kubernetes clusters, create applications for different environments (e.g., dev, staging, prod), and manage many repositories or branches. Everything can be easily achieved with a minimal source code effort.

Argo CD ApplicationSet supports several different generators. In this article, we will focus on the Git generator type. It generates Argo CD Applications based on directory structure or branch changes in a Git repository. It contains two subtypes: the Git directory generator and the Git file generator. If you are interested in other Argo CD ApplicationSet generators you can find some articles on my blog. For example, the following post shows how to use List Generator to promote images between environments. You can also find a post about the Cluster Decision Resource generator, which shows how to spread applications dynamically between multiple Kubernetes clusters.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. You must go to the appset-helm-demo directory, which contains the whole configuration required for that exercise. Then you should only follow my instructions.

Argo CD Installation

Argo CD is the only tool we need to install on our Kubernetes cluster for that exercise. We can use the official Helm chart to install it on Kubernetes. Firstly. let’s add the following Helm repository:

helm repo add argo https://argoproj.github.io/argo-helm
ShellSession

After that, we can install ArgoCD in the current Kubernetes cluster in the argocd namespace using the following command:

helm install my-argo-cd argo/argo-cd -n argocd
ShellSession

I use OpenShift in that exercise. With the OpenShift Console, I can easily install ArgoCD on the cluster using the OpenShift GitOps operator.

Once we installed it we can easily access the Argo CD dashboard.

We can sign in there using OpenShift credentials.

Motivation

Our goal in this exercise is to deploy and run some applications (a simple Java app and Postgres database) on Kubernetes with minimal source code effort. Those two applications only show how to create a standard that can be easily applied to any application type deployed on our cluster. In this standard, a directory structure determines how and where our applications are deployed on Kubernetes. My example configuration is stored in a single Git repository. However, we can easily extend it with multiple repositories, where Argoc CD switches between the central repository and other Git repositories containing a configuration for concrete applications.

Here’s a directory structure and files for deploying our two applications. Both the custom app and Postgres database are deployed in three environments: dev, test, and prod. We use Helm charts for deploying them. Each environment directory contains a Helm values file with installation parameters. The configuration distinguishes two different types of installation: apps and components. Each app is installed using the same Helm chart dedicated to a standard deployment. Each component is installed using a custom Helm chart provided by that component. For example, for Postgres, we will use the following Bitnami chart.

.
├── apps
│   ├── aaa-1
│   │   └── basic
│   │       ├── prod
│   │       │   └── values.yaml
│   │       ├── test
│   │       │   └── values.yaml
│   │       ├── uat
│   │       │   └── values.yaml
│   │       └── values.yaml
│   ├── aaa-2
│   └── aaa-3
└── components
    └── aaa-1
        └── postgresql
            ├── prod
            │   ├── config.yaml
            │   └── values.yaml
            ├── test
            │   ├── config.yaml
            │   └── values.yaml
            └── uat
                ├── config.yaml
                └── values.yaml
ShellSession

Before deploying the application, we should prepare namespaces with quotas, Argo CD projects, and ApplicationSet generators for managing application deployments. Here’s the structure of a global configuration repository. It also uses Helm chart to apply that part of manifests to the Kubernetes cluster. Each directory inside the projects directory determines our project name. On the other hand, a project contains several Kubernetes namespaces. Each project may contain several different Kubernetes Deployments.

.
└── projects
    ├── aaa-1
    │   └── values.yaml
    ├── aaa-2
    │   └── values.yaml
    └── aaa-3
        └── values.yaml
ShellSession

Prepare Global Cluster Configuration

Helm Template for Namespaces and Quotas

Here’s the Helm template for creating namespaces and quotas for each namespace. We will create the project namespace per each environment (stage). 

{{- range .Values.stages }}
---
apiVersion: v1
kind: Namespace
metadata:
  name: {{ $.Values.projectName }}-{{ .name }}
---
apiVersion: v1
kind: ResourceQuota
metadata:
  name: default-quota
  namespace: {{ $.Values.projectName }}-{{ .name }}
spec:
  hard:
    {{- if .config }}
    {{- with .config.quotas }}
    pods: {{ .pods | default "10" }}
    requests.cpu: {{ .cpuRequest | default "2" }}
    requests.memory: {{ .memoryRequest | default "2Gi" }}
    limits.cpu: {{ .cpuLimit | default "8" }}
    limits.memory: {{ .memoryLimit | default "8Gi" }}
    {{- end }}
    {{- else }}
    pods: "10"
    requests.cpu: "2"
    requests.memory: "2Gi"
    limits.cpu: "8"
    limits.memory: "8Gi"
    {{- end }}
{{- end }}
chart/templates/namespace.yaml

Helm Template for the Argo CD AppProject

Helm chart will also create a dedicated Argo CD AppProject object per our project.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: {{ .Values.projectName }}
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  clusterResourceWhitelist:
    - group: '*'
      kind: '*'
  destinations:
    - namespace: '*'
      server: '*'
  sourceRepos:
    - '*'
chart/templates/appproject.yaml

Helm Template for Argo CD ApplicationSet

After that, we can proceed to the most tricky part of our exercise. Helm chart also defines a template for creating the Argo CD ApplicationSet. This ApplicationSet must analyze the repository structure, which contains the configuration of apps and components. We define two ApplicationSets per each project. The first uses the Git Directory generator to determine the structure of the apps catalog and deploy the apps in all environments using my custom spring-boot-api-app chart. The chart parameters can be overridden with Helm values placed in each app directory.

The second ApplicationSet uses the Git Files generator to determine the structure of the components catalog. It reads the contents of the config.yaml file in each directory. The config.yaml file sets the repository, name, and version of the Helm chart that must be used to install the component on Kubernetes.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-apps-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          {{- range .Values.stages }}
          - path: appset-helm-demo/apps/{{ $.Values.projectName }}/*/{{ .name }}
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: spring-boot-api-app
          repoURL: 'https://piomin.github.io/helm-charts/'
          targetRevision: 0.3.8
          helm:
            valueFiles:
              - $values/appset-helm-demo/apps/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ .Values.projectName }}-components-config'
  namespace: {{ .Values.argoNamespace | default "argocd" }}
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        files:
          {{- range .Values.stages }}
          - path: appset-helm-demo/components/{{ $.Values.projectName }}/*/{{ .name }}/config.yaml
          {{- end }}
  template:
    metadata:
      name: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'
    spec:
      destination:
        namespace: '{{`{{ index .path.segments 2 }}`}}-{{`{{ index .path.segments 4 }}`}}'
        server: 'https://kubernetes.default.svc'
      project: '{{ .Values.projectName }}'
      sources:
        - chart: '{{`{{ .chart.name }}`}}'
          repoURL: '{{`{{ .chart.repository }}`}}'
          targetRevision: '{{`{{ .chart.version }}`}}'
          helm:
            valueFiles:
              - $values/appset-helm-demo/components/{{ .Values.projectName }}/{{`{{ index .path.segments 3 }}`}}/{{`{{ index .path.segments 4 }}`}}/values.yaml
            parameters:
              - name: appName
                value: '{{ .Values.projectName }}'
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
chart/templates/applicationsets.yaml

There are several essential elements in this configuration, which we should pay attention to. Both Helm and ApplicationSet use templating engines based on {{ ... }} placeholders. So to avoid conflicts we should escape Argo CD ApplicationSet templating elements from the Helm templating elements. The following part of the template responsible for generating the Argo CD Application name is a good example of that approach: '{{`{{ index .path.segments 3 }}`}}-{{`{{ index .path.segments 4 }}`}}'. First, we use the AppliocationSet Git generator parameter index .path.segments 3 that returns the name of the third part of the directory path. Those elements are escaped with the ` char so Helm doesn’t try to analyze it.

Helm Chart Structure

Our ApplicationSets use the “Multiple Sources for Application” feature to read parameters from Helm values files and inject them into the Helm chart from a remote repository. Thanks to that, our configuration repositories for apps and components contain only values.yaml files in the standardized directory structure. The only chart we store in the sample repository has been described above and is responsible for creating the configuration required to run app Deployments on the cluster.

.
└── chart
    ├── Chart.yaml
    ├── templates
    │   ├── additional.yaml
    │   ├── applicationsets.yaml
    │   ├── appproject.yaml
    │   └── namespaces.yaml
    └── values.yaml
ShellSession

By default, each project defines three environments (stages): test, uat, prod.

stages:
  - name: test
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}
chart/values.yml

We can override a default behavior for the specific project in Helm values. Each project directory contains the values.yaml file. Here are Helm parameters for the aaa-3 project that override CPU request quota from 2 CPUs to 4 CPUs only for the test environment.

stages:
  - name: test
    config:
      quotas:
        cpuRequest: 4
    additionalObjects: {}
  - name: uat
    additionalObjects: {}
  - name: prod
    additionalObjects: {}
projects/aaa-3/values.yaml

Run the Synchronization Process

Generate Global Structure on the Cluster

To start a process we must create the ApplicationSet that reads the structure of the projects directory. Each subdirectory in the projects directory indicates the name of our project. Our ApplicationSet uses a Git directory generator to create an Argo CD Application per each project. Its name contains the name of the subdirectory and the config suffix. Each generated Application uses the previously described Helm chart to create all namespaces, quotas, and other resources requested by the project. It also leverages the “Multiple Sources for Application” feature to allow us to override default Helm chart settings. It reads a project name from the directory name and passes it as a parameter to the generated Argo CD Application. 

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: global-config
  namespace: openshift-gitops
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://github.com/piomin/argocd-showcase.git
        revision: HEAD
        directories:
          - path: appset-helm-demo/projects/*
  template:
    metadata:
      name: '{{.path.basename}}-config'
    spec:
      destination:
        namespace: '{{.path.basename}}'
        server: 'https://kubernetes.default.svc'
      project: default
      sources:
        - path: appset-helm-demo/chart
          repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          helm:
            valueFiles:
              - $values/appset-helm-demo/projects/{{.path.basename}}/values.yaml
            parameters:
              - name: projectName
                value: '{{.path.basename}}'
              - name: argoNamespace
                value: openshift-gitops
        - repoURL: 'https://github.com/piomin/argocd-showcase.git'
          targetRevision: HEAD
          ref: values
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
YAML

Once we create the global-config ApplicationSet object magic happens. Here’s the list of Argo CD Applications generated from our directories in Git configuration repositories.

argo-cd-applicationset-all-apps

First, there are three Argo CD Applications with the projects’ configuration. That’s happening because we defined 3 subdirectories in the projects directory with names aaa-1, aaa-2 and aaa-3.

The configuration applied by those Argo CD Applications is pretty similar since they are using the same Helm chart. We can look at the list of resources managed by the aaa-3-config Application. There are three namespaces (aaa-3-test, aaa-3-uat, aaa-3-prod) with resource quotas, a single Argo CD AppProject, and two ApplicationSet objects responsible for generating Argo CD Applications for apps and components directories.

argo-cd-applicationset-global-config

In this configuration, we can verify if the value of the request.cpu ResourceQuota object has been overridden from 2 CPUs to 4 CPUs.

Let’s analyze what happened. Here’s a list of Argo CD ApplicationSets. The global-config ApplicationSet generated Argo CD Application per each detected project inside the projects directory. Then, each of these Applications applied two ApplicationSet objects to cluster using the Helm template. 

$ kubectl get applicationset
NAME                      AGE
aaa-1-components-config   29m
aaa-1-apps-config         29m
aaa-2-components-config   29m
aaa-2-apps-config         29m
aaa-3-components-config   29m
aaa-3-apps-config         29m
global-config             29m
ShellSession

There’s also a list of created namespaces:

$ kubectl get ns
NAME                                               STATUS   AGE
aaa-1-prod                                         Active   34m
aaa-1-test                                         Active   34m
aaa-1-uat                                          Active   34m
aaa-2-prod                                         Active   34m
aaa-2-test                                         Active   34m
aaa-2-uat                                          Active   34m
aaa-3-prod                                         Active   34m
aaa-3-test                                         Active   34m
aaa-3-uat                                          Active   34m
ShellSession

Generate and Apply Deployments

Our sample configuration contains only two Deployments. We defined the basic subdirectory in the apps directory and the postgres subdirectory in the components directory inside the aaa-1 project. The aaa-2 and aaa-3 projects don’t contain any Deployments for simplification. However, the more subdirectories with the values.yaml file we create there, the more applications will be deployed on the cluster. Here’s a typical values.yaml file for a simple app deployed with a standard Helm chart. It defines the image repository, name, and tag. It also set the Deployment name and environment.

image:
  repository: piomin/basic
  tag: 1.0.0
app:
  name: basic
  environment: prod
YAML

For the postgres component we must set more parameters in Helm values. Here’s the final list:

global:
  compatibility:
    openshift:
      adaptSecurityContext: force

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data
YAML

The following Argo CD Application has been generated by the aaa-1-apps-config ApplicationSet. It detected the basic subdirectory in the apps directory. The basic subdirectory contained 3 subdirectories: test, uat and prod with values.yaml file. As a result, we have Argo CD per environment responsible for deploying the basic app in the target namespaces.

argo-cd-applicationset-basic-apps

Here’s a list of resources managed by the basic-prod Application. It uses my custom Helm chart and applies Deployment and Service objects to the cluster.

The following Argo CD Application has been generated by the aaa-1-components-config ApplicationSet. It detected the basic subdirectory in the components directory. The postgres subdirectory contained 3 subdirectories: test, uat and prod with values.yaml and config.yaml files. The ApplicationSet Files generator reads the repository, name, and version from the configuration in the config.yaml file.

Here’s the config.yaml file with the Bitnami Postgres chart settings. We could place here any other chart we want to install something else on the cluster.

chart:
  repository: https://charts.bitnami.com/bitnami
  name: postgresql
  version: 15.5.38
components/aaa-1/postgresql/prod/config.yaml

Here’s the list of resources installed by the Bitnami Helm chart used by the generated Argo CD Applications.

argo-cd-applicationset-postgres

Final Thoughts

This article proves that Argo CD ApplicationSet and Helm templates can be used together to create advanced configuration structures. It shows how to use ApplicationSet Git Directory and Files generators to analyze the structure of directories and files in the Git config repository. With that approach, we can propose a standardization in the configuration structure across the whole organization and propagate it similarly for all the applications deployed in the Kubernetes clusters. Everything can be easily managed at the cluster admin level with the single global Argo CD ApplicationSet that accesses many different repositories with configuration.

The post The Art of Argo CD ApplicationSet Generators with Kubernetes appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2025/03/20/the-art-of-argo-cd-applicationset-generators-with-kubernetes/feed/ 2 15624 Azure DevOps with OpenShift https://piotrminkowski.com/2024/09/12/azure-devops-with-openshift/ https://piotrminkowski.com/2024/09/12/azure-devops-with-openshift/#comments Thu, 12 Sep 2024 10:38:24 +0000 https://piotrminkowski.com/?p=15375 This article will teach you how to integrate Azure DevOps with the OpenShift cluster to build and deploy your app there. You will learn how to run Azure Pipelines self-hosted agents on OpenShift and use the oc client from your pipelines. If you are interested in Azure DevOps you can read my previous article about […]

The post Azure DevOps with OpenShift appeared first on Piotr's TechBlog.

]]> This article will teach you how to integrate Azure DevOps with the OpenShift cluster to build and deploy your app there. You will learn how to run Azure Pipelines self-hosted agents on OpenShift and use the oc client from your pipelines. If you are interested in Azure DevOps you can read my previous article about that platform and Terraform used together to prepare the environment and run the Spring Boot app on the Azure cloud.

Before we begin, let me clarify some things and explain my decisions. If you were searching for something about Azure DevOps and OpenShift integration, you came across several articles about the Red Hat Azure DevOps extension for OpenShift. I won’t use that extension. In my opinion, it is not actively developed right now, and therefore it provides some limitations that may complicate our integration process. On the other hand, it won’t offer many useful features, so we can do just as well without it.

You will also find articles that show how to prepare a self-hosted agent image based on the Red Hat dotnet-runtime as a base image (e.g. this one). I also won’t use it. Instead, I’m going to leverage the image built on top of UBI9 provided by the tool called Blue Agent (formerly Azure Pipelines Agent). It is a self-hosted Azure Pipelines agent for Kubernetes, easy to run, secure, and auto-scaled. We will have to modify that image slightly, but more details later.

Prerequisites

In order to proceed with the exercise, we need an active subscription to the Azure cloud and an instance of Azure DevOps. We also have to run the OpenShift cluster, which is accessible for our pipelines running on Azure DevOps. I’m running that cluster also on the Azure cloud using the Azure Red Hat OpenShift managed service (ARO). The details about Azure DevOps creation or OpenShift installation are out of the scope of that article.

Source Code

If you would like to try this exercise by yourself, you may always take a look at my source code. Today you will have to clone two sample Git repositories. The first one contains a sample Spring Boot app used in our exercise. We will try to build that app with Azure Pipelines and deploy it on OpenShift. The second repository is a fork of the official Blue Agent repository. It contains a new version of the Dockerfile for our sample self-agent image based on Red Hat UBI9. Once you clone both of these repositories, you just need to follow my instructions.

Azure DevOps Self-Hosted Agent on OpenShift with Blue Agent

Build the Agent Image

In this section, we will build the image of the self-agent based on UBI9. Then, we will run it on the OpenShift cluster. We need to open the Dockerfile in the repository located in the src/docker/Dockerfile-ubi9 path. We don’t need to change much inside that file. It contains several clients, e.g. for AWS or Azure interaction. We will include the line for installing the oc client, which allows us to interact with OpenShift.

RUN curl -s https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz -o - | tar zxvf - -C /usr/bin/
Dockerfile

After that, we need to build a new image. You can completely omit this step and pull the final image published on my Docker account and available under the piomin/blue-agent:ubi9 tag. 

$ docker build -t piomin/blue-agent:ubi9 -f Dockerfile-ubi9 \
  --build-arg JQ_VERSION=1.6 \
  --build-arg AZURE_CLI_VERSION=2.63.0 \
  --build-arg AWS_CLI_VERSION=2.17.42 \
  --build-arg GCLOUD_CLI_VERSION=490.0.0 \
  --build-arg POWERSHELL_VERSION=7.2.23 \
  --build-arg TINI_VERSION=0.19.0 \
  --build-arg BUILDKIT_VERSION=0.15.2  \
  --build-arg AZP_AGENT_VERSION=3.243.1 \
  --build-arg ROOTLESSKIT_VERSION=2.3.1 \
  --build-arg GO_VERSION=1.22.7 \
  --build-arg YQ_VERSION=4.44.3 .
ShellSession

We can use a Helm chart to install Blue Agent on Kubernetes. This project is still under active development. I could not customize it with parameters to prepare a chart for installing Blue Agent on OpenShift. So, I just set some of them inside the values.yaml file:

pipelines:
  organizationURL: https://dev.azure.com/pminkows
  personalAccessToken: <AZURE_DEVOPS_PERSONAL_ACCESS_TOKEN>
  poolName: Default

image:
  repository: piomin/blue-agent
  version: 2
  flavor: ubi9
YAML

Deploy Agent on OpenShift

We can generate the YAML manifests for the defined parameters without installing it with the following command:

$ helm template -f blue-agent-values.yaml --dry-run .
ShellSession

In order to run it on OpenShift, we need to customize the Deployment object. First of all, I had to grant the privileged SCC (Security Context Constraint) to the container and remove some fields from the securityContext section. Here’s our Deployment object. It refers to the objects previously generated by the Helm chart: agent-blue-agent ServiceAccount and the Secret with the same name.

kind: Deployment
apiVersion: apps/v1
metadata:
  name: agent-blue-agent    
  labels:
    app.kubernetes.io/component: agent
    app.kubernetes.io/instance: agent
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: blue-agent
    app.kubernetes.io/part-of: blue-agent
    app.kubernetes.io/version: 3.243.1
    helm.sh/chart: blue-agent-7.0.3
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/instance: agent
      app.kubernetes.io/name: blue-agent
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: agent
        app.kubernetes.io/name: blue-agent
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: 'false'
    spec:
      nodeSelector:
        kubernetes.io/os: linux
      restartPolicy: Always
      serviceAccountName: agent-blue-agent
      schedulerName: default-scheduler
      terminationGracePeriodSeconds: 3600
      securityContext: {}
      containers:
        - resources:
            limits:
              cpu: '2'
              ephemeral-storage: 8Gi
              memory: 4Gi
            requests:
              cpu: '1'
              ephemeral-storage: 2Gi
              memory: 2Gi
          terminationMessagePath: /dev/termination-log
          lifecycle:
            preStop:
              exec:
                command:
                  - bash
                  - '-c'
                  - ''
                  - 'rm -rf ${AZP_WORK};'
                  - 'rm -rf ${TMPDIR};'
          name: azp-agent
          env:
            - name: AGENT_DIAGLOGPATH
              value: /app-root/azp-logs
            - name: VSO_AGENT_IGNORE
              value: AZP_TOKEN
            - name: AGENT_ALLOW_RUNASROOT
              value: '1'
            - name: AZP_AGENT_NAME
              valueFrom:
                fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.name
            - name: AZP_URL
              valueFrom:
                secretKeyRef:
                  name: agent-blue-agent
                  key: organizationURL
            - name: AZP_POOL
              value: Default
            - name: AZP_TOKEN
              valueFrom:
                secretKeyRef:
                  name: agent-blue-agent
                  key: personalAccessToken
            - name: flavor_ubi9
            - name: version_7.0.3
          securityContext:
            privileged: true
          imagePullPolicy: Always
          volumeMounts:
            - name: azp-logs
              mountPath: /app-root/azp-logs
            - name: azp-work
              mountPath: /app-root/azp-work
            - name: local-tmp
              mountPath: /app-root/.local/tmp
          terminationMessagePolicy: File
          image: 'piomin/blue-agent:ubi9'
      serviceAccount: agent-blue-agent
      volumes:
        - name: azp-logs
          emptyDir:
            sizeLimit: 1Gi
        - name: azp-work
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 10Gi
                storageClassName: managed-csi
                volumeMode: Filesystem
        - name: local-tmp
          ephemeral:
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 1Gi
                storageClassName: managed-csi
                volumeMode: Filesystem
      dnsPolicy: ClusterFirst
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 25%
      maxSurge: 50%
  revisionHistoryLimit: 10
  progressDeadlineSeconds: 600
YAML

After removing the --dry-run option from the helm template command we can install the solution on OpenShift. It’s worth noting that there were some problems with the Deployment object in the Helm chart, so we will apply it directly later.

azure-devops-openshift-blue-agent

Before applying Deployment we have to add privileged SCC to the ServiceAccount used by the agent. By the way, adding privileged SCC is not the best approach to managing containers on OpenShift. However, the oc client installed in the image creates a directory during the login procedure

$ oc adm policy add-scc-to-user privileged -z agent-blue-agent
ShellSession

Once we deploy an agent on OpenShift we should see three running pods. They are trying to connect to the Default pool defined in the Azure DevOps for our self-hosted agent.

$ oc get po
NAME                                READY   STATUS    RESTARTS   AGE
agent-blue-agent-5458b9b76d-2gk2z   1/1     Running   0          13h
agent-blue-agent-5458b9b76d-nwcxh   1/1     Running   0          13h
agent-blue-agent-5458b9b76d-tcpnp   1/1     Running   0          13h
ShellSession

Connect Self-hosted Agent to Azure DevOps

Let’s switch to the Azure DevOps instance. Then, we should go to our project. After that, we need to go to the Organization Settings -> Agent pools. We need to create a new agent pool. The pool name depends on the name configured for the Blue Agent deployed on OpenShift. In our case, this name is Default.

Once we create a pool we should see all three instances of agents in the following list.

azure-devops-openshift-agents

We can switch to the OpenShift once again. Then, let’s take a look at the logs printed out by one of the agents. As you see, it has been started successfully and listens for the incoming jobs.

If agents are running on OpenShift and they successfully connect with Azure DevOps, we finished the first part of our exercise. Now, we can create a pipeline for our sample Spring Boot app.

Create Azure DevOps Pipeline for OpenShift

Azure Pipeline Definition

Firstly, we need to go to the sample-spring-boot-web repository. The pipeline is configured in the azure-pipelines.yml file in the repository root directory. Let’s take a look at it. It’s very simple simple. I’m just using the Azure DevOps Command-Line task to interact with OpenShift through the oc client. Of course, it has to define the target agent pool used for running its jobs (1). In the first step, we need to log in to the OpenShift cluster (2). We can use the internal address of the Kubernetes Service, since the agent is running inside the cluster. All the actions should be performed inside our sample myapp project (3). We need to create BuildConfig and Deployment objects using the templates from the repository (4). The pipeline uses its BuildId parameter to the output image. Finally, it starts the build configured by the previously applied BuildConfig object (5).

trigger:
- master

# (1)
pool:
  name: Default

steps:
# (2)
- task: CmdLine@2
  inputs:
    script: 'oc login https://172.30.0.1:443 -u kubeadmin -p $(OCP_PASSWORD) --insecure-skip-tls-verify=true'
  displayName: Login to OpenShift
# (3)
- task: CmdLine@2
  inputs:
    script: 'oc project myapp'
  displayName: Switch to project
# (4)
- task: CmdLine@2
  inputs:
    script: 'oc process -f ocp/openshift.yaml -o yaml -p IMAGE_TAG=v1.0-$(Build.BuildId) -p NAMESPACE=myapp | oc apply -f -'
  displayName: Create build
# (5)
- task: CmdLine@2
  inputs:
    script: 'oc start-build sample-spring-boot-web-bc -w'
    failOnStderr: true
  displayName: Start build
  timeoutInMinutes: 5
- task: CmdLine@2
  inputs:
    script: 'oc status'
  displayName: Check status
YAML

Integrate Azure Pipelines with OpenShift

We use the OpenShift Template object for defining the YAML manifest with Deployment and BuildConfig. The BuildConfig object typically manages the image build process on OpenShift. In order to build the image directly from the source code, it uses the source-2-image (S2I) tool. We need to set at least three parameters to configure the process properly. The first of them is the address of the output image (1). We can use the internal OpenShift repository available at image-registry.openshift-image-registry.svc:5000 or any other external registry like Quay or Docker Hub. We should also set the name of the builder image (2). Our app requires at least Java 21. Of course, the process requires the source code repository as the input (3). At the same time, we define the Deployment object. It uses the image previously built by the BuildConfig object (4). The whole template takes two input parameters: IMAGE_TAG and NAMESPACE (5).

ind: Template
apiVersion: template.openshift.io/v1
metadata:
  name: sample-spring-boot-web-tmpl
objects:
  - kind: BuildConfig
    apiVersion: build.openshift.io/v1
    metadata:
      name: sample-spring-boot-web-bc
      labels:
        build: sample-spring-boot-web-bc
    spec:
      # (1)
      output:
        to:
          kind: DockerImage
          name: 'image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/sample-spring-boot-web:${IMAGE_TAG}'
      # (2)
      strategy:
        type: Source
        sourceStrategy:
          from:
            kind: ImageStreamTag
            namespace: openshift
            name: 'openjdk-21:stable'
      # (3)
      source:
        type: Git
        git:
          uri: 'https://github.com/piomin/sample-spring-boot-web.git'
  - kind: Deployment
    apiVersion: apps/v1
    metadata:
      name: sample-spring-boot-web
      labels:
        app: sample-spring-boot-web
        app.kubernetes.io/component: sample-spring-boot-web
        app.kubernetes.io/instance: sample-spring-boot-web
    spec:
      replicas: 1
      selector:
        matchLabels:
          deployment: sample-spring-boot-web
      template:
        metadata:
          labels:
            deployment: sample-spring-boot-web
        spec:
          containers:
            - name: sample-spring-boot-web
              # (4)
              image: 'image-registry.openshift-image-registry.svc:5000/${NAMESPACE}/sample-spring-boot-web:${IMAGE_TAG}'
              ports:
                - containerPort: 8080
                  protocol: TCP
                - containerPort: 8443
                  protocol: TCP
# (5)
parameters:
  - name: IMAGE_TAG
    displayName: Image tag
    description: The output image tag
    value: v1.0
    required: true
  - name: NAMESPACE
    displayName: Namespace
    description: The OpenShift Namespace where the ImageStream resides
    value: openshift
YAML

Currently, OpenShift doesn’t provide OpenJDK 21 image by default. So we need to manually import it to our cluster before running the pipeline.

$ oc import-image openjdk-21:stable \
  --from=registry.access.redhat.com/ubi9/openjdk-21:1.20-2.1725851045 \
  --confirm
ShellSession

Now, we can create a pipeline in Azure DevOps. In order to do it, we need to go to the Pipelines section, and then click the New pipeline button. After that, we just need to pass the address of our repository with the pipeline definition.

Our pipeline requires the OCP_PASSWORD input parameter with the OpenShift admin user password. We can set it as the pipeline secret variable. In order to do that, we need to edit the pipeline and then click the Variables button.

Run the Pipeline

Finally, we can run our pipeline. If everything finishes successfully, the status of the job is Success. It takes around one minute to execute all the steps defined in the pipeline.

We can see detailed logs for each step.

azure-devops-openshift-pipeline-logs

Each time, we run a pipeline a new build on OpenShift starts. Note, that the pipeline updates the BuildConfig object with the new version of the output image.

azure-devops-openshift-builds

We can take a look at detailed logs of each build. Within such a build OpenShift starts a new pod, which performs the whole process. It uses the S2I approach for building the image from the source code and pushing that image to the internal OpenShift registry.

Finally, let’s take a look at the Deployment. As you see, it was reloaded with the latest version of the image and worked fine.

$ oc describe deploy sample-spring-boot-web -n myapp
Name:                   sample-spring-boot-web
Namespace:              myapp
CreationTimestamp:      Wed, 11 Sep 2024 16:09:11 +0200
Labels:                 app=sample-spring-boot-web
                        app.kubernetes.io/component=sample-spring-boot-web
                        app.kubernetes.io/instance=sample-spring-boot-web
Annotations:            deployment.kubernetes.io/revision: 3
Selector:               deployment=sample-spring-boot-web
Replicas:               1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  25% max unavailable, 25% max surge
Pod Template:
  Labels:  deployment=sample-spring-boot-web
  Containers:
   sample-spring-boot-web:
    Image:        image-registry.openshift-image-registry.svc:5000/myapp/sample-spring-boot-web:v1.0-134
    Ports:        8080/TCP, 8443/TCP
    Host Ports:   0/TCP, 0/TCP
    Environment:  <none>
    Mounts:       <none>
  Volumes:        <none>
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
OldReplicaSets:  sample-spring-boot-web-78559697c9 (0/0 replicas created), sample-spring-boot-web-7985d6b844 (0/0 replicas created)
NewReplicaSet:   sample-spring-boot-web-5584447f5d (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  19m   deployment-controller  Scaled up replica set sample-spring-boot-web-5584447f5d to 1
  Normal  ScalingReplicaSet  17m   deployment-controller  Scaled down replica set sample-spring-boot-web-7985d6b844 to 0 from 1
ShellSession

By the way, Azure Pipelines jobs are “load-balanced” between the agents. Here’s the name of the agent used for the first pipeline run.

Here’s the name of the agent used for the second pipeline run.

There are three instances of the agent running on OpenShift. So, Azure DevOps can process max 3 pipeline runs. By the way, we could enable autoscaling for Blue Agent based on KEDA.

azure-devops-openshift-jobs

Final Thoughts

In this article, I showed the simplest and most OpenShift-native approach to building CI/CD pipelines on Azure DevOps. We just need to use the oc client on the agent image and the OpenShift BuildConfig object to orchestrate the whole process of building and deploying the Spring Boot app on the cluster. Of course, we could implement the same process in several different. For example, we could leverage the plugins for Kubernetes and completely omit the tools provided by OpenShift. On the other hand, it is possible to use Argo CD for the delivery phase, and Azure Pipelines for the integration phase.

The post Azure DevOps with OpenShift appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/09/12/azure-devops-with-openshift/feed/ 2 15375 IDP on OpenShift with Red Hat Developer Hub https://piotrminkowski.com/2024/07/04/idp-on-openshift-with-red-hat-developer-hub/ https://piotrminkowski.com/2024/07/04/idp-on-openshift-with-red-hat-developer-hub/#comments Thu, 04 Jul 2024 06:38:46 +0000 https://piotrminkowski.com/?p=15316 This article will teach you how to build IDP (Internal Developer Platform) on the OpenShift cluster with the Red Hat Developer Hub solution. Red Hat Developer Hub is a developer portal built on top of the Backstage project. It simplifies the installation and configuration of Backstage in the Kubernetes-native environment through the operator and dynamic […]

The post IDP on OpenShift with Red Hat Developer Hub appeared first on Piotr's TechBlog.

]]> This article will teach you how to build IDP (Internal Developer Platform) on the OpenShift cluster with the Red Hat Developer Hub solution. Red Hat Developer Hub is a developer portal built on top of the Backstage project. It simplifies the installation and configuration of Backstage in the Kubernetes-native environment through the operator and dynamic plugins. You can compare that process with the open-source Backstage installation on Kubernetes described in my previous article. If you need a quick intro to the Backstage platform you can also read my article Getting Started with Backstage.

A platform team manages an Internal Developer Platform (IDP) to build golden paths and enable developer self-service in the organization. It may consist of many different tools and solutions. On the other hand, Internal Developer Portals serve as the GUI interface through which developers can discover and access internal developer platform capabilities. In the context of OpenShift, Red Hat Developer Hub simplifies the adoption of several cluster services for developers (e.g. Kubernates-native CI/CD tools). Today, you will learn how to integrate Developer Hub with OpenShift Pipelines (Tekton) and OpenShift GitOps (Argo CD). Let’s begin!

Source Code

If you would like to try it by yourself, you may always take a look at my source code. Our sample GitHub repository contains software templates written in the Backstage technology called Skaffolder. In this article, we will analyze a template dedicated to OpenShift available in the templates/spring-boot-basic-on-openshift directory. After cloning this repository, you should just follow my instructions.

Here’s the structure of our repository. It is pretty similar to the template for Spring Boot on Kubernetes described in my previous article about Backstage. Besides the template, it also contains the Argo CD and Tekton templates with YAML deployment manifests to apply on OpenShift.

.
├── README.md
├── backstage-templates.iml
├── skeletons
│   └── argocd
│       ├── argocd
│       │   └── app.yaml
│       └── manifests
│           ├── deployment.yaml
│           ├── pipeline.yaml
│           ├── service.yaml
│           ├── tasks.yaml
│           └── trigger.yaml
├── templates
│   └── spring-boot-basic-on-openshift
│       ├── skeleton
│       │   ├── README.md
│       │   ├── catalog-info.yaml
│       │   ├── devfile.yaml
│       │   ├── k8s
│       │   │   └── deployment.yaml
│       │   ├── pom.xml
│       │   ├── renovate.json
│       │   ├── skaffold.yaml
│       │   └── src
│       │       ├── main
│       │       │   ├── java
│       │       │   │   └── ${{values.javaPackage}}
│       │       │   │       ├── Application.java
│       │       │   │       ├── controller
│       │       │   │       │   └── ${{values.domainName}}Controller.java
│       │       │   │       └── domain
│       │       │   │           └── ${{values.domainName}}.java
│       │       │   └── resources
│       │       │       └── application.yml
│       │       └── test
│       │           ├── java
│       │           │   └── ${{values.javaPackage}}
│       │           │       └── ${{values.domainName}}ControllerTests.java
│       │           └── resources
│       │               └── k6
│       │                   └── load-tests-add.js
│       └── template.yaml
└── templates.yaml
ShellSession

Prerequisites

Before we start the exercise, we need to prepare our OpenShift cluster. We have to install three operators: OpenShift GitOps (Argo CD), OpenShift Pipelines (Tekton), and of course, Red Hat Developer Hub.

Once we install the OpenShift GitOps, it automatically creates an instance of Argo CD in the openshift-gitops namespace. That instance is managed by the openshift-gitops ArgoCD CRD object.

We need to override some default configuration settings there. Then, we will add a new user backstage with privileges for creating applications, and projects and generating API keys. Finally, we change the default TLS termination method for Argo CD Route to reencrypt. It is required to integrate with the Backstage Argo CD plugin. We also add the demo namespace as an additional namespace to place Argo CD applications in.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  sourceNamespaces:
    - demo
  server:
    ...
    route:
      enabled: true
      tls:
        termination: reencrypt
  ...
  rbac:
    defaultPolicy: ''
    policy: |
      g, system:cluster-admins, role:admin
      g, cluster-admins, role:admin
      p, backstage, applications, *, */*, allow
      p, backstage, projects, *, *, allow
    scopes: '[groups]'
  extraConfig:
    accounts.backstage: 'apiKey, login'
YAML

In order to generate the apiKey for the backstage user, we need to sign in to Argo CD with the argocd CLI as the admin user. Then, we should run the following command for the backstage account and export the generated token as the ARGOCD_TOKEN env variable:

$ export ARGOCD_TOKEN=$(argocd account generate-token --account backstage)
ShellSession

Finally, let’s obtain the long-lived API token for Kubernetes by creating a secret:

apiVersion: v1
kind: Secret
metadata:
  name: default-token
  namespace: backstage
  annotations:
    kubernetes.io/service-account.name: default
type: kubernetes.io/service-account-token
YAML

Then, we can copy and export it as the OPENSHIFT_TOKEN environment variable with the following command:

$ export OPENSHIFT_TOKEN=$(kubectl get secret default-token -o go-template='{{.data.token | base64decode}}')
ShellSession

Let’s add the ClusterRole view to the Backstage default ServiceAccount:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: default-view-backstage
subjects:
- kind: ServiceAccount
  name: default
  namespace: backstage
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io
YAML

Configure Red Hat Developer Hub on OpenShift

After installing the Red Hat Developer Hub operator on OpenShift, we can use the Backstage CRD to create and configure a new instance of the portal. Firstly, we will override some default settings using the app-config-rhdh ConfigMap (1). Then we will provide some additional secrets like tokens to the third-party party tools in the app-secrets-rhdh Secret (2). Finally, we will install and configure several useful plugins with the dynamic-plugins-rhdh ConfigMap (3). Here is the required configuration in the Backstage CRD.

apiVersion: rhdh.redhat.com/v1alpha1
kind: Backstage
metadata:
  name: developer-hub
  namespace: backstage
spec:
  application:
    appConfig:
      configMaps:
        # (1)
        - name: app-config-rhdh
      mountPath: /opt/app-root/src
    # (3)
    dynamicPluginsConfigMapName: dynamic-plugins-rhdh
    extraEnvs:
      secrets:
        # (2)
        - name: app-secrets-rhdh
    extraFiles:
      mountPath: /opt/app-root/src
    replicas: 1
    route:
      enabled: true
  database:
    enableLocalDb: true
YAML

Override Default Configuration Settings

The instance of Backstage will be deployed in the backstage namespace. Since OpenShift exposes it as a Route, the address of the portal on my cluster is https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io (1). Firstly, we need to override that address in the app settings. Then we need to enable authentication through the GitHub OAuth with the GitHub Red Hat Developer Hub application (2). Then, we should set the proxy endpoint to integrate with Sonarqube through the HTTP Request Action plugin (3). Our instance of Backstage should also read templates from the particular URL location (4) and should be able to create repositories in GitHub (5).

kind: ConfigMap
apiVersion: v1
metadata:
  name: app-config-rhdh
  namespace: backstage
data:
  app-config-rhdh.yaml: |

    # (1)
    app:
     baseUrl: https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io

    backend:
      baseUrl: https://backstage-developer-hub-backstage.apps.piomin.eastus.aroapp.io

    # (2)
    auth:
      environment: development
      providers:
        github:
          development:
            clientId: ${GITHUB_CLIENT_ID}
            clientSecret: ${GITHUB_CLIENT_SECRET}

    # (3)
    proxy:
      endpoints:
        /sonarqube:
          target: ${SONARQUBE_URL}/api
          allowedMethods: ['GET', 'POST']
          auth: "${SONARQUBE_TOKEN}:"

    # (4)
    catalog:
      rules:
        - allow: [Component, System, API, Resource, Location, Template]
      locations:
        - type: url
          target: https://github.com/piomin/backstage-templates/blob/master/templates.yaml

    # (5)
    integrations:
      github:
        - host: github.com
          token: ${GITHUB_TOKEN}
          
    sonarqube:
      baseUrl: https://sonarcloud.io
      apiKey: ${SONARQUBE_TOKEN}
YAML

Integrate with GitHub

In order to use GitHub auth we need to register a new app there. Go to the “Settings > Developer Settings > New GitHub App” in your GitHub account. Then, put the address of your Developer Hub instance in the “Homepage URL” field and the callback address in the “Callback URL” field (base URL + /api/auth/github/handler/frame)

Then, let’s edit our GitHub app to generate a new secret as shown below.

The client ID and secret should be saved as the environment variables for future use. Note, that we also need to generate a new personal access token (“Settings > Developer Settings > Personal Access Tokens”).

export GITHUB_CLIENT_ID=<YOUR_GITHUB_CLIENT_ID>
exporg GITHUB_CLIENT_SECRET=<YOUR_GITHUB_CLIENT_SECRET>
export GITHUB_TOKEN=<YOUR_GITHUB_TOKEN>
ShellSession

We already have a full set of required tokens and access keys, so we can create the app-secrets-rhdh Secret to store them on our OpenShift cluster.

$ oc create secret generic app-secrets-rhdh -n backstage \
  --from-literal=GITHUB_CLIENT_ID=${GITHUB_CLIENT_ID} \
  --from-literal=GITHUB_CLIENT_SECRET=${GITHUB_CLIENT_SECRET} \
  --from-literal=GITHUB_TOKEN=${GITHUB_TOKEN} \
  --from-literal=SONARQUBE_TOKEN=${SONARQUBE_TOKEN} \
  --from-literal=SONARQUBE_URL=https://sonarcloud.io \
  --from-literal=ARGOCD_TOKEN=${ARGOCD_TOKEN}
ShellSession

Install and Configure Plugins

Finally, we can proceed to the plugins installation. Do you remember how we can do it with the open-source Backstage on Kubernetes? I described it in my previous article. Red Hat Developer Hub drastically simplifies that process with the idea of dynamic plugins. This approach is based on the Janus IDP project. Developer Hub on OpenShift comes with ~60 preinstalled plugins that allow us to integrate various third-party tools including Sonarqube, Argo CD, Tekton, Kubernetes, or GitHub. Some of them are enabled by default, some others are installed but disabled. We can verify it after signing in to the Backstage UI. We can easily verify it in the “Administration” section:

Let’s take a look at the ConfigMap which contains a list of plugins to activate. It is pretty huge since we also provide configuration for the frontend plugins. Some plugins are optional. From the perspective of our exercise goal we need to activate at least the following list of plugins:

  • janus-idp-backstage-plugin-argocd – to view the status of Argo CD synchronization in the UI
  • janus-idp-backstage-plugin-tekton – to view the status of Tekton pipelines in the UI
  • backstage-plugin-kubernetes-backend-dynamic – to integrate with the Kubernetes cluster
  • backstage-plugin-kubernetes – to view the Kubernetes app pods in the UI
  • backstage-plugin-sonarqube – to view the status of the Sonarqube scan in the UI
  • roadiehq-backstage-plugin-argo-cd-backend-dynamic – to create the Argo CD Application from the template
kind: ConfigMap
apiVersion: v1
metadata:
  name: dynamic-plugins-rhdh
  namespace: backstage
data:
  dynamic-plugins.yaml: |
    includes:
      - dynamic-plugins.default.yaml
    plugins:
      - package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-github-pull-requests
        disabled: true
        pluginConfig:
          dynamicPlugins:
            frontend:
              roadiehq.backstage-plugin-github-pull-requests:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntityGithubPullRequestsOverviewCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 4"
                          md: "span 6"
                          xs: "span 12"
                      if:
                        allOf:
                          - isGithubPullRequestsAvailable
                  - mountPoint: entity.page.pull-requests/cards
                    importName: EntityGithubPullRequestsContent
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isGithubPullRequestsAvailable
      - package: './dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-argocd'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-argocd:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: ArgocdDeploymentSummary
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 8"
                          xs: "span 12"
                      if:
                        allOf:
                          - isArgocdConfigured
                  - mountPoint: entity.page.cd/cards
                    importName: ArgocdDeploymentLifecycle
                    config:
                      layout:
                        gridColumn: '1 / -1'
                      if:
                        allOf:
                          - isArgocdConfigured
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-tekton'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-tekton:
                mountPoints:
                  - mountPoint: entity.page.ci/cards
                    importName: TektonCI
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isTektonCIAvailable
      - package: './dynamic-plugins/dist/janus-idp-backstage-plugin-topology'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              janus-idp.backstage-plugin-topology:
                mountPoints:
                  - mountPoint: entity.page.topology/cards
                    importName: TopologyPage
                    config:
                      layout:
                        gridColumn: "1 / -1"
                        height: 75vh
                      if:
                        anyOf:
                          - hasAnnotation: backstage.io/kubernetes-id
                          - hasAnnotation: backstage.io/kubernetes-namespace
      - package: './dynamic-plugins/dist/janus-idp-backstage-scaffolder-backend-module-sonarqube-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/backstage-plugin-kubernetes-backend-dynamic'
        disabled: false
        pluginConfig:
          kubernetes:
            customResources:
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'pipelines'
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'pipelineruns'
            - group: 'tekton.dev'
              apiVersion: 'v1beta1'
              plural: 'taskruns'
            - group: 'route.openshift.io'
              apiVersion: 'v1'
              plural: 'routes'
            serviceLocatorMethod:
              type: 'multiTenant'
            clusterLocatorMethods:
              - type: 'config'
                clusters:
                  - name: ocp
                    url: https://api.piomin.eastus.aroapp.io:6443
                    authProvider: 'serviceAccount'
                    skipTLSVerify: true
                    skipMetricsLookup: true
                    serviceAccountToken: ${OPENSHIFT_TOKEN}
      - package: './dynamic-plugins/dist/backstage-plugin-kubernetes'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              backstage.plugin-kubernetes:
                mountPoints:
                  - mountPoint: entity.page.kubernetes/cards
                    importName: EntityKubernetesContent
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        anyOf:
                          - hasAnnotation: backstage.io/kubernetes-id
                          - hasAnnotation: backstage.io/kubernetes-namespace
      - package: './dynamic-plugins/dist/backstage-plugin-sonarqube'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              backstage.plugin-sonarqube:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntitySonarQubeCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 4"
                          md: "span 6"
                          xs: "span 12"
                      if:
                        allOf:
                          - isSonarQubeAvailable
      - package: './dynamic-plugins/dist/backstage-plugin-sonarqube-backend-dynamic'
        disabled: false
        pluginConfig: {}
      - package: './dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd'
        disabled: false
        pluginConfig:
          dynamicPlugins:
            frontend:
              roadiehq.backstage-plugin-argo-cd:
                mountPoints:
                  - mountPoint: entity.page.overview/cards
                    importName: EntityArgoCDOverviewCard
                    config:
                      layout:
                        gridColumnEnd:
                          lg: "span 8"
                          xs: "span 12"
                      if:
                        allOf:
                          - isArgocdAvailable
                  - mountPoint: entity.page.cd/cards
                    importName: EntityArgoCDHistoryCard
                    config:
                      layout:
                        gridColumn: "1 / -1"
                      if:
                        allOf:
                          - isArgocdAvailable
      - package: './dynamic-plugins/dist/roadiehq-scaffolder-backend-argocd-dynamic'
        disabled: false
        pluginConfig: {}
      - package: ./dynamic-plugins/dist/roadiehq-backstage-plugin-argo-cd-backend-dynamic
        disabled: false
        pluginConfig:
          argocd:
            appLocatorMethods:
              - type: 'config'
                instances:
                  - name: main
                    url: "https://openshift-gitops-server-openshift-gitops.apps.piomin.eastus.aroapp.io"
                    token: "${ARGOCD_TOKEN}"
YAML

Once we provide the whole configuration described above, we are ready to proceed with our Skaffolder template for the sample Spring Boot app.

Prepare Backstage Template for OpenShift

Our template consists of several steps. Firstly, we generate the app source code and push it to the app repository. Then, we register the component in the Backstage catalog and create a configuration repository for Argo CD. It contains app deployment manifests and the definition of the Tekton pipeline and trigger. Trigger is exposed as the Route, and can be called from the GitHub repository through the webhook. Finally, we are creating the project in Sonarcloud, the application in Argo CD, and registering the webhook in the GitHub app repository. Here’s our Skaffolder template.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: spring-boot-basic-on-openshift-template
  title: Create a Spring Boot app for OpenShift
  description: Create a Spring Boot app for OpenShift
  tags:
    - spring-boot
    - java
    - maven
    - tekton
    - renovate
    - sonarqube
    - openshift
    - argocd
spec:
  owner: piomin
  system: microservices
  type: service

  parameters:
    - title: Provide information about the new component
      required:
        - orgName
        - appName
        - domainName
        - repoBranchName
        - groupId
        - javaPackage
        - apiPath
        - namespace
        - description
        - registryUrl
        - clusterDomain
      properties:
        orgName:
          title: Organization name
          type: string
          default: piomin
        appName:
          title: App name
          type: string
          default: sample-spring-boot-app-openshift
        domainName:
          title: Name of the domain object
          type: string
          default: Person
        repoBranchName:
          title: Name of the branch in the Git repository
          type: string
          default: master
        groupId:
          title: Maven Group ID
          type: string
          default: pl.piomin.services
        javaPackage:
          title: Java package directory
          type: string
          default: pl/piomin/services
        apiPath:
          title: REST API path
          type: string
          default: /api/v1
        namespace:
          title: The target namespace on Kubernetes
          type: string
          default: demo
        description:
          title: Description
          type: string
          default: Spring Boot App Generated by Backstage
        registryUrl:
          title: Registry URL
          type: string
          default: image-registry.openshift-image-registry.svc:5000
        clusterDomain:
          title: OpenShift Cluster Domain
          type: string
          default: .apps.piomin.eastus.aroapp.io
  steps:
    - id: sourceCodeTemplate
      name: Generating the Source Code Component
      action: fetch:template
      input:
        url: ./skeleton
        values:
          orgName: ${{ parameters.orgName }}
          appName: ${{ parameters.appName }}
          domainName: ${{ parameters.domainName }}
          groupId: ${{ parameters.groupId }}
          javaPackage: ${{ parameters.javaPackage }}
          apiPath: ${{ parameters.apiPath }}
          namespace: ${{ parameters.namespace }}

    - id: publish
      name: Publishing to the Source Code Repository
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=${{ parameters.orgName }}&repo=${{ parameters.appName }}
        defaultBranch: ${{ parameters.repoBranchName }}
        repoVisibility: public

    - id: register
      name: Registering the Catalog Info Component
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

    - id: configCodeTemplate
      name: Generating the Config Code Component
      action: fetch:template
      input:
        url: ../../skeletons/argocd
        values:
          orgName: ${{ parameters.orgName }}
          appName: ${{ parameters.appName }}
          registryUrl: ${{ parameters.registryUrl }}
          namespace: ${{ parameters.namespace }}
          repoBranchName: ${{ parameters.repoBranchName }}
        targetPath: ./gitops

    - id: publish
      name: Publishing to the Config Code Repository
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=${{ parameters.orgName }}&repo=${{ parameters.appName }}-config
        defaultBranch: ${{ parameters.repoBranchName }}
        sourcePath: ./gitops
        repoVisibility: public

    - id: sonarqube
      name: Create a new project on Sonarcloud
      action: http:backstage:request
      input:
        method: 'POST'
        path: '/proxy/sonarqube/projects/create?name=${{ parameters.appName }}&organization=${{ parameters.orgName }}&project=${{ parameters.orgName }}_${{ parameters.appName }}'
        headers:
          content-type: 'application/json'

    - id: create-argocd-resources
      name: Create ArgoCD Resources
      action: argocd:create-resources
      input:
        appName: ${{ parameters.appName }}
        argoInstance: main
        namespace: ${{ parameters.namespace }}
        repoUrl: https://github.com/${{ parameters.orgName }}/${{ parameters.appName }}-config.git
        path: 'manifests'

    - id: create-webhook
      name: Create GitHub Webhook
      action: github:webhook
      input:
        repoUrl: github.com?repo=${{ parameters.appName }}&owner=${{ parameters.orgName }}
        webhookUrl: https://el-${{ parameters.appName }}-${{ parameters.namespace }}.${{ parameters.clusterDomain }}

  output:
    links:
      - title: Open the Source Code Repository
        url: ${{ steps.publish.output.remoteUrl }}
      - title: Open the Catalog Info Component
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}
      - title: SonarQube project URL
        url: ${{ steps['create-sonar-project'].output.projectUrl }}
YAML

Define Templates for OpenShift Pipelines

Compared to the article about Backstage on Kubernetes, we use Tekton instead of CircleCI as a build tool. Let’s take a look at the definition of our pipeline. It consists of four steps. In the final step, we use the OpenShift S2I mechanism to build the app image and push it to the local container registry.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: ${{ values.appName }}
  labels:
    backstage.io/kubernetes-id: ${{ values.appName }}
spec:
  params:
    - description: branch
      name: git-revision
      type: string
      default: master
  tasks:
    - name: git-clone
      params:
        - name: url
          value: 'https://github.com/${{ values.orgName }}/${{ values.appName }}.git'
        - name: revision
          value: $(params.git-revision)
        - name: sslVerify
          value: 'false'
      taskRef:
        kind: ClusterTask
        name: git-clone
      workspaces:
        - name: output
          workspace: source-dir
    - name: maven
      params:
        - name: GOALS
          value:
            - test
        - name: PROXY_PROTOCOL
          value: http
        - name: CONTEXT_DIR
          value: .
      runAfter:
        - git-clone
      taskRef:
        kind: ClusterTask
        name: maven
      workspaces:
        - name: source
          workspace: source-dir
        - name: maven-settings
          workspace: maven-settings
    - name: sonarqube
      params:
        - name: SONAR_HOST_URL
          value: 'https://sonarcloud.io'
        - name: SONAR_PROJECT_KEY
          value: ${{ values.appName }}
      runAfter:
        - maven
      taskRef:
        kind: Task
        name: sonarqube-scanner
      workspaces:
        - name: source
          workspace: source-dir
        - name: sonar-settings
          workspace: sonar-settings
    - name: get-version
      params:
        - name: CONTEXT_DIR
          value: .
      runAfter:
        - sonarqube
      taskRef:
        kind: Task
        name: maven-get-project-version
      workspaces:
        - name: source
          workspace: source-dir
    - name: s2i-java
      params:
        - name: PATH_CONTEXT
          value: .
        - name: TLSVERIFY
          value: 'false'
        - name: MAVEN_CLEAR_REPO
          value: 'false'
        - name: IMAGE
          value: >-
            ${{ values.registryUrl }}/${{ values.namespace }}/${{ values.appName }}:$(tasks.get-version.results.version)
      runAfter:
        - get-version
      taskRef:
        kind: ClusterTask
        name: s2i-java
      workspaces:
        - name: source
          workspace: source-dir
  workspaces:
    - name: source-dir
    - name: maven-settings
    - name: sonar-settings
YAML

In order to run the pipeline after creating it, we need to apply the PipelineRun object.

apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: ${{ values.appName }}-init
spec:
  params:
    - name: git-revision
      value: master
  pipelineRef:
    name: ${{ values.appName }}
  serviceAccountName: pipeline
  workspaces:
    - name: source-dir
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteOnce
          resources:
            requests:
              storage: 1Gi
    - name: sonar-settings
      secret:
        secretName: sonarqube-secret-token
    - configMap:
        name: maven-settings
      name: maven-settings
YAML

In order to call the pipeline via the webhook from the app source repository, we also need to create the Tekton TriggerTemplate object. Once we push a change to the target repository, we trigger the run of the Tekton pipeline on the OpenShift cluster.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: ${{ values.appName }}
spec:
  params:
    - default: ${{ values.repoBranchName }}
      description: The git revision
      name: git-revision
    - description: The git repository url
      name: git-repo-url
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: ${{ values.appName }}-run-
      spec:
        params:
          - name: git-revision
            value: $(tt.params.git-revision)
        pipelineRef:
          name: ${{ values.appName }}
        serviceAccountName: pipeline
        workspaces:
          - name: source-dir
            volumeClaimTemplate:
              spec:
                accessModes:
                  - ReadWriteOnce
                resources:
                  requests:
                    storage: 1Gi
          - name: sonar-settings
            secret:
              secretName: sonarqube-secret-token
          - configMap:
              name: maven-settings
            name: maven-settings
YAML

Deploy the app on OpenShift

Here’s the template for the app Deployment object:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ${{ values.appName }}
  labels:
    app: ${{ values.appName }}
    app.kubernetes.io/name: spring-boot
    backstage.io/kubernetes-id: ${{ values.appName }}
spec:
  selector:
    matchLabels:
      app: ${{ values.appName }}
  template:
    metadata:
      labels:
        app: ${{ values.appName }}
        backstage.io/kubernetes-id: ${{ values.appName }}
    spec:
      containers:
        - name: ${{ values.appName }}
          image: ${{ values.registryUrl }}/${{ values.namespace }}/${{ values.appName }}:1.0
          ports:
            - containerPort: 8080
              name: http
          livenessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/liveness
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          readinessProbe:
            httpGet:
              port: 8080
              path: /actuator/health/readiness
              scheme: HTTP
            timeoutSeconds: 1
            periodSeconds: 10
            successThreshold: 1
            failureThreshold: 3
          resources:
            limits:
              memory: 1024Mi
YAML

Here’s the current version of the catalog-info.yaml file. 

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.appName }}
  title: ${{ values.appName }}
  annotations:
    janus-idp.io/tekton: ${{ values.appName }}
    tektonci/build-namespace: ${{ values.namespace }}
    github.com/project-slug: ${{ values.orgName }}/${{ values.appName }}
    sonarqube.org/project-key: ${{ values.orgName }}_${{ values.appName }}
    backstage.io/kubernetes-id: ${{ values.appName }}
    argocd/app-name: ${{ values.appName }}
  tags:
    - spring-boot
    - java
    - maven
    - tekton
    - argocd
    - renovate
    - sonarqube
spec:
  type: service
  owner: piomin
  lifecycle: experimental
YAML

Now, let’s create a new component in Red Hat Developer Hub using our template. In the first step, you should choose the “Create a Spring Boot App for OpenShift” template as shown below.

Then, provide all the parameters in the form. Probably you will have to override the default organization name to your GitHub account name and the address of your OpenShift cluster. Once you make all the required changes click the “Review” button, and then the “Create” button on the next screen. After that, Red Hat Developer Hub creates all the things we need.

After confirmation, Developer Hub redirects to the page with the progress information. There are 8 action steps defined. All of them should be finished successfully. Then, we can just click the “Open the Catalog Info Component” link.

developer-hub-openshift-create

Viewing Component in Red Hat Developer Hub UI

Our app overview tab contains general information about the component registered in Backstage, the status of the Sonarqube scan, and the status of the Argo CD synchronization process. We can switch to the several other available tabs.

developer-hub-openshift-overview

In the “CI” tab, we can see the history of the OpenShift Pipelines runs. We can switch to the logs of each pipeline step by clicking on it.

developer-hub-openshift-ci

If you are familiar with OpenShift, you can recognize that view as a topology view from the OpenShift Console developer perspective. It visualizes all the deployments in the particular namespace.

developer-hub-openshift-topology

In the “CD” tab, we can see the history of Argo CD synchronization operations.

developer-hub-openshift-cd

Final Thoughts

Red Hat Developer Hub simplifies installation and configuration of Backstage in the Kubernetes-native environment. It introduces the idea of dynamic plugins, which can be easily customized in the configuration files. You can compare this approach with my previous article about Backstage on Kubernetes.

The post IDP on OpenShift with Red Hat Developer Hub appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/07/04/idp-on-openshift-with-red-hat-developer-hub/feed/ 2 15316 Migrate from Kubernetes to OpenShift in the GitOps Way https://piotrminkowski.com/2024/04/15/migrate-from-kubernetes-to-openshift-in-the-gitops-way/ https://piotrminkowski.com/2024/04/15/migrate-from-kubernetes-to-openshift-in-the-gitops-way/#comments Mon, 15 Apr 2024 12:09:50 +0000 https://piotrminkowski.com/?p=15190 In this article, you will learn how to migrate your apps from Kubernetes to OpenShift in the GitOps way using tools like Kustomize, Helm, operators, and Argo CD. We will discuss the best practices in that area. This requires us to avoid approaches like starting a pod in the privileged mode. We will focus not […]

The post Migrate from Kubernetes to OpenShift in the GitOps Way appeared first on Piotr's TechBlog.

]]> In this article, you will learn how to migrate your apps from Kubernetes to OpenShift in the GitOps way using tools like Kustomize, Helm, operators, and Argo CD. We will discuss the best practices in that area. This requires us to avoid approaches like starting a pod in the privileged mode. We will focus not just on running your custom apps, but mostly on the popular pieces of cloud-native or legacy software including:

  • Argo CD
  • Istio
  • Apache Kafka
  • Postgres
  • HashiCorp Vault
  • Prometheus
  • Redis
  • Cert Manager

Finally, we will migrate our sample Spring Boot app. I will also show you how to build such an app on Kubernetes and OpenShift in the same way using the Shipwright tool. However, before we start, let’s discuss some differences between “vanilla” Kubernetes and OpenShift.

Introduction

What are the key differences between Kubernetes and OpenShift? That’s probably the first question you will ask yourself when considering migration from Kubernetes. Today, I will focus only on those aspects that impact running the apps from our list. First of all, OpenShift is built on top of Kubernetes and is fully compatible with Kubernetes APIs and resources. If you can do something on Kubernetes, you can do it on OpenShift in the same way unless it doesn’t compromise security policy. OpenShift comes with additional security policies out of the box. For example, by default, it won’t allow you to run containers with the root user.

Apart from security reasons, only the fact that you can do something doesn’t mean that you should do it in that way. So, you can run images from Docker Hub, but Red Hat provides many supported container images built from Red Hat Enterprise Linux. You can find a full list of supported images here. Although you can install popular software on OpenShift using Helm charts, Red Hat provides various supported Kubernetes operators for that. With those operators, you can be sure that the installation will go without any problems and the solution might be integrated with OpenShift better. We will analyze all those things based on the examples from the tools list.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. I will explain the structure of our sample in detail later. So after cloning the Git repository you should just follow my instructions.

Install Argo CD

Use Official Helm Chart

In the first step, we will install Argo CD on OpenShift. I’m assuming that on Kubernetes, you’re using the official Helm chart for that. In order to install that chart, we need to add the following Helm repository:

$ helm repo add argo https://argoproj.github.io/argo-helm
ShellSession

Then, we can install the Argo CD in the argocd namespace on OpenShift with the following command. The Argo CD Helm chart provides some parameters dedicated to OpenShift. We need to enable arbitrary uid for the repo server by setting the openshift.enabled property to true. If we want to access the Argo CD dashboard outside of the cluster we should expose it as the Route. In order to do that, we need to enable the server.route.enabled property and set the hostname using the server.route.hostname parameter (piomin.eastus.aroapp.io is my OpenShift domain).

$ helm install argocd argo/argo-cd -n argocd --create-namespace \
    --set openshift.enabled=true \
    --set server.route.enabled=true \
    --set server.route.hostname=argocd.apps.piomin.eastus.aroapp.io
ShellSession

After that, we can access the Argo CD dashboard using the Route address as shown below. The admin user password may be taken from the argocd-initial-admin-secret Secret generated by the Helm chart.

Use the OpenShift GitOps Operator (Recommended Way)

The solution presented in the previous section works fine. However, it is not the optimal approach for OpenShift. In that case, the better idea is to use OpenShift GitOps Operator. Firstly, we should find the “Red Hat GitOps Operator” inside the “Operator Hub” section in the OpenShift Console. Then, we have to install the operator.

During the installation, the operator automatically creates the Argo CD instance in the openshift-gitops namespace.

OpenShift GitOps operator automatically exposes the Argo CD dashboard through the Route. It is also integrated with OpenShift auth, so we can use cluster credentials to sign in there.

kubernetes-to-openshift-argocd

Install Redis, Postgres and Apache Kafka

OpenShift Support in Bitnami Helm Charts

Firstly, let’s assume that we use Bitnami Helm charts to install all three tools from the chapter title (Redis, Postgres, Kafka) on Kubernetes. Fortunately, the latest versions of Bitnami Helm charts provide out-of-the-box compatibility with the OpenShift platform. Let’s analyze what it means.

Beginning from the 4.11 version OpenShift introduces new Security Context Constraints (SCC) called restricted-v2. In OpenShift, security context constraints allow us to control permissions assigned to the pods. The restricted-v2 SCC includes a minimal set of privileges usually required for a generic workload to run. It is the most restrictive policy that matches the current pod security standards. As I mentioned before, the latest version of the most popular Bitnami Helm charts supports the restricted-v2 SCC. We can check which of the charts support that feature by checking if they provide the global.compatibility.openshift.adaptSecurityContext parameter. The default value of that parameter is auto. It means that it is applied only if the detected running cluster is Openshift.

So, in short, we don’t have to change anything in the Helm chart configuration used on Kubernetes to make it work also on OpenShift. However, it doesn’t mean that we won’t change that configuration. Let’s analyze it tool after tool.

Install Redis on OpenShift with Helm Chart

In the first step, let’s add the Bitnami Helm repository with the following command:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
ShellSession

Then, we can install and run a three-node Redis cluster with a single master node in the redis namespace using the following command:

$ helm install redis bitnami/redis -n redis --create-namespace
ShellSession

After installing the chart we can display a list of pods running the redis namespace:

$ oc get po
NAME               READY   STATUS    RESTARTS   AGE
redis-master-0     1/1     Running   0          5m31s
redis-replicas-0   1/1     Running   0          5m31s
redis-replicas-1   1/1     Running   0          4m44s
redis-replicas-2   1/1     Running   0          4m3s
ShellSession

Let’s take a look at the securityContext section inside one of the Redis cluster pods. It contains characteristic fields for the restricted-v2 SCC, which removes runAsUser, runAsGroup and fsGroup and let the platform use their allowed default IDs.

kubernetes-to-openshift-security-context

However, let’s stop for a moment to analyze the current situation. We installed Redis on OpenShift using the Bitnami Helm chart. By default, this chart is based on the Redis Debian image provided by Bitnami in the Docker Hub.

On the other hand, Red Hat provides its build of Redis image based on RHEL 9. Consequently, this image would be more suitable for running on OpenShift.

kubernetes-to-openshift-redis

In order to use a different Redis image with the Bitnami Helm chart, we need to override the registry, repository, and tag fields in the image section. The full address of the current latest Red Hat Redis image is registry.redhat.io/rhel9/redis-7:1-16. In order to make the Bitnami chart work with that image, we need to override the default data path to /var/lib/redis/data and disable the container’s Security Context read-only root filesystem for the slave pods. 

image:
  tag: 1-16
  registry: registry.redhat.io
  repository: rhel9/redis-7

master:
  persistence:
    path: /var/lib/redis/data

replica:
  persistence:
    path: /var/lib/redis/data
  containerSecurityContext:
    readOnlyRootFilesystem: false
YAML

Install Postgres on OpenShift with Helm Chart

With Postgres, we have every similar as before with Redis. The Bitnami Helm chart also supports OpenShift restricted-v2 SCC and Red Hat provide the Postgres image based on RHEL 9. Once again, we need to override some chart parameters to adapt to a different image than the default one provided by Bitnami.

image:
  tag: 1-54
  registry: registry.redhat.io
  repository: rhel9/postgresql-15

primary:
  containerSecurityContext:
    readOnlyRootFilesystem: false
  persistence:
    mountPath: /var/lib/pgsql
  extraEnvVars:
    - name: POSTGRESQL_ADMIN_PASSWORD
      value: postgresql123

postgresqlDataDir: /var/lib/pgsql/data
YAML

Of course, we can consider switching to one of the available Postgres operators. From the “Operator Hub” section we can install e.g. Postgres using Crunchy or EDB operators. However, these are not operators provided by Red Hat. Of course, you can use them on “vanilla” Kubernetes as well. In that case, the migration to OpenShift also won’t be complicated.

Install Kafka on OpenShift with the Strimzi Operator

The situation is slightly different in the case of Apache Kafka. Of course, we can use the Kafka Helm chart provided by Bitnami. However, Red Hat provides a supported version of Kafka through the Strimzi operator. This operator is a part of the Red Hat product ecosystem and is available commercially as the AMQ Streams. In order to install Kafka with AMQ Streams on OpenShift, we need to install the operator first.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: amq-streams
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: amq-streams
  source: redhat-operators
  sourceNamespace: openshift-marketplace
YAML

Once we install the operator with the Strimzi CRDs we can provision the Kafka instance on OpenShift. In order to do that, we need to define the Kafka object. The name of the cluster is my-cluster. We should install it after a successful installation of the operator CRD, so we set the higher value of the Argo CD sync-wave parameter than for the amq-streams Subscription object. Argo CD should also ignore missing CRDs installed by the operator during sync thanks to the SkipDryRunOnMissingResource option.

apiVersion: kafka.strimzi.io/v1beta2
kind: Kafka
metadata:
  name: my-cluster
  namespace: kafka
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  kafka:
    config:
      offsets.topic.replication.factor: 3
      transaction.state.log.replication.factor: 3
      transaction.state.log.min.isr: 2
      default.replication.factor: 3
      min.insync.replicas: 2
      inter.broker.protocol.version: '3.6'
    storage:
      type: persistent-claim
      size: 5Gi
      deleteClaim: true
    listeners:
      - name: plain
        port: 9092
        type: internal
        tls: false
      - name: tls
        port: 9093
        type: internal
        tls: true
    version: 3.6.0
    replicas: 3
  entityOperator:
    topicOperator: {}
    userOperator: {}
  zookeeper:
    storage:
      type: persistent-claim
      deleteClaim: true
      size: 2Gi
    replicas: 3
YAML

GitOps Strategy for Kubernetes and OpenShift

In this section, we will focus on comparing differences in the GitOps manifest between Kubernetes and Openshift. We will use Kustomize to configure two overlays: openshift and kubernetes. Here’s the structure of our configuration repository:

.
├── base
│   ├── kustomization.yaml
│   └── namespaces.yaml
└── overlays
    ├── kubernetes
    │   ├── kustomization.yaml
    │   ├── namespaces.yaml
    │   ├── values-cert-manager.yaml
    │   └── values-vault.yaml
    └── openshift
        ├── cert-manager-operator.yaml
        ├── kafka-operator.yaml
        ├── kustomization.yaml
        ├── service-mesh-operator.yaml
        ├── values-postgres.yaml
        ├── values-redis.yaml
        └── values-vault.yaml
ShellSession

Configuration for Kubernetes

In addition to the previously discussed tools, we will also install “cert-manager”, Prometheus, and Vault using Helm charts. Kustomize allows us to define a list of managed charts using the helmCharts section. Here’s the kustomization.yaml file containing a full set of installed charts:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - namespaces.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
  - name: kafka
    repo: https://charts.bitnami.com/bitnami
    releaseName: kafka
    namespace: kafka
  - name: cert-manager
    repo: https://charts.jetstack.io
    releaseName: cert-manager
    namespace: cert-manager
    valuesFile: values-cert-manager.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml
  - name: prometheus
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: prometheus
    namespace: prometheus
  - name: istio
    repo: https://prometheus-community.github.io/helm-charts
    releaseName: istio
    namespace: istio-system
overlays/kubernetes/kustomization.yaml

For some of them, we need to override default Helm parameters. Here’s the values-vault.yaml file with the parameters for Vault. We enable development mode and UI dashboard:

server:
  dev:
    enabled: true
ui:
  enabled: true
overlays/kubernetes/values-vault.yaml

Let’s also customize the default behavior of the “cert-manager” chart with the following values:

installCRDs: true
startupapicheck:
  enabled: false
overlays/kubernetes/values-cert-manager.yaml

Configuration for OpenShift

Then, we can switch to the configuration for Openshift. Vault has to be installed with the Helm chart, but for “cert-manager” we can use the operator provided by Red Hat. Since Openshift comes with built-in Prometheus, we don’t need to install it. We will also replace the Helm chart with Istio with the Red Hat-supported OpenShift Service Mesh operator. Here’s the kustomization.yaml for OpenShift:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../base
  - kafka-operator.yaml
  - cert-manager-operator.yaml
  - service-mesh-operator.yaml

helmCharts:
  - name: redis
    repo: https://charts.bitnami.com/bitnami
    releaseName: redis
    namespace: redis
    valuesFile: values-redis.yaml
  - name: postgresql
    repo: https://charts.bitnami.com/bitnami
    releaseName: postgresql
    namespace: postgresql
    valuesFile: values-postgres.yaml
  - name: vault
    repo: https://helm.releases.hashicorp.com
    releaseName: vault
    namespace: vault
    valuesFile: values-vault.yaml
overlays/openshift/kustomization.yaml

For Vault we should enable integration with Openshift and support for the Route object. Red Hat provides a Vault image based on UBI in the registry.connect.redhat.com/hashicorp/vault registry. Here’s the values-vault.yaml file for OpenShift:

server:
  dev:
    enabled: true
  route:
    enabled: true
    host: ""
    tls: null
  image:
    repository: "registry.connect.redhat.com/hashicorp/vault"
    tag: "1.16.1-ubi"
global:
  openshift: true
injector:
  enabled: false
overlays/openshift/values-vault.yaml

In order to install operators we need to define at least the Subscription object. Here’s the subscription for the OpenShift Service Mesh. After installing the operator we can create a control plane in the istio-system namespace using the ServiceMeshControlPlane CRD object. In order to apply the CRD after installing the operator, we need to use the Argo CD sync waves and define the SkipDryRunOnMissingResource parameter:

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: servicemeshoperator
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable
  installPlanApproval: Automatic
  name: servicemeshoperator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: maistra.io/v2
kind: ServiceMeshControlPlane
metadata:
  name: basic
  namespace: istio-system
  annotations:
    argocd.argoproj.io/sync-wave: "3"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  tracing:
    type: None
    sampling: 10000
  policy:
    type: Istiod
  addons:
    grafana:
      enabled: false
    jaeger:
      install:
        storage:
          type: Memory
    kiali:
      enabled: false
    prometheus:
      enabled: false
  telemetry:
    type: Istiod
  version: v2.5
overlays/openshift/service-mesh-operator.yaml

Since the “cert-manager” operator is installed in a different namespace than openshift-operators, we also need to define the OperatorGroup object.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: openshift-cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: stable-v1
  installPlanApproval: Automatic
  name: openshift-cert-manager-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
apiVersion: operators.coreos.com/v1alpha2
kind: OperatorGroup
metadata:
  name: cert-manager-operator
  namespace: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  targetNamespaces:
    - cert-manager
overlays/openshift/cert-manager-operator.yaml

Finally, OpenShift comes with built-in Prometheus monitoring, so we don’t need to install it.

Apply the Configuration with Argo CD

Here’s the Argo CD Application responsible for installing our sample configuration on OpenShift. We should create it in the openshift-gitops namespace.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: install
  namespace: openshift-gitops
spec:
  destination:
    server: 'https://kubernetes.default.svc'
  project: default
  source:
    path: overlays/openshift
    repoURL: 'https://github.com/piomin/kubernetes-to-openshift-argocd.git'
    targetRevision: HEAD
YAML

Before that, we need to enable the use of the Helm chart inflator generator with Kustomize in Argo CD. In order to do that, we can add the kustomizeBuildOptions parameter in the openshift-gitops ArgoCD object as shown below.

apiVersion: argoproj.io/v1beta1
kind: ArgoCD
metadata:
  name: openshift-gitops
  namespace: openshift-gitops
spec:
  # ...
  kustomizeBuildOptions: '--enable-helm'
YAML

After creating the Argo CD Application and triggering the sync process, the installation starts on OpenShift.

kubernetes-to-openshift-gitops

Build App Images

We installed several software solutions including the most popular databases, message brokers, and security tools. However, now we want to build and run our own apps. How to migrate them from Kubernetes to OpenShift? Of course, we can run the app images exactly in the same way as in Kubernetes. On the other hand, we can build them on OpenShift using the Shipwright project. We can install it on OpenShift using the “Builds for Red Hat OpenShift Operator”.

kubernetes-to-openshift-shipwright

After that, we need to create the ShiwrightBuild object. It needs to contain the name of the target namespace for running Shipwright in the targetNamespace field. In my case, the target namespace is builds-demo. For a detailed description of the Shipwright build, you can refer to that article on my blog.

apiVersion: operator.shipwright.io/v1alpha1
kind: ShipwrightBuild
metadata:
  name: openshift-builds
spec:
  targetNamespace: builds-demo
YAML

With Shipwright we can easily switch between multiple build strategies on Kubernetes, and on OpenShift as well. For example, on OpenShift we can use a built-in source-to-image (S2I) strategy, while on Kubernetes e.g. Kaniko or Cloud Native Buildpacks.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: sample-spring-kotlin-build
  namespace: builds-demo
spec:
  output:
    image: quay.io/pminkows/sample-kotlin-spring:1.0-shipwright
    pushSecret: pminkows-piomin-pull-secret
  source:
    git:
      url: https://github.com/piomin/sample-spring-kotlin-microservice.git
  strategy:
    name: source-to-image
    kind: ClusterBuildStrategy
YAML

Final Thoughts

Migration from Kubernetes to Openshift is not a painful process. Many popular Helm charts support OpenShift restricted-v2 SCC. Thanks to that, in some cases, you don’t need to change anything. However, sometimes it’s worth switching to the version of the particular tool supported by Red Hat.

The post Migrate from Kubernetes to OpenShift in the GitOps Way appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/04/15/migrate-from-kubernetes-to-openshift-in-the-gitops-way/feed/ 2 15190 Java Development with Odo on Podman, Kubernetes and OpenShift https://piotrminkowski.com/2024/03/15/java-development-with-odo-on-podman-kubernetes-and-openshift/ https://piotrminkowski.com/2024/03/15/java-development-with-odo-on-podman-kubernetes-and-openshift/#respond Fri, 15 Mar 2024 10:11:57 +0000 https://piotrminkowski.com/?p=15091 In this article, you will learn how to develop and deploy Java apps on Podman, Kubernetes, and OpenShift with odo. Odo is a fast and iterative CLI tool for developers who want to write, build, and deploy applications on Kubernetes-native environments. Thanks to odo you can focus on the most important aspect of programming – […]

The post Java Development with Odo on Podman, Kubernetes and OpenShift appeared first on Piotr's TechBlog.

]]> In this article, you will learn how to develop and deploy Java apps on Podman, Kubernetes, and OpenShift with odo. Odo is a fast and iterative CLI tool for developers who want to write, build, and deploy applications on Kubernetes-native environments. Thanks to odo you can focus on the most important aspect of programming – code. I have already written an article about that tool in my blog some years ago. However, a lot has changed during that time.

Today, we will also focus more on Podman, and especially Podman Desktop, as an alternative to the Docker Desktop for local development. You will learn how to integrate the odo CLI with Podman. We will also use Podman for creating clusters and switching between several Kubernetes contexts. Our sample Java app is written in Spring Boot, exposes some REST endpoints over HTTP, and connects to the Postgres database. Let’s begin!

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. The sample Spring Boot app is located inside the micro-springboot/person-service directory. Once you clone the repo and go to that directory, you should just follow my further instructions.

Create Sample Spring Boot App

The app source code is not the most important thing in our exercise. However, let’s do a quick recap of its main parts. Here’s the Maven pom.xml with a list of dependencies. It includes standard Spring Boot starters for exposing REST endpoints and integrating with the Postgres database through JPA. It also uses additional libraries for generating OpenAPI docs (Springdoc) and creating entity views (Blaze Persistence).

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
  <groupId>org.postgresql</groupId>
  <artifactId>postgresql</artifactId>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
  <groupId>org.springdoc</groupId>
  <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
  <version>2.4.0</version>
</dependency>
<dependency>
  <groupId>com.blazebit</groupId>
  <artifactId>blaze-persistence-integration-spring-data-3.1</artifactId>
  <version>${blaze.version}</version>
</dependency>
<dependency>
  <groupId>com.blazebit</groupId>
  <artifactId>blaze-persistence-integration-hibernate-6.2</artifactId>
  <version>${blaze.version}</version>
  <scope>runtime</scope>
</dependency>
<dependency>
  <groupId>com.blazebit</groupId>
  <artifactId>blaze-persistence-entity-view-processor</artifactId>
  <version>${blaze.version}</version>
</dependency>

Here’s our @Entity model class:

@Entity
public class Person {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Integer id;
    private String firstName;
    private String lastName;
    private int age;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    private Integer externalId;

}

Here’s the entity view interface used for returning persons in the REST endpoint. We can leverage the Blaze Persistence library to map between the JPA entity and a DTO view object.

@EntityView(Person.class)
public interface PersonView {

   @IdMapping
   Integer getId();
   void setId(Integer id);

   @Mapping("CONCAT(firstName,' ',lastName)")
   String getName();
   void setName(String name);
}

There are two repository interfaces. The first one is used for the modifications. It extends the standard Spring Data JPA CrudRepository.

public interface PersonRepository extends CrudRepository<Person, Integer> {}

The second one is dedicated just for read operation. It extends the Blaze Persistence EntityViewRepository interface.

@Transactional(readOnly = true)
public interface PersonViewRepository extends EntityViewRepository<PersonView, Integer> {
   PersonView findByAgeGreaterThan(int age);
}

In the @RestController implementation, we use both repository beans. Depending on the operation type, the API method uses the Spring Data JPA PersonRepository or the Blaze Persistence PersonViewRepository. 

@RestController
@RequestMapping("/persons")
public class PersonController {

   private static final Logger LOG = LoggerFactory
      .getLogger(PersonController.class);
   private final PersonRepository repository;
   private final PersonViewRepository viewRepository;

   public PersonController(PersonRepository repository, 
                           PersonViewRepository viewRepository) {
      this.repository = repository;
      this.viewRepository = viewRepository;
   }

   @GetMapping
   public List<PersonView> getAll() {
       LOG.info("Get all persons");
       return (List<PersonView>) viewRepository.findAll();
   }

   @GetMapping("/{id}")
   public PersonView getById(@PathVariable("id") Integer id) {
      LOG.info("Get person by id={}", id);
      return viewRepository.findOne(id);
   }

   @GetMapping("/age/{age}")
   public PersonView getByAgeGreaterThan(@PathVariable("age") int age) {
      LOG.info("Get person by age={}", age);
      return viewRepository.findByAgeGreaterThan(age);
   }

   @DeleteMapping("/{id}")
   public void deleteById(@PathVariable("id") Integer id) {
      LOG.info("Delete person by id={}", id);
      repository.deleteById(id);
   }

   @PostMapping
   public Person addNew(@RequestBody Person person) {
      LOG.info("Add new person: {}", person);
      return repository.save(person);
   }

   @PutMapping
   public void update(@RequestBody Person person) {
      repository.save(person);
   }
}

Here are the app configuration settings in the Spring Boot application.yml file. The app creates the database schema on startup and uses environment variables to establish a connection with the target database.

spring:
  application:
    name: person-service
  datasource:
    url: jdbc:postgresql://${DATABASE_HOST}:5432/${DATABASE_NAME}
    username: ${DATABASE_USER}
    password: ${DATABASE_PASS}
  jpa:
    hibernate:
      ddl-auto: create
    properties:
      hibernate:
        show_sql: true
        format_sql: true

Now, our goal is to build the app container image and then run it on the target environment for further development. In our case, it is the Kubernetes or OpensShift cluster. We need a tool that doesn’t require a deep understanding of Kubernetes or even Podman. Here comes odo.

Create and Manage Devfiles

In the first step, we need to configure our project. Odo is based on the open standard for defining containerized environments called Devfile. Once we execute the odo init command inside the app root directory and choose the app type, it generates the devfile.yaml automatically. Of course, we can already keep the devfile.yaml in the Git repo. Thanks to that, we don’t need to initialize the odo configuration after cloning the repo. That’s what it is in my case. 

After generating the “devfile”, we need to change it a little bit. First of all, I switched to the Java 21 base image, instead of Java 17 generated by the odo init command. We will also add the environment variables used by the Spring Boot app to establish connection with the Postgres database. Here’s the udpated fragment of the devfile.yaml responsible for running app container.

components:
- container:
    command:
    - tail
    - -f
    - /dev/null
    endpoints:
    - name: http-springboot
      targetPort: 8080
    - exposure: none
      name: debug
      targetPort: 5858
    env:
    - name: DEBUG_PORT
      value: "5858"
    - name: DATABASE_HOST
      value: localhost
    - name: DATABASE_USER
      value: springboot
    - name: DATABASE_PASS
      value: springboot123
    - name: DATABASE_NAME
      value: sampledb
    image: registry.access.redhat.com/ubi9/openjdk-21:latest

I also included an additional container with the Postgres database. Thanks to that, odo will not only build and run the app container but also the container with a database required by that app. We use the registry.redhat.io/rhel9/postgresql-15 Postgres image from Red Hat official registry. We can set a default username, password and database using the POSTGRESQL_* envs supported by the Red Hat Postgres image.

- name: postgresql
  container:
    image: registry.redhat.io/rhel9/postgresql-15
    env:
      - name: POSTGRESQL_USER
        value: springboot
      - name: POSTGRESQL_PASSWORD
        value: springboot123
      - name: POSTGRESQL_DATABASE
        value: sampledb
    endpoints:
      - name: postgresql
        exposure: internal
        targetPort: 5432
        attributes:
          discoverable: 'true'
    memoryLimit: 512Mi
    mountSources: true
    volumeMounts:
      - name: postgresql-storage
        path: /var/lib/postgresql/data
- name: postgresql-storage
  volume:
    size: 256Mi

Here’s the whole devfile.yaml after our customizations. Of course, you can find it inside the GitHub repository.

commands:
- exec:
    commandLine: mvn clean -Dmaven.repo.local=/home/user/.m2/repository package -Dmaven.test.skip=true
    component: tools
    group:
      isDefault: true
      kind: build
    workingDir: ${PROJECT_SOURCE}
  id: build
- exec:
    commandLine: mvn -Dmaven.repo.local=/home/user/.m2/repository spring-boot:run
    component: tools
    group:
      isDefault: true
      kind: run
    workingDir: ${PROJECT_SOURCE}
  id: run
- exec:
    commandLine: java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=${DEBUG_PORT},suspend=n -jar target/*.jar
    component: tools
    group:
      isDefault: true
      kind: debug
    workingDir: ${PROJECT_SOURCE}
  id: debug
components:
- container:
    command:
    - tail
    - -f
    - /dev/null
    endpoints:
    - name: http-springboot
      targetPort: 8080
    - exposure: none
      name: debug
      targetPort: 5858
    env:
    - name: DEBUG_PORT
      value: "5858"
    - name: DATABASE_HOST
      value: localhost
    - name: DATABASE_USER
      value: springboot
    - name: DATABASE_PASS
      value: springboot123
    - name: DATABASE_NAME
      value: sampledb
    image: registry.access.redhat.com/ubi9/openjdk-21:latest
    memoryLimit: 768Mi
    mountSources: true
    volumeMounts:
    - name: m2
      path: /home/user/.m2
  name: tools
- name: postgresql
  container:
# uncomment for Kubernetes
#    image: postgres:15
#    env:
#      - name: POSTGRES_USER
#        value: springboot
#      - name: POSTGRES_PASSWORD
#        value: springboot123
#      - name: POSTGRES_DB
#        value: sampledb
    image: registry.redhat.io/rhel9/postgresql-15
    env:
      - name: POSTGRESQL_USER
        value: springboot
      - name: POSTGRESQL_PASSWORD
        value: springboot123
      - name: POSTGRESQL_DATABASE
        value: sampledb
    endpoints:
      - name: postgresql
        exposure: internal
        targetPort: 5432
        attributes:
          discoverable: 'true'
    memoryLimit: 512Mi
    mountSources: true
    volumeMounts:
      - name: postgresql-storage
        path: /var/lib/postgresql/data
- name: postgresql-storage
  volume:
    size: 256Mi
- name: m2
  volume:
    size: 3Gi
metadata:
  description: Java application using Spring Boot® and OpenJDK 21
  displayName: Spring Boot®
  globalMemoryLimit: 2674Mi
  icon: https://raw.githubusercontent.com/devfile-samples/devfile-stack-icons/main/spring.svg
  language: Java
  name: person-service
  projectType: springboot
  tags:
  - Java
  - Spring
  version: 1.3.0
schemaVersion: 2.1.0

Prepare Development Environment with Podman

We have everything ready on the application side. Now, it is time to prepare a dev environment. Let’s run Podman Desktop. Podman Desktop comes with several useful features that simplify interaction with Kubernetes or OpenShift clusters for developers. It provides plugins that allow us to install Openshift Local, Minikube, or Kind on the laptop. We can also leverage the remote instance of OpenShift on the Sandbox Developer portal. It is active for 30 days and may be renewed.

odo-podman-kubernetes-desktop


Podman has one important advantage over Docker. It supports the Pod concept. So, for example, suppose we have a container that requires the Postgres container, with Podman we do not need to bind that database to a routable network.  With a pod, we are just binding to the localhost address and all containers in that pod can connect to it because of the shared network. Thanks to that, switching between Podman and Kubernetes is very simple with odo. In fact, we don’t need to change anything in the configuration.

For running the Kubernetes cluster on the local machine, we will Minikube. You can install and run Minikube by yourself or create it using the Podman Desktop. With Podman Desktop, we need to go to the Settings -> Resources section. Then find the Minikube tile as shown below and click the “Create new …” button.

odo-podman-kubernetes-minikube

Podman Desktop redirects us to the window with the creation form. Then, choose your preferred settings and click Create.

In order to create an OpenShift cluster we use a managed service available on the Red Hat Developer Sandbox website. Then, let’s choose “Start your Sandbox for free”.

You need to create an account on Red Hat Developer or sign in if you already did that. After that, you will be redirected to the Red Hat Hybrid Cloud Console, where you should choose Launch option on the “Red Hat OpenShift” tile as shown below.

In the OpenShift Console click on your username in the top-right corner and choose “Copy login command”.

odo-podman-kubernetes-ocp

Then, we can back to the Podman Desktop and paste the copied command in the Settings -> Resources -> Developer Sandbox section.

Deploy App with Database on Podman

Finally, let’s move from theory to practice. Assuming we have Podman running on our laptop, we can deploy our app with odo there by executing the following command:

$ odo dev --platform podman

The odo dev command deploys the app on the target environment and waits for any changes in the source files. Once it occurs, it redeploys the app.

The odo command exposes both the app HTTP port and the Postgres port outside Podman with port forwarding. For example, our Spring Boot app is available under the 20002 port. You can run the Swagger UI and call some endpoints to test the functionality.

odo-podman-kubernetes-swagger

Our app container can connect to the database over localhost, because both containers are running inside a single pod.

odo-podman-kubernetes-pods

Deploy App with a Database on Kubernetes

Then, we can switch to the Kubernetes cluster with our app. In this exercise, we will use Minikube. We can easily create and run the Minikube instance using the Podman Desktop plugin.

Once we create such an instance, the Kubernetes content is switched automatically. We can check it out with Podman Desktop as shown below.

Let’s deploy our app on Minikube. If we don’t activate the platform option, odo deploys on the default cluster context.

$ odo create namespace springboot
$ ode dev

Here’s the command result for my Minikube instance. Thanks to automatic port-forwarding we can access the app exactly in the same as before with Podman.

We can display a list of running with the following command:

kubectl get po
NAME                                 READY   STATUS    RESTARTS   AGE
person-service-app-b69bf8f7d-hk555   2/2     Running   0          2m50s

The full required YAML is generated automatically based on the devfile (e.g. environment variables). The only thing I changed for Kubernetes is the Postgres-based image used by odo. Instead of the image from the Red Hat registry, we are just using the official Postgres image from Docker Hub.

- name: postgresql
  container:
    image: postgres:15
    env:
      - name: POSTGRES_USER
        value: springboot
      - name: POSTGRES_PASSWORD
        value: springboot123
      - name: POSTGRES_DB
        value: sampledb

Deploy App with a Database on OpenShift

We can deploy the sample Spring Boot app on OpenShift exactly in the same way as in OpenShift. The only thing that can be changed is the Postgres image. This time we will back to the configuration used when deploying to podman. Instead of the image from the Docker Hub, we will use the registry.redhat.io/rhel9/postgresql-15 image.

As I mentioned before, we will use the remote OpenShift cluster on the Developer Sandbox. Podman Desktop provides the plugin for Developer Sandbox. With that plugin, we can map the OpenShift context to a specific name like dev-sandbox-context.

odo-podman-kubernetes-developer-sandbox

Then we can switch to the Kubernetes context related to Developer Sandbox using the Podman Desktop.

Finally, let’s run the app on the cluster with the following command:

$ odo dev

Here’s the output after running the odo dev command:

odo-podman-kubernetes-dev

We can verify that a pod is running on the OpenShift cluster. Just go to the Workloads -> Pods section in the OpenShift Console.

odo-podman-kubernetes-openshift-console

Thanks to automatic port-forwarding we can access the app on the local port. However, we can expose the service outside OpenShift with the Route object. Firstly, let’s display a list of Kubernetes services using Podman Desktop.

Then, we need to create the Route object with the following command:

$ oc expose svc/person-service-app

Here’s our Route visible in the OpenShift Console. To access it we need to open the following address in the web browser:

odo-podman-kubernetes-openshift-route

Finally, let’s access the app Swagger UI using the exposed URL address:

Final Thoughts

With Podman and the odo CLI, we can configure our development space and easily run apps across different containerized and Kubernetes-native environments. Odo with Devfile standard can similarly run the app on Podman, Kubernetes, and OpenShift. You can control the whole process using Podman Desktop.

The post Java Development with Odo on Podman, Kubernetes and OpenShift appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/03/15/java-development-with-odo-on-podman-kubernetes-and-openshift/feed/ 0 15091 Rotate SSL Certificates with OpenShift and Spring Boot https://piotrminkowski.com/2024/03/04/rotate-ssl-certificates-with-openshift-and-spring-boot/ https://piotrminkowski.com/2024/03/04/rotate-ssl-certificates-with-openshift-and-spring-boot/#respond Mon, 04 Mar 2024 11:41:08 +0000 https://piotrminkowski.com/?p=15047 This article will teach you how to dynamically create and rotate SSL certificates used in service-to-service communication with OpenShift and Spring Boot. We will achieve it with a single annotation on a Kubernetes service and the “SSL Bundles” mechanism introduced in Spring Boot 3.1. For generating the SSL on OpenShift, we will use the mechanism […]

The post Rotate SSL Certificates with OpenShift and Spring Boot appeared first on Piotr's TechBlog.

]]> This article will teach you how to dynamically create and rotate SSL certificates used in service-to-service communication with OpenShift and Spring Boot. We will achieve it with a single annotation on a Kubernetes service and the “SSL Bundles” mechanism introduced in Spring Boot 3.1. For generating the SSL on OpenShift, we will use the mechanism called “Service Serving Certificates”. It generates a certificate and key in PEM format. With Spring Boot 3.1 SslBundles, it won’t be a problem for our Java app, since it supports PEM-encoded certificates. The “Service Serving Certificates” mechanism automatically rotates CA certificates every 13 months.

There is a similar article on my blog about “cert-manager” and Spring Boot SSL hot reload more focused on Kubernetes. You can also read the post on how to integrate Spring Boot with Vault PKI to generate certs automatically.

Source Code

If you would like to try this exercise yourself, you may always take a look at my source code. In order to do that, you need to clone my GitHub repository. Let’s check out the openshift branch. Then, we need to go to the ssl directory. You will find two Spring Boot apps: secure-callme-bundle and secure-caller-bundle. After that, you should just follow my instructions. Let’s begin.

$ git clone https://github.com/piomin/sample-spring-security-microservices.git
$ git checkout openshift
$ cd ssl

How It Works

OpenShift brings a lot of interesting features to the Kubernetes platform. The “Service Serving Certificates” is one of the examples. Thanks to that mechanism, we don’t need to install any external tool for generating and rotating certificates dynamically like “cert-manager”. The OpenShift CA service automatically rotates certificates 13 months before expiration and issues new certificates for 26 months. After a rotation, the previous CA is trusted until it expires. To achieve it, OpenShift implements the root CA key rollover process described in  https://tools.ietf.org/html/rfc4210#section-4.4. This allows a grace period for all affected services to refresh their key material before the expiration. Sounds interesting? Let’s proceed to the implementation.

In order to test a mechanism, we will run two simple Spring Boot apps on OpenShift that communicate using SSL certificates. OpenShift takes care of issuing/rotating certificates and exposing the CA bundle as a ConfigMap. Access to this CA certificate allows TLS clients to verify connections to services using Service Serving Certificates”. Here’s the illustration of our apps’ architecture on OpenShift.

ssl-openshift-spring-boot-arch

Create a Server Side App

Our first app secure-callme-bundle exposes a single endpoint GET /callme over HTTP. That endpoint will be called by the secure-caller-bundle app. Here’s the @RestController implementation:

@RestController
public class SecureCallmeController {

    @GetMapping("/callme")
    public String call() {
        return "I'm `secure-callme`!";
    }

}

Now, our main goal is to enable HTTPS for that app and make it work properly with the OpenShift “Service Serving Certificates”. Firstly, we should change the default server port for the Spring Boot app to 8443 (1). Starting from Spring Boot 3.1 we can use the spring.ssl.bundle.* properties instead of the server.ssl.* properties to configure keystores for the web server (3). We can configure SSL bundles using PEM-encoded text files directly in the app with the spring.ssl.bundle.pem properties group. The certificate (tls.crt) and private key (tls.key) will be mounted as a volume to the pod. Finally, the name of the bundle needs to be set for the web server with the server.ssl.bundle property (2). Here’s the full configuration of our Spring Boot app inside the application.yml file.

# (1)
server.port: 8443

# (2)
server.ssl.bundle: server

# (3)
---
spring.config.activate.on-profile: prod
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key

The only thing we need to do to generate SSL certificates for our app is to annotate the Kubernetes Service with service.beta.openshift.io/serving-cert-secret-name. It should contain the name of the target Kubernetes Secret as a value.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: secure-callme-bundle
  annotations:
    service.beta.openshift.io/serving-cert-secret-name: secure-callme-bundle
  name: secure-callme-bundle
spec:
  ports:
    - name: https
      port: 8443
      targetPort: 8443
  selector:
    app.kubernetes.io/name: secure-callme-bundle
  type: ClusterIP

Once we apply such a Service to the cluster, OpenShift generates the secure-callme-bundle Secret that contains the certificate and private key.

ssl-openshift-spring-boot-secret

After that, we just need to deploy the app and mount the generated Secret as a volume. The volume path is passed to the app as the CERT_PATH environment variable.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-callme-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-callme-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-callme-bundle
    spec:
      containers:
        - image: piomin/secure-callme-bundle
          name: secure-callme-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            - mountPath: /opt/secret
              name: cert
      volumes:
        - name: cert
          secret:
            secretName: secure-callme-bundle

Finally, we need to expose the HTTP endpoint outside of the cluster. We can easily do it with the OpenShift Route. Don’t forget the set the TLS termination as passthrough. Thanks to that OpenShift router sends traffic directly to the destination without TLS termination.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-callme-bundle
  labels:
    app: secure-callme-bundle
    app.kubernetes.io/component: secure-callme-bundle
    app.kubernetes.io/instance: secure-callme-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-callme-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Create a Client Side App

Let’s switch to the secure-caller-bundle app. This app also exposes a single HTTP endpoint. Inside this endpoint implementation method, we call the GET /callme endpoint exposed by the secure-callme-bundle app. We use the RestTemplate bean for that.

@RestController
public class SecureCallerBundleController {

    RestTemplate restTemplate;

    @Value("${client.url}")
    String clientUrl;

    public SecureCallerBundleController(RestTemplate restTemplate) {
        this.restTemplate = restTemplate;
    }

    @GetMapping("/caller")
    public String call() {
        return "I'm `secure-caller`! calling... " +
                restTemplate.getForObject(clientUrl, String.class);
    }
}

This time we need to define two SSL bundles in the application settings. The server bundle is for the web server, which is pretty similar to the bundle defined in the previous app sample. The client bundle is dedicated to the RestTemplate bean. It uses the truststore taken from ConfigMap with CA certificates. With the certs inside the service-ca.crt file, the RestTemplate bean can authenticate against the secure-callme-bundle app. 

server.port: 8443
server.ssl.bundle: server

---
spring.config.activate.on-profile: prod
client.url: https://${HOST}:8443/callme
spring.ssl.bundle.pem:
  server:
    keystore:
      certificate: ${CERT_PATH}/tls.crt
      private-key: ${CERT_PATH}/tls.key
  client:
    truststore:
      certificate: ${CLIENT_CERT_PATH}/service-ca.crt

Don’t forget about creating the RestTemplate bean. It needs to the SslBundles bean and set it on the RestTemplateBuilder.

@Bean
RestTemplate builder(RestTemplateBuilder builder, SslBundles sslBundles) {
   return builder
      .setSslBundle(sslBundles.getBundle("client")).build();
}

We need to create the ConfigMap annotated with service.beta.openshift.io/inject-cabundle=true. After that, the OpenShift cluster automatically injects the service CA certificate into the service-ca.crt key on the config map.

apiVersion: v1
kind: ConfigMap
metadata:
  name: secure-caller-bundle
  namespace: demo-secure
  annotations:
    service.beta.openshift.io/inject-cabundle: "true"
data: {}

Let’s display the details of the secure-caller-bundle ConfigMap. As you see, OpenShift injects the PEM-encoded CA certificates under the service-ca.crt key to that config map.

ssl-openshift-spring-boot-ca-bundle

Finally, we can deploy our client-side app. Here’s the YAML manifest with the Deployment object. Since the secure-caller-bundle app acts as a web server and also uses a REST client to invoke the secure-callme-bundle we need to mount two volumes. The first one (/opt/secret) contains a certificate and private key for the web server (1). Consequently, the second one (/opt/client-cert) holds the CA bundle used by the RestTemplate client (2). This client communicates with secure-callme-bundle internally through the Kubernetes Service (3).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-caller-bundle
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: secure-caller-bundle
  template:
    metadata:
      labels:
        app.kubernetes.io/name: secure-caller-bundle
    spec:
      containers:
        - image: piomin/secure-caller-bundle
          name: secure-caller-bundle
          ports:
            - containerPort: 8443
              name: https
          env:
            - name: CERT_PATH
              value: /opt/secret
            - name: CLIENT_CERT_PATH
              value: /opt/client-cert
            # (3)
            - name: HOST
              value: secure-callme-bundle.demo-secure.svc
            - name: SPRING_PROFILES_ACTIVE
              value: prod
          volumeMounts:
            # (1)
            - mountPath: /opt/secret
              name: cert
            # (2)
            - mountPath: /opt/client-cert
              name: client-cert
      volumes:
        - name: cert
          secret:
            secretName: secure-caller-bundle
        - name: client-cert
          configMap:
            name: secure-caller-bundle

Of course, we should also create a passthrough Route to expose our app the outside of the cluster:

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: secure-caller-bundle
  labels:
    app: secure-caller-bundle
    app.kubernetes.io/component: secure-caller-bundle
    app.kubernetes.io/instance: secure-caller-bundle
  annotations:
    openshift.io/host.generated: 'true'
spec:
  tls:
    termination: passthrough
  to:
    kind: Service
    name: secure-caller-bundle
    weight: 100
  port:
    targetPort: https
  wildcardPolicy: None

Testing OpenShift Certs SSL Rotation with Spring Boot

The main problem with testing the certs rotation scenario is in the 13-month interval. We can’t change that period in the configuration. However, there is a way how we can force OpenShift to rotate the CA certs on demand. In order to do that, we need to set the spec.unsupportedConfigOverrides.forceRotation.reason property on the cluster-wide ServiceCA object with any value. Of course, you shouldn’t do that at production, since it is not supported by OpenShift. It is just designed for testing or development purposes.

apiVersion: operator.openshift.io/v1
kind: ServiceCA
metadata:
spec:
  logLevel: Normal
  managementState: Managed
  operatorLogLevel: Normal
  unsupportedConfigOverrides:
    forceRotation:
      reason: testing

We will call our apps through the OpenShift Route. So, let’s display a current list of routes related to our sample apps:

$ oc get route
NAME                   HOST/PORT                                                     PATH   SERVICES               PORT    TERMINATION   WILDCARD
secure-caller-bundle   secure-caller-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-caller-bundle   https   passthrough   None
secure-callme-bundle   secure-callme-bundle-demo-secure.apps.ocp1.eastus.aroapp.io          secure-callme-bundle   https   passthrough   None

Once we set that property, OpenShift immediately rotates that CA certificate. It will also take care of synchronizing the latest certificates with all the secrets and config maps containing annotations required by the “Service Serving Certificates” mechanism. As you see, now, our secure-caller-bundle ConfigMap contains two CAs: an old one and a new one.

$ oc describe cm secure-caller-bundle
Name:         secure-caller-bundle
Namespace:    demo-secure
Labels:       <none>
Annotations:  service.beta.openshift.io/inject-cabundle: true

Data
====
service-ca.crt:
----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIITnr+EqYsYtEwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAzMDExMDM5MTVaFw0yNjA0MzAxMDM5MTZaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDN0No0DIZ+BCdPvQ9TIvEjwGUSoUXM+DYI
Dxlk2Ef6goFv3YsAgNJTxznntRc69dJTrLtRrmFN0d31JPM/gJOMkKMvTWnz8Qzx
athuvvwbivJNKh/qn+Dhewbjrx5LGpq3v9VQ7t/5Rf5F9VMpyp738EwfAy2cfxTA
sxgYpvU/foivM9U0uMaZyXA5vLsGN6cKpEREGMqKbIKSXNYXD9/lbHv2eyr+5+s9
keYEDkGTWMceISihm5mGNwOZpLZhNwFgSZR9O63TIgGc/TEZ3EYwcIjmlVOSwQ/u
sM43HL1jtYKw3AovQOc5qc2kT5eSPV7jVCCfNYnj3BLvt0AYUXE/AgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBT3iI1f
71ZVt6OVLC/+0oxNL7BqhzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEATcT/LK5pAaOGz0j3MR4NhsfmAjo4K8SoGmFE
F4lDzDWZTaJuJHCHqpblgGKLfokU9BQC3L4NZLIisOj3CWF3zIpC6MOsU7x9qDm0
aZpI1gqhRXQZOqVpE9XJ0XBEOLP9hlzxa66SQlALlU5CWbDNEMtXsMMQf1lSgIRz
arWK8QwmPsmg6duoVtFcCMY2klkVacZctiMgM4wvf1CCJt3TkiggwO3t4IdiV8Xa
Z9d/LlV/bFzboxGxOb2IMHj5VXKPM6HmC4slBSqKeyr7PYyM7rjhorIOOner9JrX
RhbC8/uARciLbcFKyOOCSLoe5m1i8QPrRmCm/GhVS9M4x86TXw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIIFlMUWK4HjW4wDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UE
Awwrb3BlbnNoaWZ0LXNlcnZpY2Utc2VydmluZy1zaWduZXJAMTcwODY0MDc4OTAe
Fw0yNDAyMjgxMTUyMjNaFw0yNjA0MjgxMTUyMjRaMDYxNDAyBgNVBAMMK29wZW5z
aGlmdC1zZXJ2aWNlLXNlcnZpbmctc2lnbmVyQDE3MDg2NDA3ODkwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDY6AZbLN0kT2BPXbDn7iZmbyYZgyI8Y3jh
hRq9JrX0ZIIs5BvFcRRkm2IcJ8gsMF4VztGCkNoDQ07ojAIHI/FxssAU8wwoz2QP
eH2qzBpLA1lBAYvtjki//55STJvJF+7Z6qbcDwUVX6/r+hrpy5MIjaQmRV4DRLpp
ZD3HrqYozJEvCKaA2pinC2VyW8IBDB0FDvPdBxvzjrCDjssr/v0jqFgUGFJJmKEj
EaFQACeMlOS4Q5avEglfwoLS+RGgPjE1s1gpxir2dZCdnM8b9B3CFy2sS+SSHN4y
+FJRJBv4CbMAMP+DLIsHY872c2XmOlleJNFpiFWi9r33LCc7MnbjAgMBAAGjYzBh
MA4GA1UdDwEB/wQEAwICpDAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBRey/oP
FWW5kG8ifSXqIXBh2L+FdzAfBgNVHSMEGDAWgBT3iI1f71ZVt6OVLC/+0oxNL7Bq
hzANBgkqhkiG9w0BAQsFAAOCAQEAhbimn2Tv5w9Cy7i6dNdYYdVhA52cCnWWDDhf
s/dZ+Zl1aP3f179XO0xz3ovLuE227Z9AYPGtPW6c0FQgxgDgl2pYUX4J88caTUG/
eR9EG9LEfrL6OlKMjMvAFWrL64bcAAiuoz/gWB0a29xMnUsA52+7OpnzmHMduaaz
5JfRaTwkjGGe8ay2XsTUNrNGqZJGKHhE+hrUVlFsS7GpaSEe9GllkS9lGUZp2Hei
lTxwnBr9j+aP/QWXwuAVxT1DUAByJYoTPrpMhk+hKQUj/8TvxmMmsksZ57steqRI
/gWYd4j2SohN3jd1mv6rQ+jsGGAfCNmYVZL1oevNaqfnENt0IA==
-----END CERTIFICATE-----

However, if we don’t restart the pods, our apps still use the old keystore and truststore (unless they have the reload-on-update option enabled).

Here’s a list of running pods.

oc get po
NAME                                    READY   STATUS    RESTARTS   AGE
secure-caller-bundle-57c7658d85-bj4q5   1/1     Running   0          2d23h
secure-callme-bundle-cbf59569f-n6gg5    1/1     Running   0          2d23h

Let’s just restart the secure-callme-bundle app by deleting its pod. Do not restart the secure-callme-bundle pod, because we want to test the scenario where it uses the old truststore.

$ oc delete po secure-callme-bundle-cbf59569f-n6gg5
pod "secure-callme-bundle-cbf59569f-n6gg5" deleted

Then, we can call the GET /callme endpoint exposed by the secure-callme-bundle app through the Route once again. As you see, it servers the latest, reloaded certificate.

Let’s perform a final test. Now, we call the GET /caller endpoint exposed by the secure-caller-bundle app. Under the hood, it uses RestTemplate to call the GET /callme endpoint exposed by the secure-callme-bundle app via secure-callme-bundle.demo-secure.svc address. As you see the communication still works fine although the secure-caller-bundle app still uses the keystore and truststore.

Final Thoughts

The OpenShift “Service Serving Certificates” mechanism is simple to configure and doesn’t require that you handle the process of reloading apps with the rotated trusted material. It also has some drawbacks. You cannot change the rotation period from 13 months into a different value, and there is a global CA bundle shared across all the services running on the cluster. If those things are problematic for you I advise you to use “cert-manager” and a tool that allows you to restart pods automatically on the Secret or ConfigMap change. It is also possible to leverage the Spring SslBundles to reload certs without restarting the pod. With Spring Boot apps you can also use OpenShift Service Mesh, which can handle SSL communication at the platform level.

The post Rotate SSL Certificates with OpenShift and Spring Boot appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/03/04/rotate-ssl-certificates-with-openshift-and-spring-boot/feed/ 0 15047 Java Flight Recorder on Kubernetes https://piotrminkowski.com/2024/02/13/java-flight-recorder-on-kubernetes/ https://piotrminkowski.com/2024/02/13/java-flight-recorder-on-kubernetes/#respond Tue, 13 Feb 2024 07:44:13 +0000 https://piotrminkowski.com/?p=14957 In this article, you will learn how to continuously monitor apps on Kubernetes with Java Flight Recorder and Cryostat. Java Flight Recorder (JFR) is a tool for collecting diagnostic and profiling data generated by the Java app. It is designed for use even in heavily loaded production environments since it causes almost no performance overhead. […]

The post Java Flight Recorder on Kubernetes appeared first on Piotr's TechBlog.

]]> In this article, you will learn how to continuously monitor apps on Kubernetes with Java Flight Recorder and Cryostat. Java Flight Recorder (JFR) is a tool for collecting diagnostic and profiling data generated by the Java app. It is designed for use even in heavily loaded production environments since it causes almost no performance overhead. We can say that Java Flight Recorder acts similarly to an airplane’s black box. Even if the JVM crashes, we can analyze the diagnostic data collected just before the failure. This fact makes JFR especially usable in an environment with many running apps – like Kubernetes.

Assuming that we are running many Java apps on Kubernetes, we should interested in the tool that helps to automatically gather data generated by Java Flight Recorder. Here comes Cryostat. It allows us to securely manage JFR recordings for the containerized Java workloads. With the built-in discovery mechanism, it can detect all the apps that expose JFR data. Depending on the use case, we can store and analyze recordings directly on the Kubernetes cluster Cryostat Dashboard or export recorded data to perform a more in-depth analysis.

If you are interested in more topics related to Java apps on Kubernetes, you can take a look at some other posts on my blog. The following article describes a list of best practices for running Java apps Kubernetes. You can also read e.g. on how to resize CPU limit to speed up Java startup on Kubernetes here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you need to go to the callme-service directory. After that, you should just follow my instructions. Let’s begin.

Install Cryostat on Kubernetes

In the first step, we install Cryostat on Kubernetes using its operator. In order to use and manage operators on Kubernetes, we should have the Operator Lifecycle Manager (OLM) installed on the cluster. The operator-sdk binary provides a command to easily install and uninstall OLM:

$ operator-sdk olm install

Alternatively, you can use Helm chart for Cryostat installation on Kubernetes. Firstly, let’s add the following repository:
$ helm repo add openshift https://charts.openshift.io/

Then, install the chart with the following command:
$ helm install my-cryostat openshift/cryostat --version 0.4.0

Once the OLM is running on our cluster, we can proceed to the Cryostat installation. We can find the required YAML manifest with the Subscription declaration in the Operator Hub. Let’s just apply the manifest to the target with the following command:

$ kubectl create -f https://operatorhub.io/install/cryostat-operator.yaml

By default, this operator will be installed in the operators namespace and will be usable from all namespaces in the cluster. After installation, we can verify if the operator works fine by executing the following command:

$ kubectl get csv -n operators

In order to simplify the Cryostat installation process, we can use OpenShift. With OpenShift we don’t need to install OLM, since it is already there. We just need to find the “Red Hat build of Cryostat” operator in the Operator Hub and install it using OpenShift Console. By default, the operator is available in the openshift-operators namespace.

Then, let’s create a namespace dedicated to running Cryostat and our sample app. The name of the namespace is demo-jfr.

$ kubectl create ns demo-jfr

Cryostat recommends using a cert-manager for traffic encryption. In our exercise, we disable that integration for simplification purposes. However, in the production environment, you should install “cert-manager” unless you do not use another solution for encrypting traffic. In order to run Cryostat in the selected namespace, we need to create the Cryostat object. The parameter spec.enableCertManager should be set to false.

apiVersion: operator.cryostat.io/v1beta1
kind: Cryostat
metadata:
  name: cryostat-sample
  namespace: demo-jfr
spec:
  enableCertManager: false
  eventTemplates: []
  minimal: false
  reportOptions:
    replicas: 0
  storageOptions:
    pvc:
      annotations: {}
      labels: {}
      spec: {}
  trustedCertSecrets: []

If everything goes fine, you should see the following pod in the demo-jfr namespace:

$ kubectl get po -n demo-jfr
NAME                               READY   STATUS    RESTARTS   AGE
cryostat-sample-5c57c9b8b8-smzx9   3/3     Running   0          60s

Here’s a list of Kubernetes Services. The Cryostat Dashboard is exposed by the cryostat-sample Service under the 8181 port.

$ kubectl get svc -n demo-jfr
NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)             AGE
cryostat-sample           ClusterIP   172.31.56.83    <none>        8181/TCP,9091/TCP   70m
cryostat-sample-grafana   ClusterIP   172.31.155.26   <none>        3000/TCP            70m

We can access the Cryostat dashboard using the Kubernetes Ingress or OpenShift Route. Currently, there are no apps to monitor.

Create Sample Java App

We build a sample Java app using the Spring Boot framework. Our app exposes a single REST endpoint. As you see the endpoint implementation is very simple. The pingWithRandomDelay() method adds a random delay between 0 and 3 seconds and returns the string. However, there is one interesting thing inside that method. We are creating the ProcessingEvent object (1). Then, we call its begin method just before sleeping the thread (2). After the method is resumed we call the commit method on the ProcessingEvent object (3). In this inconspicuous way, we are generating our first custom JFR event. This event aims to monitor the processing time of our method.

@RestController
@RequestMapping("/callme")
public class CallmeController {

   private static final Logger LOGGER = LoggerFactory.getLogger(CallmeController.class);

   private Random random = new Random();
   private AtomicInteger index = new AtomicInteger();

   @Value("${VERSION}")
   private String version;

   @GetMapping("/ping-with-random-delay")
   public String pingWithRandomDelay() throws InterruptedException {
      int r = new Random().nextInt(3000);
      int i = index.incrementAndGet();
      ProcessingEvent event = new ProcessingEvent(i); // (1)
      event.begin(); // (2)
      LOGGER.info("Ping with random delay: id={}, name={}, version={}, delay={}", i,
             buildProperties.isPresent() ? buildProperties.get().getName() : "callme-service", version, r);
      Thread.sleep(r);
      event.commit(); // (3)
      return "I'm callme-service " + version;
   }

}

Let’s switch to the ProcessingEvent implementation. Our custom event needs to extend the jdk.jfr.Event abstract class. It contains a single parameter id. We can use some additional labels to improve the event presentation in the JFR graphical tools. The event will be visible under the name set in the @Name annotation and under the category set in the @Category annotation. We also need to annotate the parameter @Label to make it visible as part of the event.

@Name("ProcessingEvent")
@Category("Custom Events")
@Label("Processing Time")
public class ProcessingEvent extends Event {
    @Label("Event ID")
    private Integer id;

    public ProcessingEvent(Integer id) {
        this.id = id;
    }

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }
}

Of course, our app will generate a lot of standard JFR events useful for profiling and monitoring. But we could also monitor our custom event.

Build App Image and Deploy on Kubernetes

Once we finish the implementation, we may build the container image of our Spring Boot app. Spring Boot comes with a feature for building container images based on the Cloud Native Buildpacks. In the Maven pom.xml you will find a dedicated profile under the build-image id. Once you activate such a profile, it will build the image using the Paketo builder-jammy-base image.

<profile>
  <id>build-image</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-maven-plugin</artifactId>
        <configuration>
          <image>
            <builder>paketobuildpacks/builder-jammy-base:latest</builder>
            <name>piomin/${project.artifactId}:${project.version}</name>
          </image>
        </configuration>
        <executions>
          <execution>
            <goals>
              <goal>build-image</goal>
            </goals>
          </execution>
        </executions>
      </plugin>
    </plugins>
  </build>
</profile>

Before running the build we should start Docker on the local machine. After that, we should execute the following Maven command:

$ mvn clean package -Pbuild-image -DskipTests

With the build-image profile activated, Spring Boot Maven Plugin builds the image of our app. You should have a similar result as shown below. In my case, the image tag is piomin/callme-service:1.2.1.

By default, Paketo Java Buildpacks uses BellSoft Liberica JDK. With the Paketo BellSoft Liberica Buildpack, we can easily enable Java Flight Recorder for the container using the BPL_JFR_ENABLED environment variable. In order to expose data for Cryostat, we also need to enable the JMX port. In theory, we could use BPL_JMX_ENABLED and BPL_JMX_PORT environment variables for that. However, that option includes some additional configuration to the java command parameters that break the Cryostat discovery. This issue has been already described here. Therefore we will use the JAVA_TOOL_OPTIONS environment variable to set the required JVM parameters directly on the running command.

Instead of exposing the JMX port for discovery, we can include the Cryostat agent in the app dependencies. In that case, we should set the address of the Cryostat API in the Kubernetes Deployment manifest. However, I prefer an approach that doesn’t require any changes on the app side.

Now, let’s back to the Cryostat app discovery. Cryostat is able to automatically detect pods with a JMX port exposed. It requires the concrete configuration of the Kubernetes Service. We need to set the name of the port to jfr-jmx. In theory, we can expose JMX on any port we want, but for me anything other than 9091 caused discovery problems on Cryostat. In the Deployment definition, we have to set the BPL_JFR_ENABLED env to true, and the JAVA_TOOL_OPTIONS to -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.port=9091.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: callme-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: callme-service
  template:
    metadata:
      labels:
        app: callme-service
    spec:
      containers:
        - name: callme-service
          image: piomin/callme-service:1.2.1
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
            - containerPort: 9091
          env:
            - name: VERSION
              value: "v1"
            - name: BPL_JFR_ENABLED
              value: "true"
            - name: JAVA_TOOL_OPTIONS
              value: "-Dcom.sun.management.jmxremote.port=9091 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false"
---
apiVersion: v1
kind: Service
metadata:
  name: callme-service
  labels:
    app: callme-service
spec:
  type: ClusterIP
  ports:
  - port: 8080
    name: http
  - port: 9091
    name: jfr-jmx
  selector:
    app: callme-service

Let’s apply our deployment manifest to the demo-jfr namespace:

$ kubectl apply -f k8s/deployment-jfr.yaml -n demo-jfr

Here’s a list of pods of our callme-service app:

$ kubectl get po -n demo-jfr -l app=callme-service -o wide
NAME                              READY   STATUS    RESTARTS   AGE   IP            NODE
callme-service-6bc5745885-kvqfr   1/1     Running   0          31m   10.134.0.29   worker-cluster-lvsqq-1

Using Cryostat with JFR

View Default Dashboards

Cryostat automatically detects all the pods related to the Kubernetes Service that expose the JMX port. Once we switch to the Cryostat Dashboard, we will see the name of our pod in the “Target” dropdown. The default dashboard shows diagrams illustrating CPU load, heap memory usage, and a number of running Java threads.

java-flight-recorder-kubernetes-dashboard

Then, we can go to the “Recordings” section. It shows a list of active recordings made by Java Flight Recorder for our app running on Kubernetes. By default, Cryostat creates and starts a single recording per each detected target.

We can expand the selected record to see a detailed view. It provides a summarized panel divided into several different categories like heap, memory leak, or exceptions. It highlights warnings with a yellow color and problems with a red color.

java-flight-recorder-kubernetes-panel

We can display a detailed description of each case. We just need to click on the selected field with a problem name. The detailed description will appear in the context menu.

java-flight-recorder-kubernetes-description

Create and Use a Custom Event Template

We can create a custom recording strategy by defining a new event template. Firstly, we need to go to the “Events” section, and then to the “Event Templates” tab. There are three built-in templates. We can use each of them as a base for our custom template. After deciding which of them to choose we can download it to our laptop. The default file extension is *.jfc.

java-flight-recorder-kubernetes-event-templates

In order to edit the *.jfc files we need a special tool called JDK Mission Control. Each vendor provides such a tool for their distribution of JDK. In our case, it is BellSoft Liberica. Once we download and install Liberica Mission Control on the laptop we should go to Window -> Flight Recording Template Manager.

java-flight-recorder-kubernetes-mission-control

With the Flight Recording Template Manager, we can import and edit an exported event template. I choose the higher monitoring for “Garbage Collection”, “Allocation Profiling”, “Compiler”, and “Thread Dump”.

java-flight-recorder-kubernetes-template-manager

Once a new template is ready, we should save it under the selected name. For me, it is the “Continuous Detailed” name. After that, we need to export the template to the file.

Then, we need to switch to the Cryostat Dashboard. We have to import the newly created template exported to the *.jfc file.

Once you import the template, you should see a new strategy in the “Event Templates” section.

We can create a recording based on our custom “Continuous_Detailed” template. After some time, Cryostat should gather data generated by the Java Flight Recorder for the app running on Kubernetes. However, this time we want to make some advanced analysys using Liberica Mission Control rather than just with the Cryostat Dashboard. Therefore we will export the recording to the *.jfr file. Such a file may be then imported to the JDK Mission Control tool.

Use the JDK Mission Control Tool

Let’s open the exported *.jfr file with Liberica Mission Control. Once we do it, we can analyze all the important aspects related to the performance of our Java app. We can display a table with memory allocation per the object type.

We can display a list of running Java threads.

Finally, we go to the “Event Browser” section. In the “Custom Events” category we should find our custom event under the name determined by the @Label annotation on the ProcessingEvent class. We can see the history of all generated JFR events together with the duration, start time, and the name of the processing thread.

Final Thoughts

Cryostat helps you to manage the Java Flight Recorder on Kubernetes at scale. It provides a graphical dashboard that allows to monitoring of all the Java workloads that expose JFR data over JMX. The important thing is that even after an app crash we can export the archived monitoring report and analyze it using advanced tools like JDK Mission Control.

The post Java Flight Recorder on Kubernetes appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/02/13/java-flight-recorder-on-kubernetes/feed/ 0 14957 OpenShift Builds with Shipwright and Cloud Native Buildpacks https://piotrminkowski.com/2024/02/08/openshift-builds-with-shipwright-and-cloud-native-buildpacks/ https://piotrminkowski.com/2024/02/08/openshift-builds-with-shipwright-and-cloud-native-buildpacks/#respond Thu, 08 Feb 2024 13:04:38 +0000 https://piotrminkowski.com/?p=14928 In this article, you will learn how to build the images of your apps on OpenShift with Shipwright and Cloud Native Buildpacks. Cloud Native Buildpacks allow us to easily build container images from the app source code. On the other hand, Shipwright is a Kubernetes-native tool that provides mechanisms for declaring and reusing several build […]

The post OpenShift Builds with Shipwright and Cloud Native Buildpacks appeared first on Piotr's TechBlog.

]]> In this article, you will learn how to build the images of your apps on OpenShift with Shipwright and Cloud Native Buildpacks. Cloud Native Buildpacks allow us to easily build container images from the app source code. On the other hand, Shipwright is a Kubernetes-native tool that provides mechanisms for declaring and reusing several build image strategies. Such a strategy is correlated with the tool used for building images. Shipwright supports Kaniko, OpenShift Source-to-image (S2I), Buildah, ko, BuiltKit, and of course, Cloud Native Buildpacks.

You can also run Cloud Native Buildpacks on “vanilla” Kubernetes together with e.g. Tekton. In the following article on my blog, you will find information on how to use Buildpacks in your Tekton CI pipeline for building Java app images.

Introduction

If you already have some experience with OpenShift, you probably know that it comes with its own mechanism for building container images. Such a build is fully executed on the OpenShift cluster. OpenShift provides the BuildConfig CRD responsible for a build configuration. There are three primary build strategies available: Docker, S2I, and custom build. By the way, you can build the image in any way you want and still run such a container on OpenShift. However, today we are discussing the OpenShift recommended approach for building container images called “OpenShift Builds”.

Recently, Red Hat has introduced a new supported way of using “OpenShift Builds” based on the Shipwright project. Therefore, you can currently find two chapters in the OpenShift documentation: “Builds with BuildConfig” and “Builds with Shipwright”. Maybe Shipwright will become a single, default strategy in the future, but for now, we can decide which of them to use.

In this article, I’ll show you how to install and manage Shipwright on OpenShift. By default, Red Hat supports Source-to-Image (S2I) and Buildah build strategies. We will create our own strategy for using the Cloud Native Buildpacks.

Install Builds for Red Hat OpenShift Operator

In order to enable Shiwright builds on OpenShift, we need to install the operator called “Builds for Red Hat OpenShift”. It depends on the OpenShift Pipelines operator. However, we don’t have to take care of it, since that operator is automatically installed together with the “Builds for Red Hat OpenShift” operator. In the OpenShift Console go to the Operators -> Operator Hub, find and install the required operator.

openshift-shipwright-operators

After that, we need to create the ShiwrightBuild object. It needs to contain the name of the target namespace for running Shipwright in the targetNamespace field. In my case, the target namespace is openshift-builds.

apiVersion: operator.shipwright.io/v1alpha1
kind: ShipwrightBuild
metadata:
  name: openshift-builds
spec:
  targetNamespace: openshift-builds

We can easily create the object on the operator page in OpenShift Console. If the status is “Ready” it means that Shipwright is running on our cluster.

Let’s display a list of pods inside the openshift-builds namespace. As you see, two pods are running there.

$ oc get po -n openshift-builds
NAME                                           READY   STATUS    RESTARTS   AGE
shipwright-build-controller-5d5b4f7d97-llj4f   1/1     Running   0          2m31s
shipwright-build-webhook-76775cf988-pjk6k      1/1     Running   0          2m31s

This step is optional. We can install the Shipwright CLI on our laptop to interact with the cluster. The release binary is available in the following repository under the Releases section. Once you install it, you can execute the following command for verification:

$ shp version

OpenShift Console supports Shipwright Builds. Once you install and configure the operator, you will have a dedicated space inside the “Builds” section. You can create a new build, run a build, and display a history of previous runs.

openshift-shipwright-builds-section

Configure Build Strategy for Buildpacks

Before we will define any build, we need to configure the build strategy. From a high-level perspective, build strategy defines how to build an application with an image-building tool. We can create the BuildStrategy object for the namespace-scoped strategy, and the ClusterBuildStrategy object for the cluster-wide strategy. The “Builds for Red Hat OpenShift” operator comes with two predefined global strategies: buildah and source-to-image.

However, our goal is to create a strategy for Cloud Native Buildpacks. Fortunately, we can just copy such a strategy from the samples provided in the Shiwright repository on GitHub. We won’t get into the details of that strategy implementation. The YAML manifest is visible below. It uses the paketobuildpacks/builder-jammy-full:latest image during the build. We should apply that manifest to our OpenShift cluster.

apiVersion: shipwright.io/v1beta1
kind: ClusterBuildStrategy
metadata:
  name: buildpacks-v3
spec:
  volumes:
    - name: platform-env
      emptyDir: {}
  parameters:
    - name: platform-api-version
      description: The referenced version is the minimum version that all relevant buildpack implementations support.
      default: "0.7"
  steps:
    - name: build-and-push
      image: docker.io/paketobuildpacks/builder-jammy-full:latest
      env: 
        - name: CNB_PLATFORM_API
          value: $(params.platform-api-version)
        - name: PARAM_SOURCE_CONTEXT
          value: $(params.shp-source-context)
        - name: PARAM_OUTPUT_IMAGE
          value: $(params.shp-output-image)
      command:
        - /bin/bash
      args:
        - -c
        - |
          set -euo pipefail

          echo "> Processing environment variables..."
          ENV_DIR="/platform/env"

          envs=($(env))

          # Denying the creation of non required files from system environments.
          # The creation of a file named PATH (corresponding to PATH system environment)
          # caused failure for python source during pip install (https://github.com/Azure-Samples/python-docs-hello-world)
          block_list=("PATH" "HOSTNAME" "PWD" "_" "SHLVL" "HOME" "")

          for env in "${envs[@]}"; do
            blocked=false

            IFS='=' read -r key value string <<< "$env"

            for str in "${block_list[@]}"; do
              if [[ "$key" == "$str" ]]; then
                blocked=true
                break
              fi
            done

            if [ "$blocked" == "false" ]; then
              path="${ENV_DIR}/${key}"
              echo -n "$value" > "$path"
            fi
          done

          LAYERS_DIR=/tmp/.shp/layers
          CACHE_DIR=/tmp/.shp/cache

          mkdir -p "$CACHE_DIR" "$LAYERS_DIR"

          function announce_phase {
            printf "===> %s\n" "$1" 
          }

          announce_phase "ANALYZING"
          /cnb/lifecycle/analyzer -layers="$LAYERS_DIR" "${PARAM_OUTPUT_IMAGE}"

          announce_phase "DETECTING"
          /cnb/lifecycle/detector -app="${PARAM_SOURCE_CONTEXT}" -layers="$LAYERS_DIR"

          announce_phase "RESTORING"
          /cnb/lifecycle/restorer -cache-dir="$CACHE_DIR" -layers="$LAYERS_DIR"

          announce_phase "BUILDING"
          /cnb/lifecycle/builder -app="${PARAM_SOURCE_CONTEXT}" -layers="$LAYERS_DIR"

          exporter_args=( -layers="$LAYERS_DIR" -report=/tmp/report.toml -cache-dir="$CACHE_DIR" -app="${PARAM_SOURCE_CONTEXT}")
          grep -q "buildpack-default-process-type" "$LAYERS_DIR/config/metadata.toml" || exporter_args+=( -process-type web ) 

          announce_phase "EXPORTING"
          /cnb/lifecycle/exporter "${exporter_args[@]}" "${PARAM_OUTPUT_IMAGE}"

          # Store the image digest
          grep digest /tmp/report.toml | tail -n 1 | tr -d ' \"\n' | sed s/digest=// > "$(results.shp-image-digest.path)"
      volumeMounts:
        - mountPath: /platform/env
          name: platform-env
      resources:
        limits:
          cpu: 500m
          memory: 1Gi
        requests:
          cpu: 250m
          memory: 65Mi
  securityContext:
    runAsUser: 1001
    runAsGroup: 1000

As you see, a new build strategy is available under the buildpacks-v3 name.

openshift-shipwright-strategies

Create Build for the Sample App

Once we create a strategy, we can proceed to the build creation. We will push the image to the registry on quay.io. It requires to have an existing account there. The quay.io registry allows us to create a robot account on a private account. Then we may use it to authenticate against the repository during the build. Firstly, go to your account settings and find the “Robot Accounts” section there. If you don’t have any robot account you need to create it. On the existing account choose the settings icon, and then the “View Credentials” item in the context menu.

In the “Kubernetes Secret” section download the YAML manifest containing the authentication credentials.

Our robot account must have the Write access to the target repository. In my case, it is the pminkows/sample-kotlin-spring registry.

Once you download the manifest you have to apply it to the OpenShift cluster. Let’s also create a new project with the demo-builds name.

$ oc new-project demo-builds
$ oc apply -f pminkows-piomin-secret.yml

The manifest contains a single Secret with our Quay robot account authentication credentials.

openshift-shipwright-pull-secret

Finally, we can create the Build object. It contains three sections. In the first of them, we are defining the address of the container image repository to push our output image (1). We need to authenticate against the repository with the previously created pminkows-piomin-pull-secret Secret. In the source section, we should provide the address of the repository with the app source code (2). The repository with a sample Spring Boot app is located on my GitHub account. In the last section, we need to set the build strategy (3). We choose the previously created buildpacks-v3 strategy for Cloud Native Buildpacks.

apiVersion: shipwright.io/v1beta1
kind: Build
metadata:
  name: sample-spring-kotlin-build
  namespace: demo-builds
spec:
  # (1)
  output:
    image: quay.io/pminkows/sample-kotlin-spring:1.0-shipwright
    pushSecret: pminkows-piomin-pull-secret
  # (2)
  source:
    git:
      url: https://github.com/piomin/sample-spring-kotlin-microservice.git
  # (3)
  strategy:
    name: buildpacks-v3
    kind: ClusterBuildStrategy

Once you apply the sample-spring-kotlin-build Build object you can switch to the “Builds” section in the OpenShift Console. In the “Shipwright Builds” tab, we can display a list of builds in the demo-builds namespace. In order to run the build, we just need to choose the “Start” option in the context menu.

openshift-shipwright-build-run

Running the Shipwright Build on OpenShift

Before we run the build, we have to set some permissions for the ServiceAccount. By default, Shipwright uses the pipeline ServiceAccount to run builds. You should already have the pipeline account created in the demo-builds namespace automatically with the project creation.

Shipwright requires a privileged SCC assigned to the pipeline ServiceAccount to run the build pods. Let’s add the required permission with the following commands:

$ oc adm policy add-scc-to-user privileged -z pipeline -n demo-builds
$ oc adm policy add-role-to-user edit -z pipeline -n demo-builds

We can start the build in the OpenShift Console or with the Shipwright CLI. Firstly, let’s display a list of builds using the following shp command:

$ shp build list
NAME				OUTPUT							STATUS
sample-spring-kotlin-build	quay.io/pminkows/sample-kotlin-spring:1.0-shipwright	all validations succeeded

Let’s run the build with the following command:

$ shp build run sample-spring-kotlin-build --follow

We can switch to the OpenShift Console. As you see, our build is running.

Shipwright runs the pod with Paketo Buildpacks Builder according to the strategy.

Here’s the output of the shp command. Once the image is ready, it is pushed to the Quay registry.

The build has been finished successfully. We can see it in the Shipwright “Builds” section in the OpenShift Console (status Succeeded).

openshift-shipwright-history

Let’s switch to the quai.io dashboard. As you see, the image tag 1.0-shipwright is already there.

Of course, we can build many times. The history of executions is available in the BuildRuns tab for each Build.

Final Thoughts

With Shipwright you can easily switch between different build image strategies on OpenShift. You are not limited only to the Source-to-image (S2I) or Docker strategy, but you can use different tools for that including e.g. Cloud Native Buildpacks. For now, the strategy based on Buildpacks is not supported by Red Hat as the Shipwright builds feature. However, I hope it will change soon 🙂

The post OpenShift Builds with Shipwright and Cloud Native Buildpacks appeared first on Piotr's TechBlog.

]]> https://piotrminkowski.com/2024/02/08/openshift-builds-with-shipwright-and-cloud-native-buildpacks/feed/ 0 14928