The other potential solution in that context is the Vertical Pod Autoscaler. In the latest version, it supports in-place pod resize. Vertical Pod Autoscaler (VPA) in Kubernetes automatically adjusts CPU and memory requests/limits for pods based on their actual usage, ensuring containers receive the appropriate resources. Unlike the Horizontal Pod Autoscaler (HPA), which scales resources, not replicas, it may restart pods to apply changes. For now, VPA does not support this use case, but once this feature is implemented, the situation will change.

On the other hand, Kube Startup CPU Boost is a dedicated feature for scenarios with high CPU requirements during app startup. It is a controller that increases CPU resource requests and limits during Kubernetes workload startup. Once the workload is up and running, the resources are set back to their original values. Let’s see how this solution works in practice!

If you’re looking to go deeper into the practical side of running Java applications on Kubernetes, I explore these topics in much more detail in my book, Hands-On Java with Kubernetes.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. The sample application is based on Spring Boot and exposes several REST endpoints. However, in this exercise, we will use a ready-made image published on my Quay: quay.io/pminkows/sample-kotlin-spring:1.5.1.1.

Install Kube Startup CPU Boost

The Kubernetes cluster you are using must enable the In-Place Pod Resize feature. The activation method may vary by Kubernetes distribution. For the Minikube I am using in today’s example, it looks like this:

minikube start --memory='8gb' --cpus='6' --feature-gates=InPlacePodVerticalScaling=true

minikube start --memory='8gb' --cpus='6' --feature-gates=InPlacePodVerticalScaling=true

After that, we can proceed with installing the Kube Startup CPU Boost controller. There are several ways to achieve it. The easiest way to do this is with a Helm chart. Let’s add the following Helm repository:

helm repo add kube-startup-cpu-boost https://google.github.io/kube-startup-cpu-boost

helm repo add kube-startup-cpu-boost https://google.github.io/kube-startup-cpu-boost

Then, we can install the kube-startup-cpu-boost chart in the dedicated kube-startup-cpu-boost-system namespace using the following command:

helm install -n kube-startup-cpu-boost-system kube-startup-cpu-boost \
  kube-startup-cpu-boost/kube-startup-cpu-boost --create-namespace

helm install -n kube-startup-cpu-boost-system kube-startup-cpu-boost \
  kube-startup-cpu-boost/kube-startup-cpu-boost --create-namespace

If the installation was successful, you should see the following pod running in the kube-startup-cpu-boost-system namespace as below.

$ kubectl get pod -n kube-startup-cpu-boost-system
NAME                                                         READY   STATUS    RESTARTS   AGE
kube-startup-cpu-boost-controller-manager-75f95d5fb6-692s6   1/1     Running   0          36s

$ kubectl get pod -n kube-startup-cpu-boost-system
NAME                                                         READY   STATUS    RESTARTS   AGE
kube-startup-cpu-boost-controller-manager-75f95d5fb6-692s6   1/1     Running   0          36s

Install Monitoring Stack (optional)

Then, we can install Prometheus monitoring. It is an optional step to verify pod resource usage in the graphical form. Firstly, let’s install the following Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

After that, we can install the latest version of the kube-prometheus-stack chart in the monitoring namespace.

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

helm install my-kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

Let’s verify the installation succeeded by listing the pods running in the monitoring namespace.

$ kubectl get pods -n monitoring
NAME                                                         READY   STATUS    RESTARTS   AGE
alertmanager-my-kube-prometheus-stack-alertmanager-0         2/2     Running   0          38s
my-kube-prometheus-stack-grafana-f8bb6b8b8-mzt4l             3/3     Running   0          48s
my-kube-prometheus-stack-kube-state-metrics-99f4574c-bf5ln   1/1     Running   0          48s
my-kube-prometheus-stack-operator-6d58dd9d6c-6srtg           1/1     Running   0          48s
my-kube-prometheus-stack-prometheus-node-exporter-tdwmr      1/1     Running   0          48s
prometheus-my-kube-prometheus-stack-prometheus-0             2/2     Running   0          38s

$ kubectl get pods -n monitoring
NAME                                                         READY   STATUS    RESTARTS   AGE
alertmanager-my-kube-prometheus-stack-alertmanager-0         2/2     Running   0          38s
my-kube-prometheus-stack-grafana-f8bb6b8b8-mzt4l             3/3     Running   0          48s
my-kube-prometheus-stack-kube-state-metrics-99f4574c-bf5ln   1/1     Running   0          48s
my-kube-prometheus-stack-operator-6d58dd9d6c-6srtg           1/1     Running   0          48s
my-kube-prometheus-stack-prometheus-node-exporter-tdwmr      1/1     Running   0          48s
prometheus-my-kube-prometheus-stack-prometheus-0             2/2     Running   0          38s

Finally, we can expose the Prometheus console over localhost using the port forwarding feature:

kubectl port-forward svc/my-kube-prometheus-stack-prometheus 9090:9090 -n monitoring

kubectl port-forward svc/my-kube-prometheus-stack-prometheus 9090:9090 -n monitoring

Configure Kube Startup CPU Boost

The Kube Startup CPU Boost configuration is pretty intuitive. We need to create a StartupCPUBoost resource. It can manage multiple applications based on a given selector. In our case, it is a single sample-kotlin-spring Deployment determined by the app.kubernetes.io/name label (1). The next step is to define the resource management policy (2). The Kube Startup CPU Boost increases both request and limit by 50%. Resources should only be increased for the duration of the startup (3). Therefore, once the readiness probe succeeds, the resource level will return to its initial state. Of course, everything happens in-place without restarting the container.

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions: # (1)
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy: # (2)
    containerPolicies:
    - containerName: sample-kotlin-spring
      percentageIncrease:
        value: 50
  durationPolicy: # (3)
    podCondition:
      type: Ready
      status: "True"

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions: # (1)
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy: # (2)
    containerPolicies:
    - containerName: sample-kotlin-spring
      percentageIncrease:
        value: 50
  durationPolicy: # (3)
    podCondition:
      type: Ready
      status: "True"

Next, we will deploy our sample application. Here’s the Deployment manifest of our Spring Boot app. The name of the app container is sample-kotlin-spring, which matches the target Deployment name defined inside the StartupCPUBoost object (1). Then, we set the CPU limit to 500 millicores (2). There’s also a new field resizePolicy. It tells Kubernetes whether a change to CPU or memory can be applied in-place or requires a Pod restart. (3). The NotRequired value means that changing the resource limit or request will not trigger a pod restart. The Deployment object also contains a readiness probe that calls the GET/actuator/health/readiness exposed with the Spring Boot Actuator (4).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  namespace: demo
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/name: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
        app.kubernetes.io/name: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring # (1)
        image: quay.io/pminkows/sample-kotlin-spring:1.5.1.1
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m # (2)
            memory: "1Gi"
          requests:
            cpu: 200m
            memory: "256Mi"
        resizePolicy: # (3)
        - resourceName: "cpu"
          restartPolicy: "NotRequired"
        readinessProbe: # (4)
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  namespace: demo
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/name: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
        app.kubernetes.io/name: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring # (1)
        image: quay.io/pminkows/sample-kotlin-spring:1.5.1.1
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 500m # (2)
            memory: "1Gi"
          requests:
            cpu: 200m
            memory: "256Mi"
        resizePolicy: # (3)
        - resourceName: "cpu"
          restartPolicy: "NotRequired"
        readinessProbe: # (4)
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3

Here are the pod requests and limits configured by Kube Startup CPU Boost. As you can see, the request is set to 300m, while the limit is completely removed.

Once the application startup process completes, Kube Startup CPU Boost restores the initial request and limit.

Now we can switch to the Prometheus console to see the history of CPU request values for our pod. As you can see, the request was temporarily increased during the pod startup.

The chart below illustrates CPU usage when the application is launched and then during normal operation.

We can also define the fixed resources for a target container. The CPU requests and limits of the selected container will be set to the given values (1). If you do not want the operator to remove the CPU limit during boost time, set the REMOVE_LIMITS environment variable to false in the kube-startup-cpu-boost-controller-manager Deployment.

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions:
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy:
    containerPolicies: # (1)
    - containerName: sample-kotlin-spring
      fixedResources:
        requests: "500m"
        limits: "2"
  durationPolicy:
    podCondition:
      type: Ready
      status: "True"

apiVersion: autoscaling.x-k8s.io/v1alpha1
kind: StartupCPUBoost
metadata:
  name: sample-kotlin-spring
  namespace: demo
selector:
  matchExpressions:
  - key: app.kubernetes.io/name
    operator: In
    values: ["sample-kotlin-spring"]
spec:
  resourcePolicy:
    containerPolicies: # (1)
    - containerName: sample-kotlin-spring
      fixedResources:
        requests: "500m"
        limits: "2"
  durationPolicy:
    podCondition:
      type: Ready
      status: "True"

Conclusion

There are many ways to address application CPU demand during startup. First, you don’t need to set a CPU limit for Deployment. What’s more, many people believe that setting a CPU limit doesn’t make sense, but for different reasons. In this situation, the request issue remains, but given the short timeframe and the significantly higher usage than in the declaration, it isn’t material.

Other solutions are related to strictly Java features. If we compile the application natively with GraalVM or use the CRaC feature, we will significantly speed up startup and reduce CPU requirements.

Finally, several solutions rely on in-place resizing. If you use Kyverno, consider its mutate policy, which can modify resources in response to an application startup event. The Kube Startup CPU Boost tool described in this article operates similarly but is designed exclusively for this use case. In the near future, Vertical Pod Autoscaler will also offer a CPU boost via in-place resize.

The post Startup CPU Boost in Kubernetes with In-Place Pod Resize appeared first on Piotr's TechBlog.

In this article, I’ll show how you can achieve all the described things with the GitOps (Argo CD) approach and several tools around OpenShift/Kubernetes like Kyverno or Red Hat Service Interconnect (open-source Skupper project). We will also use Advanced Cluster Management for Kubernetes (ACM) to create and handle “ephemeral” OpenShift clusters. If you need an introduction to the GitOps approach in a multicluster OpenShift environment read the following article. It is also to familiarize with the idea behind multicluster communication through the Skupper project. In order to do that you can read the article about multicluster load balancing with Skupper on my blog.

Source Code

If you would like to try it by yourself, you can always take a look at my source code. In order to do that, you need to clone my GitHub repository. It contains several YAML manifests that allow us to manage OpenShift clusters in a GitOps way. For that exercise, we will use the manifests under the clusterpool directory. There are two subdirectories there: hub and managed. The manifests inside the hub directory should be applied to the management cluster, while the manifests inside the managed directory to the managed cluster. In our traffic bursts scenario, a single OpenShift acts as a hub and managed cluster, and it creates another managed (ephemeral) cluster.

Prerequisites

In order to start the exercise, we need a running Openshift that acts as a management cluster. It will create and configure the ephemeral cluster on AWS used to handle traffic volume peaks. In the first step, we need to install two operators on the management cluster: “Openshift GitOps” and “Advanced Cluster Management for Kubernetes”.

After that, we have to create the MultiClusterHub object, which runs and configures ACM:

kind: MultiClusterHub
apiVersion: operator.open-cluster-management.io/v1
metadata:
  name: multiclusterhub
  namespace: open-cluster-management
spec: {}

We also need to install Kyverno. Since there is no official operator for it, we have to leverage the Helm chart. Firstly, let’s add the following Helm repository:

$ helm repo add kyverno https://kyverno.github.io/kyverno/

Then, we can install the latest version of Kyverno in the kyverno namespace using the following command:

$ helm install my-kyverno kyverno/kyverno -n kyverno --create-namespace

Using OpenShift Cluster Pool

With ACM we can create a pool of Openshift clusters. That pool contains running or hibernated clusters. While a running cluster is just ready to work, a hibernated cluster needs to be resumed by ACM. We are defining a pool size and the number of running clusters inside that pool. Once we create the ClusterPool object ACM starts to provision new clusters on AWS. In our case, the pool size is 1, but the number of running clusters is 0. The object declaration also contains all things required to create a new cluster like the installation template (the aws-install-config Secret) or AWS account credentials reference (the aws-aws-creds Secret). Each cluster within that pool is automatically assigned to the interconnect ManagedClusterSet. The cluster set approach allows us to group multiple OpenShift clusters.

apiVersion: hive.openshift.io/v1
kind: ClusterPool
metadata:
  name: aws
  namespace: aws
  labels:
    cloud: AWS
    cluster.open-cluster-management.io/clusterset: interconnect
    region: us-east-1
    vendor: OpenShift
spec:
  baseDomain: sandbox449.opentlc.com
  imageSetRef:
    name: img4.12.36-multi-appsub
  installConfigSecretTemplateRef:
    name: aws-install-config
  platform:
    aws:
      credentialsSecretRef:
        name: aws-aws-creds
      region: us-east-1
  pullSecretRef:
    name: aws-pull-secret
  size: 1

So, as a result, there is only one cluster in the pool. ACM keeps that cluster in the hibernated state. It means that all the VMs with master and worker nodes are stopped. In order to resume the hibernated cluster we need to create the ClusterClaim object that refers to the ClusterPool. It is similar to clicking the Claim cluster link visible below. However, we don’t want to create that object directly, but as a reaction to the Kubernetes event.

Before we proceed, let’s just take a look at a list of virtual machines on AWS related to our cluster. As you see they are not running.

Claim Cluster From the Pool on Scaling Event

Now, the question is – what kind of event should result in getting a cluster from the pool? A single app could rely on the scaling event. So once the number of deployment pods exceeds the assumed threshold we will resume a hibernated cluster and run the app there. With Kyverno we can react to such scaling events by creating the ClusterPolicy object. As you see our policy monitors the Deployment/scale resource. The assumed maximum allowed pod for our app on the main cluster is 4. We need to put such a value in the preconditions together with the Deployment name. Once all the conditions are met we may generate a new Kubernetes resource. That resource is the ClusterClaim which refers to the ClusterPool we created in the previous section. It will result in getting a hibernated cluster from the pool and resuming it.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: aws
spec:
  background: true
  generateExisting: true
  rules:
    - generate:
        apiVersion: hive.openshift.io/v1
        data:
          spec:
            clusterPoolName: aws
        kind: ClusterClaim
        name: aws
        namespace: aws
        synchronize: true
      match:
        any:
          - resources:
              kinds:
                - Deployment/scale
      preconditions:
        all:
          - key: '{{request.object.spec.replicas}}'
            operator: Equals
            value: 4
          - key: '{{request.object.metadata.name}}'
            operator: Equals
            value: sample-kotlin-spring
  validationFailureAction: Audit

Kyverno requires additional permission to create the ClusterClaim object. We can easily achieve this by creating a properly annotated ClusterRole:

kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kyverno:create-claim
  labels:
    app.kubernetes.io/component: background-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
rules:
  - verbs:
      - create
      - patch
      - update
      - delete
    apiGroups:
      - hive.openshift.io
    resources:
      - clusterclaims

Once the cluster is ready we are going to assign it to the interconnect group represented by the ManagedClusterSet object. This group of clusters is managed by our instance of Argo CD from the openshift-gitops namespace. In order to achieve it we need to apply the following objects to the management OpenShift cluster:

apiVersion: cluster.open-cluster-management.io/v1beta2
kind: ManagedClusterSetBinding
metadata:
  name: interconnect
  namespace: openshift-gitops
spec:
  clusterSet: interconnect
---
apiVersion: cluster.open-cluster-management.io/v1beta1
kind: Placement
metadata:
  name: interconnect
  namespace: openshift-gitops
spec:
  predicates:
    - requiredClusterSelector:
        labelSelector:
          matchExpressions:
            - key: vendor
              operator: In
              values:
                - OpenShift
---
apiVersion: apps.open-cluster-management.io/v1beta1
kind: GitOpsCluster
metadata:
  name: argo-acm-importer
  namespace: openshift-gitops
spec:
  argoServer:
    argoNamespace: openshift-gitops
    cluster: openshift-gitops
  placementRef:
    apiVersion: cluster.open-cluster-management.io/v1beta1
    kind: Placement
    name: interconnect
    namespace: openshift-gitops

After applying the manifest visible above you should see that the openshift-gitops is managing the interconnect cluster group.

Automatically Sync Configuration for a New Cluster with Argo CD

In Argo CD we can define the ApplicationSet with the “Cluster Decision Resource Generator” (1). You can read more details about that type of generator here in the docs. It will create the Argo CD Application per each Openshift cluster in the interconnect group (2). Then, the newly created Argo CD Application will automatically apply manifests responsible for creating our sample Deployment. Of course, those manifests are available in the same repository inside the clusterpool/managed directory (3).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: cluster-init
  namespace: openshift-gitops
spec:
  generators:
    - clusterDecisionResource: # (1)
        configMapRef: acm-placement
        labelSelector:
          matchLabels:
            cluster.open-cluster-management.io/placement: interconnect # (2)
        requeueAfterSeconds: 180
  template:
    metadata:
      name: 'cluster-init-{{name}}'
    spec:
      ignoreDifferences:
        - group: apps
          kind: Deployment
          jsonPointers:
            - /spec/replicas
      destination:
        server: '{{server}}'
        namespace: interconnect
      project: default
      source:
        path: clusterpool/managed # (3)
        repoURL: 'https://github.com/piomin/openshift-cluster-config.git'
        targetRevision: master
      syncPolicy:
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Here’s the YAML manifest that contains the Deployment object and the Openshift Route definition. Pay attention to the three skupper.io/* annotations. We will let Skupper generate the Kubernetes Service to load balance between all running pods of our app. Finally, it will allow us to load balance between the pods spread across two Openshift clusters.

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app.kubernetes.io/instance: sample-kotlin-spring
  annotations:
    skupper.io/address: sample-kotlin-spring
    skupper.io/port: '8080'
    skupper.io/proxy: http
  name: sample-kotlin-spring
spec:
  replicas: 2
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
    spec:
      containers:
        - image: 'quay.io/pminkows/sample-kotlin-spring:1.4.39'
          name: sample-kotlin-spring
          ports:
            - containerPort: 8080
          resources:
            limits:
              cpu: 1000m
              memory: 1024Mi
            requests:
              cpu: 100m
              memory: 128Mi
---
apiVersion: route.openshift.io/v1
kind: Route
metadata:
  labels:
    app: sample-kotlin-spring
    app.kubernetes.io/component: sample-kotlin-spring
    app.kubernetes.io/instance: sample-spring-kotlin
  name: sample-kotlin-spring
spec:
  port:
    targetPort: port8080
  to:
    kind: Service
    name: sample-kotlin-spring
    weight: 100
  wildcardPolicy: None

Let’s check out how it works. I won’t simulate traffic bursts on OpenShift. However, you can easily imagine that our app is autoscaled with HPA (Horizontal Pod Autoscaler) and therefore is able to react to the traffic volume peak. I will just manually scale up the app to 4 pods:

Now, let’s switch to the All Clusters view. As you see Kyverno sent a cluster claim to the aws ClusterPool. The claim stays in the Pending status until the cluster won’t be resumed. In the meantime, ACM creates a new cluster to fill up the pool.

Once the cluster is ready you will see it in the Clusters view.

ACM automatically adds a cluster from the aws pool to the interconnect group (ManagedClusterSet). Therefore Argo CD is seeing a new cluster and adding it as a managed.

Finally, Argo CD generates the Application for a new cluster to automatically install all required Kubernetes objects.

Using Red Hat Service Interconnect

In order to enable Skupper for our apps we first need to install the Red Hat Service Interconnect operator. We can also do it in the GitOps way. We need to define the Subscription object as shown below (1). The operator has to be installed on both hub and managed clusters. Once we install the operator we need to enable Skupper in the particular namespace. In order to do that we need to define the ConfigMap there with the skupper-site name (2). Those manifests are also applied by the Argo CD Application described in the previous section.

apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: skupper-operator
  namespace: openshift-operators
  annotations:
    argocd.argoproj.io/sync-wave: "2"
spec:
  channel: alpha
  installPlanApproval: Automatic
  name: skupper-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
---
kind: ConfigMap
apiVersion: v1
metadata:
  name: skupper-site

Here’s the result of synchronization for the managed cluster.

We can switch to the OpenShift Console of the new cluster. The Red Hat Service Interconnect operator is ready.

Finally, we are at the final phase of our exercise. Both our clusters are running. We have already installed our sample app and the Skupper operator on both of them. Now, we need to link the apps running on different clusters into a single Skupper network. In order to do that, we need to let Skupper generate a connection token. Here’s the Secret object responsible for that. It doesn’t contain any data – just the skupper.io/type label with the connection-token-request value. Argo CD has already applied it to the management cluster in the interconnect namespace.

apiVersion: v1
kind: Secret
metadata:
  labels:
    skupper.io/type: connection-token-request
  name: token-req
  namespace: interconnect

As a result, Skupper fills the Secret object with certificates and a private key. It also overrides the value of the skupper.io/type label.

So, now our goal is to copy that Secret to the managed cluster. We won’t do that in the GitOps way directly, since the object was dynamically generated on OpenShift. However, we may use the SelectorSyncSet object provided by ACM. It can copy the secrets between the hub and managed clusters.

apiVersion: hive.openshift.io/v1
kind: SelectorSyncSet
metadata:
  name: skupper-token-sync
spec:
  clusterDeploymentSelector:
    matchLabels:
      cluster.open-cluster-management.io/clusterset: interconnect
  secretMappings:
    - sourceRef:
        name: token-req
        namespace: interconnect
      targetRef:
        name: token-req
        namespace: interconnect

Once the token is copied into the managed cluster, it should connect to the Skupper network existing on the main cluster. We can verify that everything works fine with the skupper CLI command. The following command prints all the pods from the Skupper network. As you see, we have 4 pods on the main (local) cluster and 2 pods on the managed (linked) cluster.

Let’s display the route of our service:

$ oc get route sample-kotlin-spring

Now, we can make a final test. Here’s the siege request for my route and cluster domain. It will send 10k requests via the Route. After running it, you can verify the logs to see if the traffic comes to all six pods spread across our two clusters.

$ siege -r 1000 -c 10  http://sample-kotlin-spring-interconnect.apps.jaipxwuhcp.eastus.aroapp.io/persons

Final Thoughts

Handling traffic bursts is one of the more interesting scenarios for a hybrid-cloud environment with OpenShift. With the approach described in that article, we can dynamically provision clusters and redirect traffic from on-prem to the cloud. We can do it in a fully automated, GitOps-based way. The features and tools around OpenShift allow us to cut down the cloud costs and speed up cluster startup. Therefore it reduces system downtime in case of any failures or unexpected situations.

The post Handle Traffic Bursts with Ephemeral OpenShift Clusters appeared first on Piotr's TechBlog.

Motivation

If you are running Java apps on Kubernetes you probably have already encountered the problem with slow startup after setting too low CPU limit. It occurs because Java apps usually need significantly more CPU during initialization than during standard work. If such applications specify requests and limits suited for regular operation, they may suffer from very long startup times. On the other hand, if they specify a high CPU limit just to start fast it may not be the optimal approach for managing resource limits on Kubernetes. You can find some considerations in this area in my article about best practices for Java apps on Kubernetes.

Thanks to the new feature such pods can request a higher CPU at the time of pod creation and can be resized down to normal running needs once the application has finished initializing. We will also consider how to automatically apply such changes on the cluster once the pod is ready. In order to do that, we will use Kyverno. Kyverno policies can mutate Kubernetes resources in reaction to admission callbacks – which perfectly matches our needs in this exercise.

I think our goal is pretty clear. Let’s begin!

Enable In-place Pod Vertical Scaling

Since the “in-place pod vertical scaling” feature is still in the alpha state we need to explicitly enable it on Kubernetes. I’m testing that feature on Minikube. Here’s my minikube starting command (you try with lower memory if you wish):

$ minikube start --memory='8g' \
  --feature-gates=InPlacePodVerticalScaling=true

Install Kyverno on Kubernetes

Before we deploy the app we need to install Kyverno and create its policy. However, our scenario is not very standard for Kyverno. Let’s take some time to analyze it. When creating a new Kubernetes Deployment we should set the right CPU to allow fast startup of our Java app. Once our app has started and is ready to work we will resize the limit to match the standard app requirements. We cannot do it until the app startup procedure is in progress. In other words, we are not waiting for the pod Running status…

… but for app container readiness inside the pod.

Here’s a picture that illustrates our scenario. We will set the CPU limit to 2 cores during startup. Once our app is started we decrease it to 500 millicores.

Now, let’s go back to Kyverno. We will install it on Kubernetes using the official Helm chart. In the first step we need to add the following Helm repository:

$ helm repo add kyverno https://kyverno.github.io/kyverno/

During the installation, we need to customize a single property. By default, Kyverno filters out updates made on Kubernetes by the members of the system:nodes group. One of those members is kubelet, which is responsible for updating the state of containers running on the node. So, if we want to catch the container-ready event from kubelet we need to override that behavior. That’s why we set the config.excludeGroups property as an empty array. Here’s our values.yaml file:

config:
  excludeGroups: []

Finally, we can install Kyverno on Kubernetes using the following Helm command:

$ helm install kyverno kyverno/kyverno -n kyverno \
  --create-namespace -f values.yaml

Kyverno has been installed in the kyverno namespace. Just to verify if everything works fine we can display a list of running pods:

$ kubectl get po -n kyverno
NAME                                             READY   STATUS    RESTARTS   AGE
kyverno-admission-controller-79dcbc777c-8pbg2    1/1     Running   0          55s
kyverno-background-controller-67f4b647d7-kp5zr   1/1     Running   0          55s
kyverno-cleanup-controller-566f7bc8c-w5q72       1/1     Running   0          55s
kyverno-reports-controller-6f96648477-k6dcj      1/1     Running   0          55s

Create a Policy for Resizing the CPU Limit

We want to trigger our Kyverno policy on pod start and its status update (1). We will apply the change to the resource only if the current readiness state is true (2). It is possible to select a target container using a special element called “anchor” (3). Finally, we can define a new CPU limit for the container inside the target pod with the patchStrategicMerge section (4).

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: resize-pod-policy
spec:
  mutateExistingOnPolicyUpdate: false
  rules:
    - name: resize-pod-policy
      match:
        any:
          - resources: # (1)
              kinds:
                - Pod/status
                - Pod
      preconditions: 
        all: # (2)
          - key: "{{request.object.status.containerStatuses[0].ready}}"
            operator: Equals
            value: true
      mutate:
        targets:
          - apiVersion: v1
            kind: Pod
            name: "{{request.object.metadata.name}}"
        patchStrategicMerge:
          spec:
            containers:
              - (name): sample-kotlin-spring # (3)
                resources:
                  limits:
                    cpu: 0.5 # (4)

Let’s apply the policy.

We need to add some additional privileges that allow the Kyverno background controller to update pods. We don’t need to create ClusterRoleBinding, but just a ClusterRole with the right aggregation labels in order for those permissions to take effect.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:update-pods
  labels:
    app.kubernetes.io/component: background-controller
    app.kubernetes.io/instance: kyverno
    app.kubernetes.io/part-of: kyverno
rules:
  - verbs:
      - patch
      - update
    apiGroups:
      - ''
    resources:
      - pods

After that, we may try to create a policy once again. As you see, this time there were no more problems with that.

Deploy the Java App and Resize CPU Limit After Startup

Let’s take a look at the Deployment manifest of our Java app. The name of the app container is sample-kotlin-spring, which matches the conditional "anchor" in the Kyverno policy (1). As you see I’m setting the CPU limit to 2 cores (2). There’s also a new field used here resizePolicy (3). I would not have to set it since the default value is NotRequired. It means that changing the resource limit or request will not result in a pod restart. The Deployment object also contains a readiness probe that calls the GET/actuator/health/readiness exposed with the Spring Boot Actuator (4).

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-kotlin-spring
  namespace: demo
  labels:
    app: sample-kotlin-spring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-kotlin-spring
  template:
    metadata:
      labels:
        app: sample-kotlin-spring
    spec:
      containers:
      - name: sample-kotlin-spring # (1)
        image: quay.io/pminkows/sample-kotlin-spring:1.5.1.1
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 2 # (2)
            memory: "1Gi"
          requests:
            cpu: 0.1
            memory: "256Mi"
        resizePolicy: # (3)
        - resourceName: "cpu"
          restartPolicy: "NotRequired"
        readinessProbe: # (4)
          httpGet:
            path: /actuator/health/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 5
          successThreshold: 1
          failureThreshold: 3

Once we deploy the app a new pod is starting. We can verify its current resource limits. As you see it is still 2 CPUs.

Our app starts around 10-15 seconds. Therefore the readiness check also waits 15 seconds after it begins to call the Actuator endpoint (the initialDelaySeconds parameter). After that, it finishes with success and our container switches to the ready state.

Then, Kyverno detects container status change and triggers the policy. The policy precondition is met since the container is ready. Now, we can verify the current CPU limit on the same pod. It is 500 millicores. You can also take a look at the Annotations field. It indicates

That’s exactly what we want to achieve. Now, we can scale up the number of running instances of our app just to continue testing. Then, you can verify by yourself that a new pod will also have its CPU limit modified after startup by Kyverno to 0.5 core.

$ kubectl scale --replicas=2 deployment sample-kotlin-spring -n demo

And the last thing. How long would it take to start our app if we set 500 millicores as the CPU limit at the beginning? For my app and such a CPU limit, it is around 40 seconds. So the difference is significant.

Final Thoughts

Finally, there is a solution for Java apps on Kubernetes to dynamically resize the CPU limit after startup. In this article, you can find my proposal for managing it automatically using a Kyverno policy. In our example, the pods have a different CPU limit than the limit declared in the Deployment object. However, I can imagine a policy that consists of two rules and just modifies the limit only for the time of startup.

The post Resize CPU Limit To Speed Up Java Startup on Kubernetes appeared first on Piotr's TechBlog.

Introduction

If I want to easily test a scenario with multiple Kubernetes clusters I usually use kind for that. You can find examples in some of my previous articles. For example, here is the article about Cilium cluster mesh. Or another one about mirroring traffic between multiple clusters with Istio. This time I’m going to try a slightly different solution – vcluster. It allows us to run virtual Kubernetes clusters inside the namespaces of other clusters. Those virtual clusters have a separate API server and a separate data store. We can easily interact with them the same as with the “real” clusters through the Kube context on the local machine. The vcluster and all of its workloads will be hosted in a single underlying host namespace. Once we delete a namespace we will remove the whole virtual cluster with all workloads.

How vcluster may help in our exercise? First of all, it creates all the resources on the “hosting” Kubernetes cluster. There is a dedicated namespace that contains a Secret with a certificate and private key. Based on that Secret we can automatically add a newly created cluster to the clusters managed by Argo CD. I’ll show you how can leverage Kyverno ClusterPolicy for that. I will trigger on new Secret creation in the virtual cluster namespace, and then generate a new Secret in the Argo CD namespace containing the cluster details.

Here is the diagram that illustrates our architecture. ArgoCD is managing multiple Kubernetes clusters and deploying the app across those clusters using the ApplicationSet object. Once a new cluster is created it is automatically included in the list of clusters managed by Argo CD. It is possible thanks to Kyverno policy that generates a new Secret with the argocd.argoproj.io/secret-type: cluster label in the argocd namespace.

Prerequisites

Of course, you need to have a Kubernetes cluster. In this exercise, I’m using Kubernetes on Docker Desktop. But you can as well use any other local distribution like minikube or a cloud-hosted instance. No matter which distribution you choose you also need to have:

1. Helm CLI – used to install Argo CD, Kyverno and vcluster on the “hosting” Kubernetes cluster
2. vcluster CLI – used to interact with virtual Kubernetes clusters. We can also use vcluster to create a virtual cluster, however, we can also do it directly using the Helm chart. You will vcluster CLI installation instructions are available here.

Running Virtual Clusters on Kubernetes

Let’s create our first virtual cluster on Kubernetes. In that approach, we can use the vcluster create command for that. Additionally, we need to sign the cluster certificate using the internal DNS name containing the name of the Service and a target namespace. Assuming that the name of the cluster is vc1, the default namespace name is vcluster-vc1. Therefore, the API server certificate should be signed for the vc1.vcluster-vc1 domain. Here is the appropriate values.yaml file that overrides default chart properties.

syncer:
  extraArgs:
  - --tls-san=vc1.vc1-vcluster

Then, we can install the first virtual cluster in the vcluster-vc1 namespace. By default, vcluster uses k3s distribution (to decrease resource consumption), so we will switch to vanilla k8s using the distro parameter:

$ vcluster create vc1 --upgrade --connect=false \
  --distro k8s \
  -f values.yaml 

We need to create another two virtual clusters with names vc2 and vc3. So you should repeat the same steps using the values.yaml and the vcluster create command dedicated to each of them. After completing the required steps we can display a list of running virtual clusters:

Each cluster has a dedicated namespace that contains all the required pods for k8s distribution.

$ kubectl get pod -n vcluster-vc1
NAME                                           READY   STATUS    RESTARTS   AGE
coredns-586cbcd49f-pkn5q-x-kube-system-x-vc1   1/1     Running   0          20m
vc1-7985c794d6-7pqln                           1/1     Running   0          21m
vc1-api-6564bf7bbf-lqqxv                       1/1     Running   0          39s
vc1-controller-9f98c7f9c-87tqb                 1/1     Running   0          23s
vc1-etcd-0                                     1/1     Running   0          21m

Now, we can switch to the newly create Kube context using the vcluster connect command. Under the hood, vcluster creates a Kube context with the vcluster_vc1_vcluster-vc1_docker-desktop name and exposes API outside of the cluster using the NodePort Service.

For example, we can display a list of namespaces. As you see it is different than a list on the “hosting” cluster.

$ kubectl get ns   
NAME              STATUS   AGE
default           Active   25m
kube-node-lease   Active   25m
kube-public       Active   25m
kube-system       Active   25m

In order to switch back to the “hosting” cluster just run the following command:

$ vcluster disconnect

Installing Argo CD on Kubernetes

In the next step, we will install Argo CD on Kubernetes. To do that, we will use an official Argo CD Helm chart. First, let’s add the following Helm repo:

$ helm repo add argo https://argoproj.github.io/argo-helm

Then we can install the latest version of Argo CD in the selected namespace. For it is the argocd namespace.

$ helm install argocd argo/argo-cd -n argocd --create-namespace

After a while, Argo CD should be installed. We will then use the UI dashboard to interact with Argo CD. Therefore let’s expose it outside the cluster using the port-forward command for the argocd-server Service. After that we can access the dashboard under the local port 8080:

$ kubectl port-foward svc/argocd-server 8080:80 -n argocd

The default username is admin. ArgoCD Helm chart generates the password automatically during the installation. And you will find it inside the argocd-initial-admin-secret Secret.

$ kubectl get secret argocd-initial-admin-secret \
  --template={{.data.password}} \
  -n argocd | base64 -D

Automatically Adding Argo CD Clusters with Kyverno

The main goal here is to automatically add a newly create virtual Kubernetes to the clusters managed by Argo CD. Argo CD stores the details about each managed cluster inside the Kubernetes Secret labeled with argocd.argoproj.io/secret-type: cluster. On the other hand vcluster stores cluster credentials in the Secret inside a namespace dedicated to the particular cluster. The name of the Secret is the name of cluster prefixed with vc-. For example, the Secret name for the vc1 cluster is vc-vc1.

Probably, there are several ways to achieve the goal described above. However, for me, the simplest way is through the Kyverno ClusterPolicy. Kyverno is able to not only validate the resources it can also create additional resources when a resource is created or updated. Before we start, we need to install Kyverno on Kubernetes. As usual, we will Helm chart for that. First, let’s add the required Helm repository:

$ helm repo add kyverno https://kyverno.github.io/kyverno/

Then, we can install it for example in the kyverno namespace with the following command:

$ helm install kyverno kyverno/kyverno -n kyverno --create-namespace

That’s all – we may create our Kyverno policy. Let’s discuss the ClusterPolicy fields step by step. By default, the policy will not be applied to the existing resource when it is installed. To change this behavior we need to set the generateExistingOnPolicyUpdate parameter to true (1). Now it will also be for existing resources (our virtual clusters are already running). The policy triggers for any existing or newly created Secret with name starting from vc- (2). It sets several variables using the context field (3).

The policy has an access to the source Secret fields, so it is able to get API server CA (4), client certificate (5), and private key (6). Finally, it generates a new Secret with the same name as a cluster name (8). We can retrieve the name of the cluster from the namespace of the source Secret (7). The generated Secret should contain the label argocd.argoproj.io/secret-type: cluster (10) and should be placed in the argocd namespace (9). We fill all the required fields of Secret using variables (11). ArgoCD can access vcluster internally using Kubernetes Service with the same as the vcluster name (12).

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: sync-secret
spec:
  generateExistingOnPolicyUpdate: true # (1)
  rules:
  - name: sync-secret
    match:
      any:
      - resources: # (2)
          names:
          - "vc-*"
          kinds:
          - Secret
    exclude:
      any:
      - resources:
          namespaces:
          - kube-system
          - default
          - kube-public
          - kyverno
    context: # (3)
    - name: namespace
      variable:
        value: "{{ request.object.metadata.namespace }}"
    - name: name
      variable:
        value: "{{ request.object.metadata.name }}"
    - name: ca # (4)
      variable: 
        value: "{{ request.object.data.\"certificate-authority\" }}"
    - name: cert # (5)
      variable: 
        value: "{{ request.object.data.\"client-certificate\" }}"
    - name: key # (6)
      variable: 
        value: "{{ request.object.data.\"client-key\" }}"
    - name: vclusterName # (7)
      variable:
        value: "{{ replace_all(namespace, 'vcluster-', '') }}"
        jmesPath: 'to_string(@)'
    generate:
      kind: Secret
      apiVersion: v1
      name: "{{ vclusterName }}" # (8)
      namespace: argocd # (9)
      synchronize: true
      data:
        kind: Secret
        metadata:
          labels:
            argocd.argoproj.io/secret-type: cluster # (10)
        stringData: # (11)
          name: "{{ vclusterName }}"
          server: "https://{{ vclusterName }}.{{ namespace }}:443" # (12)
          config: |
            {
              "tlsClientConfig": {
                "insecure": false,
                "caData": "{{ ca }}",
                "certData": "{{ cert }}",
                "keyData": "{{ key }}"
              }
            }

Once you created the policy you can display its status with the following command:

$ kubectl get clusterpolicy
NAME          BACKGROUND   VALIDATE ACTION   READY
sync-secret   true         audit             true

Finally, you should see the three following secrets inside the argocd namespace:

Deploy the App Across Multiple Kubernetes Clusters with ArgoCD

We can easily deploy the same app across multiple Kubernetes clusters with the ArgoCD ApplicationSet object. The ApplicationSet controller is automatically installed by the ArgoCD Helm chart. So, we don’t have to do anything additional to use it. ApplicationSet is doing a very simple thing. Based on the defined criteria it generates several ArgoCD Applications. There are several types of criteria available. One of them is the list of Kubernetes clusters managed by ArgoCD.

In order to create the Application per a managed cluster, we need to use a “Cluster Generator”. The ApplicationSet visible above automatically uses all clusters managed by ArgoCD (1). It provides several parameter values to the Application template. We can use them to generate a unique name (2) or set the target cluster name (4). In this exercise, we will deploy a simple Spring Boot app that exposes some endpoints over HTTP. The configuration is stored in the following GitHub repo inside the apps/simple path (3). The target namespace name is demo (5). The app is synchronized automatically with the configuration stored in the Git repo (6).

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: sample-spring-boot
  namespace: argocd
spec:
  generators:
  - clusters: {} # (1)
  template:
    metadata:
      name: '{{name}}-sample-spring-boot' # (2)
    spec:
      project: default
      source: # (3)
        repoURL: https://github.com/piomin/openshift-cluster-config.git
        targetRevision: HEAD
        path: apps/simple
      destination:
        server: '{{server}}' # (4)
        namespace: demo # (5)
      syncPolicy: # (6)
        automated:
          selfHeal: true
        syncOptions:
          - CreateNamespace=true

Let’s switch to the ArgoCD dashboard. We have four clusters managed by ArgoCD: three virtual clusters and a single “real” cluster in-cluster.

Therefore you should have four ArgoCD applications generated and automatically synchronized. It means that our Sporing Boot app is currently running on all the clusters.

Let’s connect with the vc1 virtual cluster:

$ vcluster connect vc1

We can display a list of running pods inside the demo namespace. Of course, you can repeat the same steps for another two virtual clusters.

We can access the HTTP through the Kubernetes Service just by running the following command:

$ kubectl port-forward svc/sample-spring-kotlin 8080:8080 -n demo

The app exposes Swagger UI with the list of available endpoints. You can access it under the /swagger-ui.html path.

Final Thoughts

In this article, I focused on simplifying deployment across multiple Kubernetes clusters as much as possible. We deployed our sample app across all running clusters using a single ApplicationSet CRD. We were able to add managed clusters with Kyverno policy automatically. Finally, we perform the whole exercise using a single “real” cluster, which hosted several virtual Kubernetes clusters with the vcluster tool. There is also a very interesting solution dedicated to a similar challenge based on OpenShift GitOps and Advanced Cluster Management for Kubernetes. You can read more it in my previous article.

The post Manage Multiple Kubernetes Clusters with ArgoCD appeared first on Piotr's TechBlog.