From the technical point of view, Backstage is a web frontend designed to run on Node.js. It is mostly written in TypeScript using the React framework. It has an extensible nature. Each time we need to integrate Backstage with some third-party software we need to install a dedicated plugin. Plugins are essentially individually packaged React components. Today you will learn how to install and configure plugins to integrate with GitHub, CircleCI, and Sonarqube.

It is the first article about Backstage on my blog. You can expect more in the future. However, before proceeding with this article, it is worth reading the following post. It explains how I create my repositories on GitHub and what tools I’m using to check the quality of the code and be up-to-date.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. This time, there are two sample Git repositories. The first of them contains software templates written in the Backstage technology called Skaffolder. On the other hand, the second repository contains the source code of the simple Spring Boot generated from the Backstage template. Once you clone both of those repos, you should just follow my further instructions.

Writing Software Templates in Backstage

We can create our own software templates using YAML notation or find existing examples on the web. Such software templates are very similar to the definition of Kubernetes objects. They have the apiVersion, kind (Template) fields, metadata, and spec fields. Each template must define a list of input variables and then a list of actions that the scaffolding service executes.

The Structure of the Repository with Software Templates

Let’s take a look at the structure of the repository containing our Spring Boot template. As you see, there is the template.yaml file with the software template YAML manifest and the skeleton directory with our app source code. Besides Java files, there is the Maven pom.xml, Renovate and CircleCI configuration manifests. The Scaffolder template input parameters determine the name of Java classes or packages. In the Skaffolder template, we set the default base package name and a domain object name. The catalog-info.yaml file contains a definition of the object required to register the app in the software catalog. As you can see, we are also parametrizing the names of the files with the domain object names.

.
├── skeleton
│   ├── .circleci
│   │   └── config.yml
│   ├── README.md
│   ├── catalog-info.yaml
│   ├── pom.xml
│   ├── renovate.json
│   └── src
│       ├── main
│       │   ├── java
│       │   │   └── ${{values.javaPackage}}
│       │   │       ├── Application.java
│       │   │       ├── controller
│       │   │       │   └── ${{values.domainName}}Controller.java
│       │   │       └── domain
│       │   │           └── ${{values.domainName}}.java
│       │   └── resources
│       │       └── application.yml
│       └── test
│           └── java
│               └── ${{values.javaPackage}}
│                   └── ${{values.domainName}}ControllerTests.java
└── template.yaml

13 directories, 11 files

Building Templates with Scaffolder

Now, let’s take a look at the most important element in our repository – the template manifest. It defines several input parameters with default values. We can set the name of our app (appName), choose a default branch name inside the Git repository (repoBranchName), Maven group ID (groupId), the name of the default Java package (javaPackage), or the base REST API controller path (apiPath). All these parameters are then used during code generation. If you need more customization in the templates, you should add other parameters in the manifest.

The steps section in the manifest defines the actions required to create a new app in Backstage. We are doing three things here. In the first step, we need to generate the Spring Boot app source by filling the templates inside the skeleton directory with parameters defined in the Scaffolder manifest. Then, we are publishing the generated code in the newly created GitHub repository. The name of the repository is the same as the app name (the appName parameter). The owner of the GitHub repository is determined by the values of the orgName parameter. Finally, we are registering the new component in the Backstage catalog by calling the catalog:register action.

apiVersion: scaffolder.backstage.io/v1beta3
kind: Template
metadata:
  name: spring-boot-basic-template
  title: Create a Spring Boot app
  description: Create a Spring Boot app
  tags:
    - spring-boot
    - java
    - maven
    - circleci
    - renovate
    - sonarqube
spec:
  owner: piomin
  system: piomin
  type: service

  parameters:
    - title: Provide information about the new component
      required:
        - orgName
        - appName
        - domainName
        - repoBranchName
        - groupId
        - javaPackage
        - apiPath
        - description
      properties:
        orgName:
          title: Organization name
          type: string
          default: piomin
        appName:
          title: App name
          type: string
          default: sample-spring-boot-app
        domainName:
          title: Name of the domain object
          type: string
          default: Person
        repoBranchName:
          title: Name of the branch in the Git repository
          type: string
          default: master
        groupId:
          title: Maven Group ID
          type: string
          default: pl.piomin.services
        javaPackage:
          title: Java package directory
          type: string
          default: pl/piomin/services
        apiPath:
          title: REST API path
          type: string
          default: /api/v1
        description:
          title: Description
          type: string
          default: Sample Spring Boot App
          
  steps:
    - id: sourceCodeTemplate
      name: Generating the Source Code Component
      action: fetch:template
      input:
        url: ./skeleton
        values:
          orgName: ${{ parameters.orgName }}
          appName: ${{ parameters.appName }}
          domainName: ${{ parameters.domainName }}
          groupId: ${{ parameters.groupId }}
          javaPackage: ${{ parameters.javaPackage }}
          apiPath: ${{ parameters.apiPath }}

    - id: publish
      name: Publishing to the Source Code Repository
      action: publish:github
      input:
        allowedHosts: ['github.com']
        description: ${{ parameters.description }}
        repoUrl: github.com?owner=${{ parameters.orgName }}&repo=${{ parameters.appName }}
        defaultBranch: ${{ parameters.repoBranchName }}
        repoVisibility: public

    - id: register
      name: Registering the Catalog Info Component
      action: catalog:register
      input:
        repoContentsUrl: ${{ steps.publish.output.repoContentsUrl }}
        catalogInfoPath: /catalog-info.yaml

  output:
    links:
      - title: Open the Source Code Repository
        url: ${{ steps.publish.output.remoteUrl }}
      - title: Open the Catalog Info Component
        icon: catalog
        entityRef: ${{ steps.register.output.entityRef }}

Generating Spring Boot Source Code

Here’s the template of the Maven pom.xml. Our app uses the current latest version of the Spring Boot framework and Java 21 for compilation. The Maven groupId and artifactId are taken from the Scaffolder template parameters. The pom.xml file also contains data required to integrate with a specific project on Sonarcloud (sonar.projectKey and sonar.organization).

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>${{ values.groupId }}</groupId>
    <artifactId>${{ values.appName }}</artifactId>
    <version>1.0-SNAPSHOT</version>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.3.0</version>
    </parent>

    <properties>
        <sonar.projectKey>${{ values.orgName }}_${{ values.appName }}</sonar.projectKey>
        <sonar.organization>${{ values.orgName }}</sonar.organization>
        <sonar.host.url>https://sonarcloud.io</sonar.host.url>
        <java.version>21</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springdoc</groupId>
            <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
            <version>2.5.0</version>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-devtools</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.instancio</groupId>
            <artifactId>instancio-junit</artifactId>
            <version>4.7.0</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <goals>
                            <goal>build-info</goal>
                        </goals>
                        <configuration>
                            <additionalProperties>
                                <java.target>${java.version}</java.target>
                                <time>${maven.build.timestamp}</time>
                            </additionalProperties>
                        </configuration>
                    </execution>
                </executions>
            </plugin>
            <plugin>
                <groupId>pl.project13.maven</groupId>
                <artifactId>git-commit-id-plugin</artifactId>
                <configuration>
                    <failOnNoGitDirectory>false</failOnNoGitDirectory>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.jacoco</groupId>
                <artifactId>jacoco-maven-plugin</artifactId>
                <version>0.8.12</version>
                <executions>
                    <execution>
                        <goals>
                            <goal>prepare-agent</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>report</id>
                        <phase>test</phase>
                        <goals>
                            <goal>report</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>

</project>

There are several Java classes generated during bootstrap. Here’s the @RestController class template. It uses three parameters defined in the Scaffolder template: groupId, domainName and apiPath. It imports the domain object class and exposes REST methods for CRUD operations. As you see, the implementation is very simple. It just uses the in-memory Java List to store the domain objects. However, it perfectly shows the idea behind Scaffolder templates.

package ${{ values.groupId }}.controller;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.web.bind.annotation.*;
import ${{ values.groupId }}.domain.${{ values.domainName }};

import java.util.ArrayList;
import java.util.List;

@RestController
@RequestMapping("${{ values.apiPath }}")
public class ${{ values.domainName }}Controller {

    private final Logger LOG = LoggerFactory.getLogger(${{ values.domainName }}Controller.class);
    private final List<${{ values.domainName }}> objs = new ArrayList<>();

    @GetMapping
    public List<${{ values.domainName }}> findAll() {
        return objs;
    }

    @GetMapping("/{id}")
    public ${{ values.domainName }} findById(@PathVariable("id") Long id) {
        ${{ values.domainName }} obj = objs.stream().filter(it -> it.getId().equals(id))
                .findFirst()
                .orElseThrow();
        LOG.info("Found: {}", obj.getId());
        return obj;
    }

    @PostMapping
    public ${{ values.domainName }} add(@RequestBody ${{ values.domainName }} obj) {
        obj.setId((long) (objs.size() + 1));
        LOG.info("Added: {}", obj);
        objs.add(obj);
        return obj;
    }

    @DeleteMapping("/{id}")
    public void delete(@PathVariable("id") Long id) {
        ${{ values.domainName }} obj = objs.stream().filter(it -> it.getId().equals(id)).findFirst().orElseThrow();
        objs.remove(obj);
        LOG.info("Removed: {}", id);
    }

    @PutMapping
    public void update(@RequestBody ${{ values.domainName }} obj) {
        ${{ values.domainName }} objTmp = objs.stream()
                .filter(it -> it.getId().equals(obj.getId()))
                .findFirst()
                .orElseThrow();
        objs.set(objs.indexOf(objTmp), obj);
        LOG.info("Updated: {}", obj.getId());
    }

}

Then, we can generate a test class to verify @RestController endpoints. The app is starting on the random port during the JUnit tests. In the first test, we are adding a new object into the store. Then we are verifying the GET /{id} endpoint works fine. Finally, we are removing the object from the store by calling the DELETE /{id} endpoint.

package ${{ values.groupId }};

import org.instancio.Instancio;
import org.junit.jupiter.api.MethodOrderer;
import org.junit.jupiter.api.Order;
import org.junit.jupiter.api.Test;
import org.junit.jupiter.api.TestMethodOrder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.boot.test.web.client.TestRestTemplate;
import ${{ values.groupId }}.domain.${{ values.domainName }};

import static org.junit.jupiter.api.Assertions.*;

@SpringBootTest(webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class ${{ values.domainName }}ControllerTests {

    private static final String API_PATH = "${{values.apiPath}}";

    @Autowired
    private TestRestTemplate restTemplate;

    @Test
    @Order(1)
    void add() {
        ${{ values.domainName }} obj = restTemplate.postForObject(API_PATH, Instancio.create(${{ values.domainName }}.class), ${{ values.domainName }}.class);
        assertNotNull(obj);
        assertEquals(1, obj.getId());
    }

    @Test
    @Order(2)
    void findAll() {
        ${{ values.domainName }}[] objs = restTemplate.getForObject(API_PATH, ${{ values.domainName }}[].class);
        assertTrue(objs.length > 0);
    }

    @Test
    @Order(2)
    void findById() {
        ${{ values.domainName }} obj = restTemplate.getForObject(API_PATH + "/{id}", ${{ values.domainName }}.class, 1L);
        assertNotNull(obj);
        assertEquals(1, obj.getId());
    }

    @Test
    @Order(3)
    void delete() {
        restTemplate.delete(API_PATH + "/{id}", 1L);
        ${{ values.domainName }} obj = restTemplate.getForObject(API_PATH + "/{id}", ${{ values.domainName }}.class, 1L);
        assertNull(obj.getId());
    }

}

Integrate with CircleCI and Renovate

Once I create a new repository on GitHub I want to automatically integrate it with CircleCI builds. I also want to update Maven dependencies versions automatically with Renovate to keep the project up to date. Since my GitHub account is connected to the CircleCI account and the Renovate app is installed there I just need to provide two configuration manifests inside the generated repository. Here’s the CircleCI config.yaml file. It runs the Maven build with JUnit tests and performs the Sonarqube scan in the sonarcloud.io portal.

version: 2.1

jobs:
  analyze:
    docker:
      - image: 'cimg/openjdk:21.0.2'
    steps:
      - checkout
      - run:
          name: Analyze on SonarCloud
          command: mvn verify sonar:sonar

executors:
  jdk:
    docker:
      - image: 'cimg/openjdk:21.0.2'

orbs:
  maven: circleci/maven@1.4.1

workflows:
  maven_test:
    jobs:
      - maven/test:
          executor: jdk
      - analyze:
          context: SonarCloud

Here’s the renovate.json manifest. Renovate will create a PR in GitHub each time it detects a new version of Maven dependency.

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base",":dependencyDashboard"
  ],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
      "automerge": true
    }
  ],
  "prCreation": "not-pending"
}

Register a new Component in the Software Catalog

Once we generate the whole code and publish it as the GitHub repository, we need to register a new component in the Backstage catalog. In order to achieve this, our repository needs to contain the Component manifest as shown below. Once again, we need to fill it with the parameter values during the bootstrap phase. It contains a reference to the CircleCI and Sonarqube projects and a generated GitHub repository in the annotations section. Here’s the catalog-info.yaml template:

apiVersion: backstage.io/v1alpha1
kind: Component
metadata:
  name: ${{ values.appName }}
  title: ${{ values.appName }}
  annotations:
    circleci.com/project-slug: github/${{ values.orgName }}/${{ values.appName }}
    github.com/project-slug: ${{ values.orgName }}/${{ values.appName }}
    sonarqube.org/project-key: ${{ values.orgName }}_${{ values.appName }}
  tags:
    - spring-boot
    - java
    - maven
    - circleci
    - renovate
    - sonarqube
spec:
  type: service
  owner: piotr.minkowski@gmail.com
  lifecycle: experimental

Running Backstage

Once we have the whole template ready, we can proceed to run the Backstage on our local machine. As I mentioned before, Backstage is a Node.js app, so we need to have several tools installed to be able to run it. By the way, the list of prerequisites is pretty large. You can find it under the following link. First of all, I had to downgrade the version of Node.js from the latest 22 to 18. We also need to install yarn and npx. If you have the following versions of those tools, you shouldn’t have any problems with running Backstage according to further instructions.

$ node --version
v18.20.3

$ npx --version
10.7.0

$ yarn --version
1.22.22

Running a Standalone Server Locally

We are going to run Backstage locally in the development mode as a standalone server. In order to achieve this, we first need to run the following command. It will create a new directory with a Backstage app inside. 

$ npx @backstage/create-app@latest

This may take some time. But if you see a similar result, it means that your instance is ready. However, before we start it we need to install some plugins and include some configuration settings. In that case, the name of our instance is backstage1.

Firstly, we should go to the backstage1 directory and take a look at the project structure. The most important elements for us are: the app-config.yaml file with configuration and the packages directory with the source code of the backend and frontend (app) modules.

├── README.md
├── app-config.local.yaml
├── app-config.production.yaml
├── app-config.yaml
├── backstage.json
├── catalog-info.yaml
├── dist-types
├── examples
├── lerna.json
├── node_modules
├── package.json
├── packages
│   ├── app
│   └── backend
├── playwright.config.ts
├── plugins
├── tsconfig.json
└── yarn.lock

The app is not configured according to our needs yet. However, we can run it with the following command just to try it out:

$ yarn dev

We can visit the UI available under the http://localhost:3000:

Provide Configuration and Install Plugins

In the first, we will analyze the app-config.yaml and add some configuration settings there. I will focus only on the aspects important to our exercise. The default configuration comes with enabled built-in integration with GitHub. We just need to generate a personal access token in GitHub and provide it as the GITHUB_TOKEN environment variable (1). Then, we need to integrate with Sonarqube also through the access token (2). Our portal will also display a list of CircleCI builds. Therefore, we need to include the CircleCI token as well (3). Finally, we should include the URL address of our custom Scaffolder template in the catalog section (4). It is located in our sample GitHub repository: https://github.com/piomin/backstage-templates/blob/master/templates/spring-boot-basic/skeleton/catalog-info.yaml.

app:
  title: Scaffolded Backstage App
  baseUrl: http://localhost:3000

organization:
  name: piomin

backend:
  baseUrl: http://localhost:7007
  listen:
    port: 7007
  csp:
    connect-src: ["'self'", 'http:', 'https:']
  cors:
    origin: http://localhost:3000
    methods: [GET, HEAD, PATCH, POST, PUT, DELETE]
    credentials: true
  database:
    client: better-sqlite3
    connection: ':memory:'

# (1)
integrations:
  github:
    - host: github.com
      token: ${GITHUB_TOKEN}

# (2)
sonarqube:
  baseUrl: https://sonarcloud.io
  apiKey: ${SONARCLOUD_TOKEN}

# (3)
proxy:
  '/circleci/api':
    target: https://circleci.com/api/v1.1
    headers:
      Circle-Token: ${CIRCLECI_TOKEN}
      
auth:
  providers:
    guest: {}

catalog:
  import:
    entityFilename: catalog-info.yaml
    pullRequestBranchName: backstage-integration
  rules:
    - allow: [Component, System, API, Resource, Location]
  locations:`
    - type: file
      target: ../../examples/entities.yaml
    - type: file
      target: ../../examples/template/template.yaml
      rules:
        - allow: [Template]
    # (4)
    - type: url
      target: https://github.com/piomin/backstage-templates/blob/master/templates/spring-boot-basic/template.yaml
      rules:
        - allow: [ Template ]
    - type: file
      target: ../../examples/org.yaml
      rules:
        - allow: [User, Group]

So, before starting Backstage we need to export both GitHub and Sonarqube tokens.

$ export GITHUB_TOKEN=<YOUR_GITHUB_TOKEN>
$ export SONARCLOUD_TOKEN=<YOUR_SONARCLOUD_TOKEN>
$ export CIRCLECI_TOKEN=<YOUR_CIRCLECI_TOKEN>

For those of you who didn’t generate Sonarcloud tokens before:

And similar operation for CicleCI:

Unfortunately, that is not all. Now, we need to install several required plugins. To be honest with you, plugin installation in Backstage is quite troublesome. Usually, we not only need to install such a plugin with yarn but also provide some changes in the packages directory. Let’s begin!

Enable GitHub Integration

Although integration with GitHub is enabled by default in the configuration settings we still need to install the plugin to be able to make some actions related to repositories. Firstly, from your Backstage instance root directory, you need to execute the following command:

$ yarn --cwd packages/backend add @backstage/plugin-scaffolder-backend-module-github

Then, go to the packages/backend/app/index.ts file and add a single highlighted line there. It imports the @backstage/plugin-scaffolder-backend-module-github module to the backend.

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));
backend.add(import('@backstage/plugin-auth-backend'));
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
backend.add(
  import('@backstage/plugin-catalog-backend-module-scaffolder-entity-model'),
);
backend.add(import('@backstage/plugin-permission-backend/alpha'));
backend.add(
  import('@backstage/plugin-permission-backend-module-allow-all-policy'),
);
backend.add(import('@backstage/plugin-search-backend/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-catalog/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-techdocs/alpha'));

backend.add(import('@backstage/plugin-scaffolder-backend-module-github'));
backend.add(import('@backstage-community/plugin-sonarqube-backend'));

backend.start();

After that, it will be possible to call the publish:github action defined in our template, which is responsible for creating a new GitHub repository with the Spring Boot app source code.

Enable Sonarqube Integration

In the next step, we need to install and configure the Sonarqube plugin. This time, we need to install both backend and frontend modules. Let’s begin with a frontend part. Firstly, we have to execute the following yarn command from the project root directory:

$ yarn --cwd packages/app add @backstage-community/plugin-sonarqube

Then, we need to edit the packages/app/src/components/catalog/EntityPage.tsx file to import the EntitySonarQubeCard object from the @backstage-community/plugin-sonarqube plugin. After that, we can include EntitySonarQubeCard component to the frontend page. For example, it can be placed as a part of the overview content.

import { EntitySonarQubeCard } from '@backstage-community/plugin-sonarqube';

// ... other imports
// ... other contents

const overviewContent = (
  <Grid container spacing={3} alignItems="stretch">
    {entityWarningContent}
    <Grid item md={6}>
      <EntityAboutCard variant="gridItem" />
    </Grid>
    <Grid item md={6} xs={12}>
      <EntityCatalogGraphCard variant="gridItem" height={400} />
    </Grid>
    <Grid item md={6}>
      <EntitySonarQubeCard variant="gridItem" />
    </Grid>
    <Grid item md={4} xs={12}>
      <EntityLinksCard />
    </Grid>
    <Grid item md={8} xs={12}>
      <EntityHasSubcomponentsCard variant="gridItem" />
    </Grid>
  </Grid>
);

Then, we can proceed with the backend plugin. Once again, we are installing with the yarn command:

$ yarn --cwd packages/backend add @backstage-community/plugin-sonarqube-backend

Finally, the same as for the GitHub plugin, go to the packages/backend/app/index.ts file and add a single highlighted line there to import the @backstage-community/plugin-sonarqube-backend to the backend module.

import { createBackend } from '@backstage/backend-defaults';

const backend = createBackend();

backend.add(import('@backstage/plugin-app-backend/alpha'));
backend.add(import('@backstage/plugin-proxy-backend/alpha'));
backend.add(import('@backstage/plugin-scaffolder-backend/alpha'));
backend.add(import('@backstage/plugin-techdocs-backend/alpha'));
backend.add(import('@backstage/plugin-auth-backend'));
backend.add(import('@backstage/plugin-auth-backend-module-guest-provider'));
backend.add(import('@backstage/plugin-catalog-backend/alpha'));
backend.add(
  import('@backstage/plugin-catalog-backend-module-scaffolder-entity-model'),
);
backend.add(import('@backstage/plugin-permission-backend/alpha'));
backend.add(
  import('@backstage/plugin-permission-backend-module-allow-all-policy'),
);
backend.add(import('@backstage/plugin-search-backend/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-catalog/alpha'));
backend.add(import('@backstage/plugin-search-backend-module-techdocs/alpha'));

backend.add(import('@backstage/plugin-scaffolder-backend-module-github'));
backend.add(import('@backstage-community/plugin-sonarqube-backend'));

backend.start();

Note that previously, we added the sonarqube section with the access token to the app-config.yaml file and we included annotation with the SonarCloud project key into the Backstage Component manifest. Thanks to that, we don’t need to do anything more in this part.

Enable CircleCI Integration

In order to install the CircleCI plugin, we need to execute the following yarn command:

$ yarn add --cwd packages/app @circleci/backstage-plugin

Then, we have to edit the packages/app/src/components/catalog/EntityPage.tsx file. The same as before we need to include the import section and choose a place on the frontend page to display the content.

// ... other imports

import {
  EntityCircleCIContent,
  isCircleCIAvailable,
} from '@circleci/backstage-plugin';

// ... other contents

const cicdContent = (
  <EntitySwitch>
    <EntitySwitch.Case if={isCircleCIAvailable}>
      <EntityCircleCIContent />
    </EntitySwitch.Case>
    <EntitySwitch.Case>
      <EmptyState
        title="No CI/CD available for this entity"
        missing="info"
        description="You need to add an annotation to your component if you want to enable CI/CD for it. You can read more about annotations in Backstage by clicking the button below."
        action={
          <Button
            variant="contained"
            color="primary"
            href="https://backstage.io/docs/features/software-catalog/well-known-annotations"
          >
            Read more
          </Button>
        }
      />
    </EntitySwitch.Case>
  </EntitySwitch>
);

Final Run

That’s all we need to configure before running the Backstage instance. Once again, we need to start the instance with the yarn dev command. After running the app, we should go to the “Create…” section in the left menu pane. You should see our custom template under the name “Create a Spring Boot app”. Click the “CHOOSE” button to create a new component from that template.

Then, we will see the form with several input parameters. I will just change the app name to the sample-spring-boot-app-backstage and leave the default values everywhere else.

Then, let’s just click the “CREATE” button on the next page.

After that, Backstage will generate all the required things from our sample template.

We can go to the app page in the Backstage catalog. As you see it contains the “Code Quality” section with the latest Sonrqube report for our newly generated app.

We can also switch to the “CI/CD” tab to see the history of the app builds in the CircleCI.

If you want to visit the example repository generated from the sample template discussed today, you can find it here.

Final Thoughts

Backstage is an example of a no-code IDP (Internal Developer Portal). IDP is an important part of the relatively new trend in software development called “Platform Engineering”. In this article, I showed you how to create “Golden Path Templates” using the technology called Scaffolder. Then, you could see how to run Backstage on the local machine and how to create an app source from the template. We installed some useful plugins to integrate our portal with GitHub, CircleCI, and Sonarqube. Plugins installation may cause some problems, especially for people without experience in Node.js and React. Hope it helps!

The post Getting Started with Backstage appeared first on Piotr's TechBlog.

Let’s consider what’s the problem with that approach. I’m usually sharing the same repository across multiple articles if they are closely related to each other. But despite that, I have more than 100 repositories with code examples. Once I create a repository I usually don’t have time to keep it up to date. I need a tool that will do that automatically for me. This, however, forces me to improve automatic tests. If I configure a tool that automatically updates a code in GitHub repositories, I need to verify that the change is valid and will not break the demo app.

There is another problem related to that. Classics to the genre – lack of automation tests… I was always focusing on creating the example app to show the use case described in the post, but not on building the valuable tests. It’s time to fix that! This is my first New Year’s resolution As you probably guessed, my work is still in progress. But even now, I can show you which tools I’m using for that and how to configure them. I will also share some first thoughts. Let’s begin!

First Problem: Not Maintained Repositories

Did you ever try to run the app from the source code created some years ago? In theory, everything should go fine. But in practice, several things may have changed. I can use a different version of e.g. Java or Maven than before. Even if have automated tests, they may not work fine especially since I didn’t use any tool to run the build and tests remotely. Of course, I don’t have such many old, unmaintained repositories. Sometimes, I was updating them manually, in particular, those more popular and shared across several articles.

Let’s just take a look at this example. It is from the following repository. I’m trying to generate a class definition from the Protocol Buffers schema file. As you see, the plugin used for that is not able to find the protoc executable. Honestly, I don’t remember how it worked before. Maybe I installed something on my laptop… Anyway, the solution was to use another plugin that doesn’t require any additional executables. Of course, I need to do it manually.

Let’s analyze another example. This time it fails during integration tests from another repository. The test is trying to connect to the Docker container. The problem here is that I was using Windows some years ago and Docker Toolbox was, by default, available under the 192.168.99.100 address. I should not leave such an address in the test. However, once again, I was just running all the tests locally, and at that time they finished successfully.

By the way, moving such a test to the CircleCI pipeline is not a simple thing to do. In order to run some containers (pact-broker with posgtresql) before the pipeline I decided to use Docker Compose. To run containers with Docker Compose I had to enable remote docker for CicrleCI as described here.

Second Problem: Updating Dependencies

If you manage application repositories that use several libraries, you probably know that an update is sometimes not just a formality. Even if that’s a patch or a minor update. Although my applications are usually not very complicated, the update of the Spring Boot version may be challenging. In the following example of Netflix DGS usage (GraphQL framework), I tried to update from the 2.4.2 to the 2.7.7 version. Here’s the result.

In that particular case, my app was initiating the H2 database with some data from the data.sql file. But since one of the 2.4.X Spring Boot version, the records from the data.sql are loaded before database schema initialization. The solution is to replace that file with the import.sql script or add the property spring.jpa.defer-datasource-initialization=true to the application properties. After choosing the second option we solved the problem… and then another one occurs. This time it is related to Netflix DGS and GraphQL Java libraries as described here.

Currently, according to the comments, there is no perfect solution to that problem with Maven. Probably I will have to wait for the next release of Netflix DGS or wait until they will propose the right solution.

Let’s analyze another example – once again with the Spring Boot update. This time it is related to the Spring Data and Embedded Mongo. The case is very interesting since it fails just on the remote builder. When I’m running the test on my local machine everything works perfectly fine.

A similar issue has been described here. However, the described solution doesn’t help me anymore. Probably I will decide to migrate my tests to the Testcontainers. By the way, it is also a very interesting example, since it has an impact only on the tests. So, even with a high level of automation, you will still need to do manual work.

Third Problem: Lack of Automated Tests

It is some kind of paradox – although I’m writing a lot about continuous delivery or tests I have a lot of repositories without any tests. Of course, when I was creating real applications for several companies I was adding many tests to ensure they will fine on production. But even for simple demo apps it is worth adding several tests that verify if everything works fine. In that case, I don’t have many small unit tests but rather a test that runs a whole app and verifies e.g. all the endpoints. Fortunately, the frameworks like Spring Boot or Quarkus provide intuitive tools for that. There are helpers for almost all popular solutions. Here’s my @SprignBootTest for GraphQL queries.

@SpringBootTest(webEnvironment = 
      SpringBootTest.WebEnvironment.RANDOM_PORT)
public class EmployeeQueryResolverTests {

    @Autowired
    GraphQLTestTemplate template;

    @Test
    void employees() throws IOException {
        Employee[] employees = template
           .postForResource("employees.graphql")
           .get("$.data.employees", Employee[].class);
        Assertions.assertTrue(employees.length > 0);
    }

    @Test
    void employeeById() throws IOException {
        Employee employee = template
           .postForResource("employeeById.graphql")
           .get("$.data.employee", Employee.class);
        Assertions.assertNotNull(employee);
        Assertions.assertNotNull(employee.getId());
    }

    @Test
    void employeesWithFilter() throws IOException {
        Employee[] employees = template
           .postForResource("employeesWithFilter.graphql")
           .get("$.data.employeesWithFilter", Employee[].class);
        Assertions.assertTrue(employees.length > 0);
    }
}

In the previous test, I’m using an in-memory H2 database in the background. If I want to test smth with the “real” database I can use Testcontainers. This tool runs the required container on Docker during the test. In the following example, we run PostgreSQL. After that, the Spring Boot application automatically connects to the database thanks to the @DynamicPropertySource annotation that sets the generated URL as the Spring property.

@SpringBootTest(webEnvironment = 
      SpringBootTest.WebEnvironment.RANDOM_PORT)
@Testcontainers
@TestMethodOrder(MethodOrderer.OrderAnnotation.class)
public class PersonControllerTests {

   @Autowired
   TestRestTemplate restTemplate;

   @Container
   static PostgreSQLContainer<?> postgres = 
      new PostgreSQLContainer<>("postgres:15.1")
           .withExposedPorts(5432);

   @DynamicPropertySource
   static void registerMySQLProperties(DynamicPropertyRegistry registry) {
       registry.add("spring.datasource.url", 
          postgres::getJdbcUrl);
       registry.add("spring.datasource.username", 
          postgres::getUsername);
       registry.add("spring.datasource.password", 
          postgres::getPassword);
   }

   @Test
   @Order(1)
   void add() {
       Person person = Instancio.of(Person.class)
               .ignore(Select.field("id"))
               .create();
       person = restTemplate
          .postForObject("/persons", person, Person.class);
       Assertions.assertNotNull(person);
       Assertions.assertNotNull(person.getId());
   }

   @Test
   @Order(2)
   void updateAndGet() {
       final Integer id = 1;
       Person person = Instancio.of(Person.class)
               .set(Select.field("id"), id)
               .create();
       restTemplate.put("/persons", person);
       Person updated = restTemplate
          .getForObject("/persons/{id}", Person.class, id);
       Assertions.assertNotNull(updated);
       Assertions.assertNotNull(updated.getId());
       Assertions.assertEquals(id, updated.getId());
   }

   @Test
   @Order(3)
   void getAll() {
       Person[] persons = restTemplate
          .getForObject("/persons", Person[].class);
       Assertions.assertEquals(1, persons.length);
   }

   @Test
   @Order(4)
   void deleteAndGet() {
       restTemplate.delete("/persons/{id}", 1);
       Person person = restTemplate
          .getForObject("/persons/{id}", Person.class, 1);
       Assertions.assertNull(person);
   }

}

In some cases, we may have multiple applications (or microservices) communicating with each other. We can mock that communication with the libraries like Mockito. On the other, we can simulate real HTTP traffic with the libraries like Hoverfly or Wiremock. Here’s the example with Hoverfly and the Spring Boot Test module.

@SpringBootTest(properties = { "POD_NAME=abc", "POD_NAMESPACE=default"}, 
   webEnvironment = SpringBootTest.WebEnvironment.RANDOM_PORT)
@ExtendWith(HoverflyExtension.class)
public class CallerControllerTests {

   @LocalServerPort
   int port;
   @Autowired
   TestRestTemplate restTemplate;

   @Test
   void ping(Hoverfly hoverfly) {
      String msg = "callme-service v1.0-SNAPSHOT (id=1): abc in default";
      hoverfly.simulate(dsl(
            service("http://callme-service.serverless.svc.cluster.local")
               .get("/callme/ping")
               .willReturn(success(msg, "text/plain"))));

      String response = restTemplate
         .getForObject("/caller/ping", String.class);
      assertNotNull(response);

      String c = "caller-service(id=1): abc in default is calling " + msg;
      assertEquals(c, response);
   }
}

Of course, these are just examples of tests. There are a lot of different tests and technologies used in all my repositories. Some others would be added in the near future Now, let’s go to the point.

Choosing the Right Tools

As mentioned in the introduction, I will use CircleCI and Renovate for managing my GitHub repositories. CircleCI is probably the most popular choice for running builds of open-source projects stored in GitHub repositories. GitHub also provides a tool for updating dependencies called Dependabot. However, Renovate has some significant advantages over Dependabot. It provides a lot of configuration options, may be run anywhere (including Kubernetes – more details here), and can integrate also with GitLab or Bitbucket. We will also use SonarCloud for a static code quality analysis.

Renovate is able to analyze not only the descriptors of traditional package managers like npm, Maven, or Gradle but also e.g. CircleCI configuration files or Docker image tags. Here’s a list of my requirements that the following tool needs to meet:

1. It should be able to perform different actions depending on the dependency update type (major, patch, or minor)
2. It needs to create PR on change and auto-merge it only if the build performed by CircleCI finishes successfully. Therefore it needs to wait for the status of that build
3. Auto-merge should not be enabled for major updates. They require approval from the repository admin

Renovate meets all these requirements. We can also easily install Renovate on GitHub and use it to update CircleCI configuration files inside repositories. In order to install Renovate on GitHub you need to go to the marketplace. After you install it, go the Settings, and then the Applications menu item. In order to set the list of repositories enabled for Renovate click the Configure button.

Then in the Repository Access section, you can enable all your repositories or choose several from the whole list.

Configure Renovate and CircleCI inside the GitHub Repository

Each GitHub repository has to contain CircleCI and Renovate configuration files. Renovate tries to detect the renovate.json file in the repository root directory. We don’t provide many configuration settings to achieve the expected results. By default, Renovate creates a pull request once it detects a new version of dependency but does not auto-merge it. We want to auto-merge all non-major changes. Therefore, we need to set a list of all update types merged automatically (minor, patch, pin, and digest).

By default, Renovate creates PR just after it creates a branch with a new version of the dependency. Because we are auto-merging all non-major PRs we need to force Renovate to create them only after the build on CircleCI finishes successfully. Once, all the tests on the newly created branch will be passed, Renovate creates PR and auto-merge if it does not contain major changes. Otherwise, it leaves the PR for approval. To achieve it, we to set the property prCreation to not-pending. Here’s the renovate.json file I’m using for all my GitHub repositories.

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": [
    "config:base",":dependencyDashboard"
  ],
  "packageRules": [
    {
      "matchUpdateTypes": ["minor", "patch", "pin", "digest"],
      "automerge": true
    }
  ],
  "prCreation": "not-pending"
}

The CircleCI configuration is stored in the .circleci/config.yml file. I mostly use Maven as a build tool. Here’s a typical CircleCI configuration file for my repositories. It defines two jobs: a standard maven/test job for building the project and running unit tests and a job for running SonarQube analysis.

version: 2.1

jobs:
  analyze:
    docker:
      - image: 'cimg/openjdk:17.0'
    steps:
      - checkout
      - run:
          name: Analyze on SonarCloud
          command: mvn verify sonar:sonar

executors:
  j17:
    docker:
      - image: 'cimg/openjdk:17.0'

orbs:
  maven: circleci/maven@1.4.0

workflows:
  maven_test:
    jobs:
      - maven/test:
          executor: j17
      - analyze:
          context: SonarCloud

By default, CicrcleCI runs builds on Docker containers. However, this approach is not suitable everywhere. For Testcontainers we need a machine executor that has full access to the Docker process. Thanks to that, it is able to run additional containers during tests with e.g. databases.

version: 2.1

jobs:
  analyze:
    docker:
      - image: 'cimg/openjdk:11.0'
    steps:
      - checkout
      - run:
          name: Analyze on SonarCloud
          command: mvn verify sonar:sonar -DskipTests

orbs:
  maven: circleci/maven@1.3.0

executors:
  machine_executor_amd64:
    machine:
      image: ubuntu-2204:2022.04.2
    environment:
      architecture: "amd64"
      platform: "linux/amd64"

workflows:
  maven_test:
    jobs:
      - maven/test:
          executor: machine_executor_amd64
      - analyze:
          context: SonarCloud

Finally, the last part of configuration – an integration between CircleCI and SonarCloud. We need to add some properties to Maven pom.xml to enable SonarCloud context.

<properties>
  <sonar.projectKey>piomin_sample-spring-redis</sonar.projectKey>
  <sonar.organization>piomin</sonar.organization>
  <sonar.host.url>https://sonarcloud.io</sonar.host.url>
</properties>

How It Works

Let’s verify how it works. Once you provide the required configuration for Renovate, CircleCI, and SonarCloud in your GitHub repository the process starts. Renovate initially detects a list of required dependency updates. Since I enabled the dependency dashboard, Renovate immediately creates an issue with a list of changes as shown below. It just provides a summary view showing a list of changes in the dependencies.

Here’s a list of detected package managers in this repository. Besides Maven and CircleCI, there is also Dockerfile and Gitlab CI configuration file there.

Some pull requests has already been automerged by Renovate, if the build on CircleCI has finished successfully.

Some other pull requests are still waiting in the Open state – waiting for approval (a major update from Java 11 to Java 17) or for a fix because the build on CicrcleCI failed.

We can go into the details of the selected PR. Let’s do that for the first PR (#11) on the list visible above. Renovate is trying to update Spring Boot from 2.6.1 to the latest 2.7.7. It created the branch renovate/spring-boot that contains the required changes.

The PR could be merged automatically. However the build failed, so it didn’t happen.

We can go to the details of the build. As you see in the CircleCI dashboard all the tests failed. In this particular case, I have already tried to fix the by updating the version of embedded Mongo. However, it didn’t solve the problem.

Here’s a list of commits in the master branch. As you see Renovate is automatically updating the repository after the build of the particular branch finishes successfully.

As you see, each time a new branch is created CircleCI runs a build to verify if it does not break the tests.

Conclusion

I have some conclusions after making the described changes in my repository:

If you have something like that in my repository main site, it means that I already have reviewed the project and added all the described mechanisms in that article.

The post Manage Multiple GitHub Repositories with Renovate and CircleCI appeared first on Piotr's TechBlog.

In the next part of this article, we will build a sample CI/CD process for a Java application. Some steps of that process will be managed by Tekton, and some others by ArgoCD. Let’s take a look at the diagram below. In the first step, we are going to clone the Git repository with the application source code. Then we will run JUnit tests. After that, we will trigger a source code analysis with SonarQube. Finally, we will build the application image. All these steps are a part of the continuous integration process. Argo CD is responsible for the deployment phase. Also, in case of any changes in the configuration is synchronizes the state of the application on Kubernetes with the Git repository.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. This time there is a second repository – dedicated to storing a configuration independently from application source code. You can clone the following repository and go to the cicd/apps/sample-spring-kotlin directory. After that, you should just follow my instructions. Let’s begin.

Prerequisites

Before we begin, we need to install Tekton and ArgoCD on Kubernetes. We can do this in several different ways. The simplest one is by using OpenShift operators. Their names may be a bit confusing. But in fact, Red Hat OpenShift Pipeline installs Tekton, while Red Hat OpenShift GitOps installs ArgoCD.

Build CI Pipeline with Tekton

The idea behind Tekton is very simple and typical for the CI/CD approach. We are building pipelines. Pipelines consist of several independent steps – tasks. In order to run a pipeline, we should create the PipelineRun object. It manages the PipelineResources passed to tasks as inputs and outputs. Tekton executes each task in its own Kubernetes pod. For more details, you may visit the Tekton documentation site.

Here’s the definition of our pipeline. Before adding tasks we define workspaces at the pipeline global level. We need a workspace for keeping application source code during the build and also a place to store Sonarqube settings. Such workspaces are required by particular tasks.

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: sample-java-pipeline
spec:
  tasks:
    ...
  workspaces:
    - name: source-dir
    - name: sonar-settings

For typical operations like git clone or Maven build, we may use predefined tasks. We can find them on Tekton Hub available here. If you are testing Tekton on OpenShift, some of them are already available as ClusterTask there just after the installation with an operator.

Task 1: Clone Git repository

Here’s the first step of our pipeline. It is referencing to the git-clone ClusterTask. We need to pass the address of the GitHub repository and the name of the branch. Also, we have to assign the workspace to the task using the output name, which is required by that task.

- name: git-clone
  params:
    - name: url
      value: 'https://github.com/piomin/sample-spring-kotlin-microservice.git'
    - name: revision
      value: master
  taskRef:
    kind: ClusterTask
    name: git-clone
  workspaces:
    - name: output
      workspace: source-dir

Task 2: Run JUnit tests with Maven

In the next step, we are running JUnit tests. This time we also use a predefined ClusterTask called maven. In order to run JUnit tests, we should set the GOAL parameter to test. This task requires two workspaces: a workspace with a source code and a second workspace with Maven settings. Because we do not override any Maven configuration I’m just passing there the workspace with source code.

- name: junit-tests
  params:
    - name: GOALS
      value:
        - test
  runAfter:
    - git-clone
  taskRef:
    kind: ClusterTask
    name: maven
  workspaces:
    - name: source
      workspace: source-dir
    - name: maven-settings
      workspace: source-dir

Task 3: Execute Sonarqube scanning

The two next steps will be a little bit more complicated. In order to run Sonarqube scanning, we first need to import the sonarqube-scanner task from Tekton Hub.

$ kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/sonarqube-scanner/0.1/sonarqube-scanner.yaml

After that, we may refer to the already imported task. We should set the address of our Sonarqube instance in the SONAR_HOST_URL parameter, and the unique name of the project in the SONAR_PROJECT_KEY parameter. The task takes two input workspaces. The first of them contains source code, while the second may contain properties file to override some Sonarqube settings. Since it is not possible to pass Sonarqube organization name in task parameters, we will have to do that using the sonar-project.properties file.

- name: sonarqube
  params:
    - name: SONAR_HOST_URL
      value: 'https://sonarcloud.io'
    - name: SONAR_PROJECT_KEY
      value: sample-spring-boot-kotlin
  runAfter:
    - junit-tests
  taskRef:
    kind: Task
    name: sonarqube-scanner
  workspaces:
    - name: source-dir
      workspace: source-dir
    - name: sonar-settings
      workspace: sonar-settings

Task 4: Get the version of the application from pom.xml

In the next step, we are going to retrieve the version number of our application. We will use the version property available inside the Maven pom.xml file. In order to read the value of the version property, we will execute the evaluate command provided by the Maven Helper Plugin. Then we are going to emit that version as a task result. Since there is not such predefined task available on Tekton Hub, we will create our own custom task.

The definition of our custom task is visible below. After executing the mvn help:evaluate command we set that value as a task result with the name version.

apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: maven-get-project-version
spec:
  workspaces:
    - name: source
  params:
    - name: MAVEN_IMAGE
      type: string
      description: Maven base image
      default: gcr.io/cloud-builders/mvn@sha256:57523fc43394d6d9d2414ee8d1c85ed7a13460cbb268c3cd16d28cfb3859e641
    - name: CONTEXT_DIR
      type: string
      description: >-
        The context directory within the repository for sources on
        which we want to execute maven goals.
      default: "."
  results:
    - description: Project version read from pom.xml
      name: version
  steps:
    - name: mvn-command
      image: $(params.MAVEN_IMAGE)
      workingDir: $(workspaces.source.path)/$(params.CONTEXT_DIR)
      script: |
        #!/usr/bin/env bash
        VERSION=$(/usr/bin/mvn help:evaluate -Dexpression=project.version -q -DforceStdout)
        echo -n $VERSION | tee $(results.version.path)

Then let’s just create a task on Kubernetes. The YAML manifest is available in the GitHub repository under the cicd/pipelines/ directory.

$ kubectl apply -f cicd/pipelines/tekton-maven-version.yaml
$ kubectl get task
NAME                        AGE
maven-get-project-version   17h
sonarqube-scanner           2d19h

Finally, we just need to refer to the already created task and set a workspace with the application source code.

- name: get-version
  runAfter:
    - sonarqube
  taskRef:
    kind: Task
    name: maven-get-project-version
  workspaces:
    - name: source
      workspace: source-dir

Task 5: Build and push image

Finally, we may proceed to the last step of our pipeline. We will build the application image and push it to the registry. The output image will be tagged using the Maven version number. That’s why our task needs to refer to the result emitted by the previous task using the following notation: tasks.get-version.results.version. Then, this property is passed as an input parameter to the jib-maven task responsible for building our image in Dockerless mode.

- name: build-and-push-image
  params:
    - name: IMAGE
      value: >-
        image-registry.openshift-image-registry.svc:5000/piotr-cicd/sample-spring-kotlin:$(tasks.get-version.results.version)
  runAfter:
    - get-version
  taskRef:
    kind: ClusterTask
    name: jib-maven
  workspaces:
    - name: source
      workspace: source-dir

Run Tekton pipeline

Before we run the pipeline we need to create two resources to use for workspaces. The application source code will be saved on a persistence volume. That’s why we should create PVC.

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: tekton-workspace
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

The Sonar properties file may be passed as ConfigMap to the pipeline. Because I’m going to perform source code analysis on the cloud instance, I also need to set an organization name.

kind: ConfigMap
apiVersion: v1
metadata:
  name: sonar-properties
data:
  sonar-project.properties: sonar.organization=piomin

Finally, we can start our pipeline as shown below. Of course, we could just create a PipelineRun object.

Now, our pipeline is running. Let’s take a look at the logs of the junit-tests task. As you see there were three JUnit tests executed, and all of them finished successfully.

Then we can go to the SonarCloud site and see the source code analysis report.

Also, let’s verify a list of available images. The tag of our application image is the same as the version set in Maven pom.xml.

$ oc get is
NAME                   IMAGE REPOSITORY                                                                   TAGS    UPDATED
sample-spring-kotlin   image-registry.openshift-image-registry.svc:5000/piotr-cicd/sample-spring-kotlin   1.3.0   7 minutes ago

Trigger pipeline on GitHub push

In the previous section, we started a pipeline on-demand. What about running it after pushing changes in source code to the GitHub repository? Fortunately, Tekton provides a built-in mechanism for that. We need to define Trigger and EventListener. Firstly, we should create the TriggerTemplate object. It is defining the PipelineRun object, which references to our sample-java-pipeline.

apiVersion: triggers.tekton.dev/v1alpha1
kind: TriggerTemplate
metadata:
  name: sample-github-template
spec:
  params:
    - default: main
      description: The git revision
      name: git-revision
    - description: The git repository url
      name: git-repo-url
  resourcetemplates:
    - apiVersion: tekton.dev/v1beta1
      kind: PipelineRun
      metadata:
        generateName: sample-java-pipeline-run-
      spec:
        pipelineRef:
          name: sample-java-pipeline
        serviceAccountName: pipeline
        workspaces:
          - name: source-dir
            persistentVolumeClaim:
              claimName: tekton-workspace
          - configMap:
              name: sonar-properties
            name: sonar-settings

The ClusterTriggerBinding is already available. There is a dedicated definition for GitHub push defined on OpenShift. Thanks to that our EventListener may just refer to that binding and already created TriggerTemplate.

apiVersion: triggers.tekton.dev/v1alpha1
kind: EventListener
metadata:
  name: sample-github-listener
spec:
  serviceAccountName: pipeline
  triggers:
    - bindings:
        - kind: ClusterTriggerBinding
          ref: github-push
      name: trigger-1
      template:
        ref: sample-github-template

After creating EventListener Tekton automatically creates Kubernetes Service that allows triggering push events.

$ oc get svc
NAME                                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
el-sample-github-listener             ClusterIP   172.30.88.73   <none>        8080/TCP   2d1h

On OpenShift, we can easily expose the service outside the cluster as the Route object.

$ oc expose svc el-sample-github-listener
$ oc get route
NAME         HOST/PORT                                                             PATH   SERVICES       
                PORT            TERMINATION   WILDCARD
el-example   el-sample-github-listener-piotr-cicd.apps.qyt1tahi.eastus.aroapp.io          el-sample-github-listener   http-listener                 None

After exposing the service we can go to our GitHub repository and define a webhook. In your repository go to Settings -> Webhooks -> Add webhook. Then paste the address of your Route, choose application/json as a Content type and select push event to send.

Now, you just need to push any change to your GitHub repository.

Continuous Delivery with ArgoCD

The application Kubernetes deployment.yaml manifest is available in the GitHub repository under the cicd/apps/sample-spring-kotlin directory. It is very simple. It contains only Deployment and Service definitions. The version of the Deployment manifest refers to the 1.3.0 version of the image.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: sample-spring-kotlin
spec:
  replicas: 1
  selector:
    matchLabels:
      app: sample-spring-kotlin
  template:
    metadata:
      labels:
        app: sample-spring-kotlin
    spec:
      containers:
      - name: sample-spring-kotlin
        image: image-registry.openshift-image-registry.svc:5000/piotr-cicd/sample-spring-kotlin:1.3.0
        ports:
        - containerPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: sample-spring-kotlin
spec:
  type: ClusterIP
  selector:
    app: sample-spring-kotlin
  ports:
  - port: 8080

Now, we can switch to Argo CD. We can create a new application there using UI or YAML manifest. We will use default settings, so the only thing we need to set is the address of the GitHub repository, a path to the Kubernetes manifest, and a target namespace on the cluster. It means that synchronization between the GitHub repository and Kubernetes needs to be triggered manually.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: sample-spring-kotlin
spec:
  destination:
    name: ''
    namespace: piotr-cicd
    server: 'https://kubernetes.default.svc'
  source:
    path: cicd/apps/sample-spring-kotlin
    repoURL: 'https://github.com/piomin/openshift-quickstart.git'
    targetRevision: HEAD
  project: default

Let’s take a look at the Argo CD UI console. Here’s the state initial synchronization. ArgoCD has already deployed the version 1.3.0 of our application built by the Tekton pipeline.

Now, let’s release a new version of the application. We change the version in Maven pom.xml from 1.3.0 to 1.4.0. This change, commit id and message are visible in the picture below. Push to GitHub repository triggers run of the sample-java-pipeline pipeline by calling webhook.

After a successful run, our pipeline builds and pushes a new version of the application image to the registry. The currently tagged version of the image is 1.4.0 as shown below.

$ oc get is
NAME                   IMAGE REPOSITORY                                                                   TAGS                 UPDATED
sample-spring-kotlin   image-registry.openshift-image-registry.svc:5000/piotr-cicd/sample-spring-kotlin   1.4.0,1.3.0  17 seconds ago

After that, we may switch to the configuration repository. We are going to change the version of the target image. We will also increase the number of running pods as shown below.

Argo CD automatically detects changes in the GitHub repository and sets the status to OutOfSync. It highlights the objects that have been changed by the last commit.

Now, the only thing we need to do is to click the SYNC button. After that Argo CD creates a new revision with the latest image and runs 2 application pods instead of a single one.

Final Thoughts

Tekton and ArgoCD may be used together to successfully design and run CI/CD processes on Kubernetes. Argo CD watches cluster objects stored in a Git repository and manages the create, update, and delete processes for objects within the repository. Tekton is a CI/CD tool that handles all parts of the development lifecycle, from building images to deploying cluster objects. You can easily run and manage them on OpenShift. If you want to compare a currently described approach based on Tekton and ArgoCD with Jenkins you may read my article Continuous Integration with Jenkins on Kubernetes.

The post Kubernetes CI/CD with Tekton and ArgoCD appeared first on Piotr's TechBlog.

$ docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

You are signed into the web dashboard but there are no projects created yet. To perform source code scanning you should just run one command mvn sonar:sonar. Don’t forget to add the SonarQube server address in settings.xml file as shown below.

<profile>
  <id>sonar</id>
  <activation>
    <activeByDefault>true</activeByDefault>
  </activation>
  <properties>
    <sonar.host.url>http://localhost:9000</sonar.host.url>
  </properties>
</profile>

When a SonarQube scanning finishes you will see a new project with the same name as the Maven artifact name with your code metrics and statistics. I created a sample Spring Boot application where I tried to perform some most popular mistakes which impact code quality. Source code is available on GitHub. The right module for analyse is named person-service. However, the code with many bugs and vulnerabilities is pushed to v0.1 branch. The master branch has the latest version with the corrections performed basing on SonarQube scanning. Let’s start an analyze with mvn command. We can be surprised a little – the code analyse result for the 0.1 version is rather not satisfying. Although I spend much time on making important mistakes SonarQube reported only some bugs and code smells were detected and quality gate status is ‘Passed’.

Let’s take a closer look at quality gates in SonarQube. Like I mentioned before we would not like to break the build by not fulfilling one or group of not very important rules. We can achieve it by creating a quality gate. This is a set of requirements that tells us whether or not going to deploy with a new version of a project. There is a default quality gate for Java but we can change its thresholds or create the new one. The default quality gate has thresholds set only for new code, so I decided to create the one for my sample application minimum test coverage set at 50 percent, unit test success detection, and ratings basic on full code. Now, the scanning result looks a little different

To enable scanning test coverage in SonarQube we should add jacoco plugin to maven pom.xml. During maven build mvn clean test -Dmaven.test.failure.ignore=true sonar:sonar the report would be automatically generated and uploaded to SonarQube.

<plugin>
  <groupId>org.jacoco</groupId>
  <artifactId>jacoco-maven-plugin</artifactId>
  <version>0.7.9</version>
  <executions>
    <execution>
      <id>default-prepare-agent</id>
      <goals>
        <goal>prepare-agent</goal>
      </goals>
      </execution>
      <execution>
      <id>default-report</id>
      <phase>prepare-package</phase>
      <goals>
        <goal>report</goal>
      </goals>
    </execution>
  </executions>
</plugin>

The last change that has to be done before the application rescans is to installing some plugins and enabling rules disabled by default. The list of all active and inactive rules can be displayed in Quality Profiles section. In the default profile for Java, there are more than 400 rules available and 271 active. I suggest you install FindBugs and Checkstyle plugins. Those plugins have many additional rules for Java which can be activated for our profile. Now there are about 1.1k inactive rules in many categories. Which of them should be activated depends on you, you can activate them in the default profile, create your new profile or use one of the predefined profiles, which was automatically created by plugins we installed before. In my opinion, the best way to select the right rules is to create a simple project and check which rules are suitable for you. Then you can check out the detailed description and disable the rule if needed. After activating some rules provided by Checkstyle plugin I have a report with 5 bugs and 77 code smells. The most important errors are visible in the pictures below.

All issues reported by SonarQube can be easily reviewed using the UI dashboard for each project in the Issue tab. We can also install plugin SonarLint which integrates with most popular IDEs like Eclipse or IntelliJ and all those issues will be displayed there. Now, we can proceed to fix errors. All changes which I performed to resolve issues can be display on GitHub repository from branches v0.1 to v0.6. I resolved all problems except some checked exception warnings which I set to Resolved (Won’t fix). Those issues won’t be reported after the next scans.

Finally, my project looks as you could see in the picture below. All ratings have a score ‘A’, test coverage is greater than 60% and the quality gate is ‘Passed’. The final person-service version is committed to the master branch.

Like you see there are many rules which can be applied to your project during SonarQube scanning, but sometimes it would be not enough for your organization’s needs. In that case, you may search for some additional plugins or create your own plugin with the rules that meet your specific requirements. In my sample available on GitHub, there is a module sonar-rules where I defined the rule checking whether all public classes have Javadoc comments including @author field. To create SonarQube plugin add the following fragment to your pom.xml and change packaging type to sonar-plugin.

<plugin>
  <groupId>org.sonarsource.sonar-packaging-maven-plugin</groupId>
  <artifactId>sonar-packaging-maven-plugin</artifactId>
  <version>1.17</version>
  <extensions>true</extensions>
  <configuration>
    <pluginKey>piotjavacustom</pluginKey>
    <pluginName>PiotrCustomRules</pluginName>
    <pluginDescription>For test purposes</pluginDescription>
    <pluginClass>pl.piomin.sonar.plugin.CustomRulesPlugin</pluginClass>
    <sonarLintSupported>true</sonarLintSupported>
    <sonarQubeMinVersion>6.0</sonarQubeMinVersion>
  </configuration>
</plugin>

Here’s the class with custom rule definition. First we have to get a scanned class node (Kind.CLASS), a then process first comment (Kind.TRIVIA) in the class file. The rule parameters like name or priority are set inside @Role annotation.

@Rule(key = "CustomAuthorCommentCheck",
name = "Javadoc comment should have @author name",
description = "Javadoc comment should have @author name",
priority = Priority.MAJOR,
tags = {"style"})
public class CustomAuthorCommentCheck extends IssuableSubscriptionVisitor {

  private static final String MSG_NO_COMMENT = "There is no comment under class";
  private static final String MSG_NO_AUTHOR = "There is no author inside comment";

  private Tree actualTree = null;

  @Override
  public List<Kind> nodesToVisit() {
    return ImmutableList.of(Kind.TRIVIA, Kind.CLASS);
  }

  @Override
  public void visitTrivia(SyntaxTrivia syntaxTrivia) {
    String comment = syntaxTrivia.comment();
    if (syntaxTrivia.column() != 0)
      return;
    if (comment == null) {
      reportIssue(actualTree, MSG_NO_COMMENT);
      return;
    }
    if (!comment.contains("@author")) {
      reportIssue(actualTree, MSG_NO_AUTHOR);
      return;
    }
  }

  @Override
  public void visitNode(Tree tree) {
    if (tree.is(Kind.CLASS)) {
      actualTree = tree;
    }
  }

}

Before building and deploying plugin into SonarQube server it can be easily tested using junit. Inside the src/test/file directory we should place test data – java files which are scanned during junit test. For failure test we should also create file CustomAuthorCommentCheck_java.json in the /org/sonar/l10n/java/rules/squid/ directory with rule definition.

@Test
public void testOk() {
   JavaCheckVerifier.verifyNoIssue("src/test/files/CustomAuthorCommentCheck.java", new CustomAuthorCommentCheck());
}

@Test
public void testFail() {
   JavaCheckVerifier.verify("src/test/files/CustomAuthorCommentCheckFail.java", new CustomAuthorCommentCheck());
}

Finally, build maven project and copy generated JAR artifact from target directory to SonarQube docker container into $SONAR_HOME/extensions/plugins directory. Then restart your docker container.

$ docker cp target/sonar-plugins-1.0-SNAPSHOT sonarqube:/opt/sonarqube/extensions/plugins

After SonarQube restart your plugin’s rules are visible under Rules section.

The last thing to do is to run SonarQube scanning in the Continuous Integration process. SonarQube can be easily integrated with the most popular CI server – Jenkins. Here’s the fragment of the Jenkins pipeline where we perform source code scanning and then waiting for the quality gate result. If you interested in more details about Jenkins pipelines, Continuous integration and delivery read my previous post How to setup Continuous Delivery environment.

stage('SonarQube analysis') {
  withSonarQubeEnv('My SonarQube Server') {
    sh 'mvn clean package sonar:sonar'
  }
}
stage("Quality Gate") {
  timeout(time: 1, unit: 'HOURS') {
    def qg = waitForQualityGate()
    if (qg.status != 'OK') {
      error "Pipeline aborted due to quality gate failure: ${qg.status}"
    }
  }
}

The post Code Quality with SonarQube appeared first on Piotr's TechBlog.

• Jenkins – most popular open source automation server
• GitLab – web-based Git repository manager
• Artifactory – open source Maven repository manager
• Ansible – simple open source automation engine
• SonarQube – open source platform for continuous code quality

Here’s picture showing our continuous delivery environment.

The changes pushed to Git repository managed by GitLab server are automatically propagated to Jenkins using webhook. We enable push and merge request triggers. SSL verification will be disabled. In the URL field we have to put jenkins pipeline address with authentication credentials (user and password) and secret token. This API token which is visible in jenkins user profile under Configure tab.

Here’s jenkins pipeline configuration in ‘Build triggers’ section. We have to enable option ‘Build when a change is pushed to GitLab‘. GitLab CI Service URL is the address we have already set in GitLab webhook configuration. There are push and merge request enabled from all branches. It can also be added additional restriction for branch filtering: by name or by regex. To support such kind of trigger in jenkins you need have Gitlab plugin installed.

There are two options of events which trigger jenkins build:

• push – change in source code is pushed to git repository
• merge request –  change in source code is pushed to one branch and then committer creates merge request to the build branch from GitLab management console

In case you would like to use first option you have to disable build branch protection to enable direct push to that branch. In case of using merge request branch protection need to be activated.

Merge request from GitLab console is very intuitive. Under section ‘Merge request’ we are selecting source and target branch and confirm action.

Ok, many words about GitLab and Jenkins integration… Now you know how to configure it. You only have to decide if you prefer push or merge request in your continuous delivery configuration. Merge request is used for code review in Gitlab – so it is useful additional step in your continuous pipeline. Let’s move on. We have to install some other plugins in jenkins to integrate it with Artifactory, SonarQube and Ansible. Here’s the full list of jenkins plugins I used for continuous delivery process inside my organization:

• Ansible plugin
• Artifactory plugin
• Git Plugin
• GitLab Plugin
• SonarQube plugin
• Pipeline

Here’s configuration on my jenkins pipeline for sample maven project.

[code]
node {

withEnv(["PATH+MAVEN=${tool ‘Maven3’}bin"]) {

stage(‘Checkout’) {
def branch = env.gitlabBranch
env.branch = branch
git url: ‘http://172.16.42.157/minkowp/start.git’, credentialsId: ‘5693747c-2f45-4557-ada2-a1da9bbfe0af’, branch: branch
}

stage(‘Test’) {
def pom = readMavenPom file: ‘pom.xml’
print "Build: " + pom.version
env.POM_VERSION = pom.version
sh ‘mvn clean test -Dmaven.test.failure.ignore=true’
junit ‘**/target/surefire-reports/TEST-*.xml’
currentBuild.description = "v${pom.version} (${env.branch})"
}

stage(‘QA’) {
withSonarQubeEnv(‘sonar’) {
sh ‘mvn org.sonarsource.scanner.maven:sonar-maven-plugin:3.2:sonar’
}
}

stage(‘Build’) {
def server = Artifactory.server "server1"
def buildInfo = Artifactory.newBuildInfo()
def rtMaven = Artifactory.newMavenBuild()
rtMaven.tool = ‘Maven3′
rtMaven.deployer releaseRepo:’libs-release-local’, snapshotRepo:’libs-snapshot-local’, server: server
rtMaven.resolver releaseRepo:’remote-repos’, snapshotRepo:’remote-repos’, server: server
rtMaven.run pom: ‘pom.xml’, goals: ‘clean install -Dmaven.test.skip=true’, buildInfo: buildInfo
publishBuildInfo server: server, buildInfo: buildInfo
}

stage(‘Deploy’) {
dir(‘ansible’) {
ansiblePlaybook playbook: ‘preprod.yml’
}
mail from: ‘ci@example.com’, to: ‘piotr.minkowski@play.pl’, subject: "Nowa wersja start: ‘${env.POM_VERSION}’", body: "Wdrożono nowa wersję start ‘${env.POM_VERSION}’ na środowisku preprodukcyjnym."
}

}
}
[/code]

There are five stages in my pipeline:

1. Checkout – source code checkout from git branch. Branch name is sent as parameter by GitLab webhook
2. Test – running JUnit test and creating test report visible in jenkins and changing job description
3. QA – running source code scanning using SonarQube scanner
4. Build – build package resolving artifacts from Artifactory and publishing new application release to Artifactory
5. Deploy – deploying application package and configuration on server using ansible

Following Ansible website it is a simple automation language that can perfectly describe an IT application infrastructure. It’s easy-to-learn, self-documenting, and doesn’t require a grad-level computer science degree to read. Ansible using SSH keys to authenticate on the remote host. So you have to put your SSH key to authorized_keys file in the remote host before running ansible commands on it. The main idea of that that is to create playbook with set of ansible commands. Playbooks are Ansible’s configuration, deployment, and orchestration language. They can describe a policy you want your remote systems to enforce, or a set of steps in a general IT process. Here is catalog structure with ansible configuration for application deploy.

Here’s my ansible playbook code. It defines remote host, user to connect and role name. This file is used inside jenkins pipeline on ansiblePlaybook step.

[code]
—
– hosts: pBPreprod
remote_user: default

roles:
– preprod
[/code]

Here’s main.yml file where we define set of ansible commands to on remote server.

[code]
—
– block:
– name: Copy configuration file
template: src=config.yml.j2 dest=/opt/start/config.yml

– name: Copy jar file
copy: src=../target/start.jar dest=/opt/start/start.jar

– name: Run jar file
shell: java -jar /opt/start/start.jar
[/code]

You can check out build results on jenkins console. There is also fine pipeline visualization with stage execution time. Each build history record has link to Artifactory build information and SonarQube scanner report.

The post How to setup Continuous Delivery environment appeared first on Piotr's TechBlog.