If you are interested in Kafka and Spring Boot, you may find several articles on my blog about it. To read about concurrency with Kafka and Spring Boot read the following post. For example, there is also an interesting article about Kafka transactions here.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should go to the kafka directory. After that, you should just follow my instructions. Let’s begin.

Dependencies

Let’s take a look at the list of required Maven dependencies. It is the same for both of our sample Spring Boot apps. Of course, we need to add the Spring Boot starter and the Spring Kafka for sending or receiving messages. In order to automatically generate traces related to each message, we are including the Spring Boot Actuator and the Micrometer Tracing Open Telemetry bridge. Finally, we need to include the opentelemetry-exporter-otlp library to export traces outside the app.

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.kafka</groupId>
    <artifactId>spring-kafka</artifactId>
  </dependency>
  <dependency>
    <groupId>com.fasterxml.jackson.core</groupId>
    <artifactId>jackson-databind</artifactId>
  </dependency>
  <dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
  </dependency>
  <dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
  </dependency>
</dependencies>

Spring Boot Kafka Tracing for Producer

Our apps don’t do anything complicated. They are just sending and receiving messages. Here’s the class representing the message exchanged between both apps.

public class Info {

    private Long id;
    private String source;
    private String space;
    private String cluster;
    private String message;

    public Info(Long id, String source, String space, String cluster, 
                String message) {
       this.id = id;
       this.source = source;
       this.space = space;
       this.cluster = cluster;
       this.message = message;
    }

   // GETTERS AND SETTERS
}

Let’s begin with the producer app. It generates and sends one message per second. Here’s the implementation of a @Service bean responsible for producing messages. It injects and uses the KafkaTemplate bean for that.

@Service
public class SenderService {

   private static final Logger LOG = LoggerFactory
      .getLogger(SenderService.class);

   AtomicLong id = new AtomicLong();
   @Autowired
   KafkaTemplate<Long, Info> template;

   @Value("${POD:kafka-producer}")
   private String pod;
   @Value("${NAMESPACE:empty}")
   private String namespace;
   @Value("${CLUSTER:localhost}")
   private String cluster;
   @Value("${TOPIC:info}")
   private String topic;

   @Scheduled(fixedRate = 1000)
   public void send() {
      Info info = new Info(id.incrementAndGet(), 
                           pod, 
                           namespace, 
                           cluster, 
                           "HELLO");
      CompletableFuture<SendResult<Long, Info>> result = template
         .send(topic, info.getId(), info);
      result.whenComplete((sr, ex) ->
                LOG.info("Sent({}): {}", sr.getProducerRecord().key(), 
                         sr.getProducerRecord().value()));
    }

}

Spring Boot provides an auto-configured instance of KafkaTemplate. However, to enable Kafka tracing with Spring Boot we need to customize that instance. Here’s the implementation of the KafkaTemplate bean inside the producer app’s main class. In order to enable tracing, we need to invoke the setObservationEnabled method. By default, the Micrometer module generates some generic tags. We want to add at least the name of the target topic and the Kafka message key. Therefore we are creating our custom implementation of the KafkaTemplateObservationConvention interface. It uses the KafkaRecordSenderContext to retrieve the topic name and the message key from the ProducerRecord object.

@SpringBootApplication
@EnableScheduling
public class KafkaProducer {

   private static final Logger LOG = LoggerFactory
      .getLogger(KafkaProducer.class);

   public static void main(String[] args) {
      SpringApplication.run(KafkaProducer.class, args);
   }

   @Bean
   public NewTopic infoTopic() {
      return TopicBuilder.name("info")
             .partitions(1)
             .replicas(1)
             .build();
   }

   @Bean
   public KafkaTemplate<Long, Info> kafkaTemplate(ProducerFactory<Long, Info> producerFactory) {
      KafkaTemplate<Long, Info> t = new KafkaTemplate<>(producerFactory);
      t.setObservationEnabled(true);
      t.setObservationConvention(new KafkaTemplateObservationConvention() {
         @Override
         public KeyValues getLowCardinalityKeyValues(KafkaRecordSenderContext context) {
            return KeyValues.of("topic", context.getDestination(),
                    "id", String.valueOf(context.getRecord().key()));
         }
      });
      return t;
   }

}

We also need to set the address of the Jaeger instance and decide which percentage of spans will be exported. Here’s the application.yml file with the required properties:

spring:
  application.name: kafka-producer
  kafka:
    bootstrap-servers: ${KAFKA_URL:localhost}:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.LongSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer

management:
  tracing:
    enabled: true
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://jaeger:4318/v1/traces

Spring Boot Kafka Tracing for Consumer

Let’s switch to the consumer app. It just receives and prints messages coming to the Kafka topic. Here’s the implementation of the listener @Service. Besides the whole message content, it also prints the message key and a topic partition number.

@Service
public class ListenerService {

   private static final Logger LOG = LoggerFactory
      .getLogger(ListenerService.class);

   @KafkaListener(id = "info", topics = "${app.in.topic}")
   public void onMessage(@Payload Info info,
                         @Header(name = KafkaHeaders.RECEIVED_KEY, required = false) Long key,
                         @Header(KafkaHeaders.RECEIVED_PARTITION) int partition) {
      LOG.info("Received(key={}, partition={}): {}", key, partition, info);
   }

}

In order to generate and export traces on the consumer side we need to override the ConcurrentKafkaListenerContainerFactory bean. For the container listener factory, we should obtain the ContainerProperties instance and then invoke the setObservationEnabled method. The same as before we can create a custom implementation of the KafkaTemplateObservationConvention interface to include the additional tags (optionally).

@SpringBootApplication
@EnableKafka
public class KafkaConsumer {

   private static final Logger LOG = LoggerFactory
      .getLogger(KafkaConsumer.class);

    public static void main(String[] args) {
        SpringApplication.run(KafkaConsumer.class, args);
    }

    @Value("${app.in.topic}")
    private String topic;

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, String> listenerFactory(ConsumerFactory<String, String> consumerFactory) {
        ConcurrentKafkaListenerContainerFactory<String, String> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.getContainerProperties().setObservationEnabled(true);
        factory.setConsumerFactory(consumerFactory);
        return factory;
    }

    @Bean
    public NewTopic infoTopic() {
        return TopicBuilder.name(topic)
                .partitions(10)
                .replicas(3)
                .build();
    }

}

Of course, we also need to set a Jaeger address in the application.yml file:

spring:
  application.name: kafka-consumer
  kafka:
    bootstrap-servers: ${KAFKA_URL:localhost}:9092
    consumer:
      key-deserializer: org.apache.kafka.common.serialization.LongDeserializer
      value-deserializer: org.springframework.kafka.support.serializer.JsonDeserializer
      properties:
        spring.json.trusted.packages: "*"

app.in.topic: ${TOPIC:info}

management:
  tracing:
    enabled: true
    sampling:
      probability: 1.0
  otlp:
    tracing:
      endpoint: http://jaeger:4318/v1/traces

Trying on Docker

Once we finish the implementation we can try out our solution. We will run both Kafka and Jaeger as Docker containers. Firstly, let’s build the project and container images for the producer and consumer apps. Spring Boot provides built-in tools for that. Therefore, we just need to execute the following command:

$ mvn clean package spring-boot:build-image

After that, we can define the docker-compose.yml file with a list of containers. It is possible to dynamically override Spring Boot properties using a style based on environment variables. Thanks to that, we can easily change the Kafka and Jaeger addresses for the containers. Here’s our docker-compose.yml:

version: "3.8"
services:
  broker:
    image: moeenz/docker-kafka-kraft:latest
    restart: always
    ports:
      - "9092:9092"
    environment:
      - KRAFT_CONTAINER_HOST_NAME=broker
  jaeger:
    image: jaegertracing/all-in-one:latest
    ports:
      - "16686:16686"
      - "4317:4317"
      - "4318:4318"
  producer:
    image: library/producer:1.0-SNAPSHOT
    links:
      - broker
      - jaeger
    environment:
      MANAGEMENT_OTLP_TRACING_ENDPOINT: http://jaeger:4318/v1/traces
      SPRING_KAFKA_BOOTSTRAP_SERVERS: broker:9092
  consumer:
    image: library/consumer:1.0-SNAPSHOT
    links:
      - broker
      - jaeger
    environment:
      MANAGEMENT_OTLP_TRACING_ENDPOINT: http://jaeger:4318/v1/traces
      SPRING_KAFKA_BOOTSTRAP_SERVERS: broker:9092

Let’s run all the defined containers with the following command:

$ docker compose up

Our apps are running and exchanging messages:

The Jaeger dashboard is available under the 16686 port. As you see, there are several traces with the kafka-producer and kafka-consumer spans.

We can go into the details of each entry. The trace generated by the producer app is always correlated to the trace generated by the consumer app for every single message. There are also our two custom tags (id and topic) with values added by the KafkaTemplate bean.

Running on Kubernetes

Our sample apps are prepared for being deployed on Kubernetes. You can easily do it with the Skaffold CLI. Before that, we need to install Kafka and Jaeger on Kubernetes. I will not get into details about Kafka installation. You can find a detailed description of how to run Kafka on Kubernetes with the Strimzi operator in my article available here. After that, we can proceed to the Jaeger installation. In the first step, we need to add the following Helm repository:

$ helm repo add jaegertracing https://jaegertracing.github.io/helm-charts

By default, the Jaeger Helm chart doesn’t expose OTLP endpoints. In order to enable them, we need to override some default settings. Here’s our values YAML manifest:

collector:
  service:
    otlp:
      grpc:
        name: otlp-grpc
        port: 4317
      http:
        name: otlp-http
        port: 4318

Let’s install Jaeger in the jaeger namespace with the parameters from jaeger-values.yaml:

$ helm install jaeger jaegertracing/jaeger -n jaeger \
    --create-namespace \
    -f jaeger-values.yaml

Once we install Jaeger we can verify a list of Kubernetes Services. We will use the jaeger-collector service to send traces for the apps and the jaeger-query service to access the UI dashboard.

$ kubectl get svc -n jaeger
NAME               TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                                           AGE
jaeger-agent       ClusterIP   10.96.147.104   <none>        5775/UDP,6831/UDP,6832/UDP,5778/TCP,14271/TCP     14m
jaeger-cassandra   ClusterIP   None            <none>        7000/TCP,7001/TCP,7199/TCP,9042/TCP,9160/TCP      14m
jaeger-collector   ClusterIP   10.96.111.236   <none>        14250/TCP,14268/TCP,4317/TCP,4318/TCP,14269/TCP   14m
jaeger-query       ClusterIP   10.96.88.64     <none>        80/TCP,16685/TCP,16687/TCP                        14m

Finally, we can run our sample Spring Boot apps that connect to Kafka and Jaeger. Here’s the Deployment object for the producer app. It overrides the default Kafka and Jaeger addresses by defining the KAFKA_URL and MANAGEMENT_OTLP_TRACING_ENDPOINT environment variables.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: producer
spec:
  selector:
    matchLabels:
      app: producer
  template:
    metadata:
      labels:
        app: producer
    spec:
      containers:
      - name: producer
        image: piomin/producer
        resources:
          requests:
            memory: 200Mi
            cpu: 100m
        ports:
        - containerPort: 8080
        env:
          - name: MANAGEMENT_OTLP_TRACING_ENDPOINT
            value: http://jaeger-collector.jaeger:4318/v1/traces
          - name: KAFKA_URL
            value: my-cluster-kafka-bootstrap
          - name: CLUSTER
            value: c1
          - name: TOPIC
            value: test-1
          - name: POD
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

Here’s a similar Deployment object for the consumer app:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: consumer-1
spec:
  selector:
    matchLabels:
      app: consumer-1
  template:
    metadata:
      labels:
        app: consumer-1
    spec:
      containers:
      - name: consumer
        image: piomin/consumer
        resources:
          requests:
            memory: 200Mi
            cpu: 100m
        ports:
        - containerPort: 8080
        env:
          - name: SPRING_APPLICATION_NAME
            value: kafka-consumer-1
          - name: TOPIC
            value: test-1
          - name: KAFKA_URL
            value: my-cluster-kafka-bootstrap
          - name: MANAGEMENT_OTLP_TRACING_ENDPOINT
            value: http://jaeger-collector.jaeger:4318/v1/traces

Assuming that you are inside the kafka directory in the Git repository, you just need to run the following command to deploy both apps. By the way, I’ll create two deployments of the consumer app (consumer-1 and consumer-2) just for Jaeger visualization purposes.

$ skaffold run -n strimzi --tail

Once you run the apps, you can go to the Jaeger dashboard and verify the list of traces. In order to access the dashboard, we can enable port forwarding for the jaeger-query Service.

$ kubectl port-forward svc/jaeger-query 80:80

Final Thoughts

Integration between Spring Kafka and Micrometer Tracing is a relatively new feature available since the 3.0 version. It is possible, that it will be improved soon with some new features. Anyway, currently it gives a simple way to generate and send traces from Kafka producers and consumers.

The post Kafka Tracing with Spring Boot and Open Telemetry appeared first on Piotr's TechBlog.

This time I won’t tell you about Istio basics. Although our configuration is not complicated you may read the following introduction before we start.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should go to the mesh-with-db directory. After that, you should just follow my instructions

Service Mesh Architecture

Let’s start with our microservices architecture. There are two applications: person-app and insurance-app. As you probably guessed, the person-app stores and returns information about insured people. On the other hand, the insurance-app keeps insurances data. Each service has a separate database. We deploy the person-app in two versions. The v2 version contains one additional field externalId.

The following picture illustrates our scenario. Istio splits traffic between two versions of the person-app. By default, it splits the traffic 50% to 50%. If it receives the X-Version header in the request it calls the particular version of the person-app. Of course, the possible values of the header are v1 or v2.

Distributed Tracing with Istio

Istio generates distributed trace spans for each managed service. It means that every request sent inside the Istio will have the following HTTP headers:

So, every single request incoming from the Istio gateway contains X-B3-SpanId, X-B3-TraceId, and some other B3 headers. The X-B3-SpanId indicates the position of the current operation in the trace tree. On the other hand, every span in a trace should share the X-B3-TraceId header. At first glance, you can feel surprised that Istio does not propagate B3 headers in client calls. To clarify, if one service communicates with another service using e.g. REST client, you will see two different traces. The first of them is related to the API endpoint call, while the second with the client call of another API endpoint. That’s not exactly what we would like to achieve, right?

Let’s visualize our problem. If you call the insurance-app through the Istio gateway you will have the first trace in Jaeger. During that call the insurance-app calls endpoint from the person-app using Quarkus REST client. That’s another separate trace in Jeager. Our goal here is to propagate all required B3 headers to the person-app also. You can find a list of required headers here in the Istio documentation.

Of course, that’s not the only thing we will do today. We will also prepare Istio rules in order to simulate latency in our communication. It is a good scenario to use the tracing tool. Also, I’m going to show you how to easily deploy microservice on Kubernetes using Quarkus features for that.

I’m running Istio and Jaeger on OpenShift. More precisely, I’m using OpenShift Service Mesh that is the RedHat’s SM implementation based on Istio. I doesn’t have any impact on the exercise, so you as well repeat all the steps on Kubernetes.

Create Microservices with Quarkus

Let’s begin with the insurance-app. Here’s the class responsible for the REST endpoints implementation. There are several methods there. However, the most important for us is the getInsuranceDetailsById method that calls the GET /persons/{id} endpoint in the person-app. In order to use the Quarkus REST client extension, we need to inject client bean with @RestClient annotation.

@Path("/insurances")
public class InsuranceResource {

   @Inject
   Logger log;
   @Inject
   InsuranceRepository insuranceRepository;
   @Inject @RestClient
   PersonService personService;

   @POST
   @Transactional
   public Insurance addInsurance(Insurance insurance) {
      insuranceRepository.persist(insurance);
      return insurance;
   }

   @GET
   public List<Insurance> getInsurances() {
      return insuranceRepository.listAll();
   }

   @GET
   @Path("/{id}")
   public Insurance getInsuranceById(@PathParam("id") Long id) {
      return insuranceRepository.findById(id);
   }

   @GET
   @Path("/{id}/details")
   public InsuranceDetails getInsuranceDetailsById(@PathParam("id") Long id, @HeaderParam("X-Version") String version) {
      log.infof("getInsuranceDetailsById: id=%d, version=%s", id, version);
      Insurance insurance = insuranceRepository.findById(id);
      InsuranceDetails insuranceDetails = new InsuranceDetails();
      insuranceDetails.setPersonId(insurance.getPersonId());
      insuranceDetails.setAmount(insurance.getAmount());
      insuranceDetails.setType(insurance.getType());
      insuranceDetails.setExpiry(insurance.getExpiry());
      insuranceDetails.setPerson(personService.getPersonById(insurance.getPersonId()));
      return insuranceDetails;
   }

}

As you probably remember from the previous section, we need to propagate several headers responsible for Istio tracing to the downstream Quarkus service. Let’s take a look at the implementation of the REST client in the PersonService interface. In order to send some additional headers to the request, we need to annotate the interface with @RegisterClientHeaders. Then we have two options. We can provide our custom headers factory as shown below. Otherwise, we may use the property org.eclipse.microprofile.rest.client.propagateHeaders with a list of headers.

@Path("/persons")
@RegisterRestClient
@RegisterClientHeaders(RequestHeaderFactory.class)
public interface PersonService {

   @GET
   @Path("/{id}")
   Person getPersonById(@PathParam("id") Long id);
}

Let’s just take a look at the implementation of the REST endpoint in the person-app. The method getPersonId at the bottom is responsible for finding a person by the id field. It is a target method called by the PersonService client.

@Path("/persons")
public class PersonResource {

   @Inject
   Logger log;
   @Inject
   PersonRepository personRepository;

   @POST
   @Transactional
   public Person addPerson(Person person) {
      personRepository.persist(person);
      return person;
   }

   @GET
   public List<Person> getPersons() {
      return personRepository.listAll();
   }

   @GET
   @Path("/{id}")
   public Person getPersonById(@PathParam("id") Long id) {
      log.infof("getPersonById: id=%d", id);
      Person p = personRepository.findById(id);
      log.infof("getPersonById: %s", p);
      return p;
   }
}

Finally, the implementation of our customer client headers factory. It needs to implement the ClientHeadersFactory interface and its update() method. We are not doing anything complicated here. We are forwarding B3 tracing headers and the X-Version used by Istio to route between two versions of the person-app. I also added some logs. There I didn’t use the already mentioned option of header propagation based on the property org.eclipse.microprofile.rest.client.propagateHeaders.

@ApplicationScoped
public class RequestHeaderFactory implements ClientHeadersFactory {

   @Inject
   Logger log;

   @Override
   public MultivaluedMap<String, String> update(MultivaluedMap<String, String> inHeaders,
                                                 MultivaluedMap<String, String> outHeaders) {
      String version = inHeaders.getFirst("x-version");
      log.infof("Version Header: %s", version);
      String traceId = inHeaders.getFirst("x-b3-traceid");
      log.infof("Trace Header: %s", traceId);
      MultivaluedMap<String, String> result = new MultivaluedHashMap<>();
      result.add("X-Version", version);
      result.add("X-B3-TraceId", traceId);
      result.add("X-B3-SpanId", inHeaders.getFirst("x-b3-spanid"));
      result.add("X-B3-ParentSpanId", inHeaders.getFirst("x-b3-parentspanid"));
      return result;
   }
}

Run Quarkus Applications on Kubernetes

Before we test Istio tracing we need to deploy our Quarkus microservices on Kubernetes. We may do it in several different ways. One of the methods is provided directly by Quarkus. Thanks to that we can generate the Deployment manifests automatically during the build. It turns out that we can also apply Istio manifests to the Kubernetes cluster as well. Firstly, we need to include the following two modules.

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-openshift</artifactId>
</dependency>
<dependency>
  <groupId>me.snowdrop</groupId>
  <artifactId>istio-client</artifactId>
  <version>1.7.7.1</version>
</dependency>

If you deploy on Kubernetes just replace the quarkus-openshift module with the quarkus-kubernetes module. In the next step, we need to provide some configuration settings in the application.properties file. In order to enable deployment during the build, we need to set the property quarkus.kubernetes.deploy to true. We can configure several aspects of the Kubernetes Deployment. For example, we may enable the Istio proxy by setting the annotation sidecar.istio.io/inject to true or add some labels required for routing: app and version (in that case). Finally, our application connects to the database, so we need to inject values from the Kubernetes Secret.

quarkus.container-image.group = demo-mesh
quarkus.container-image.build = true
quarkus.kubernetes.deploy = true
quarkus.kubernetes.deployment-target = openshift
quarkus.kubernetes-client.trust-certs = true

quarkus.openshift.deployment-kind = Deployment
quarkus.openshift.labels.app = quarkus-insurance-app
quarkus.openshift.labels.version = v1
quarkus.openshift.annotations."sidecar.istio.io/inject" = true
quarkus.openshift.env.mapping.postgres_user.from-secret = insurance-db
quarkus.openshift.env.mapping.postgres_user.with-key = database-user
quarkus.openshift.env.mapping.postgres_password.from-secret = insurance-db
quarkus.openshift.env.mapping.postgres_password.with-key = database-password
quarkus.openshift.env.mapping.postgres_db.from-secret = insurance-db
quarkus.openshift.env.mapping.postgres_db.with-key = database-name

Here’s a part of the configuration responsible for database connection settings. The name of the key after quarkus.openshift.env.mappings maps to the name of the environment variable, for example, the postgres_password property to the POSTGRES_PASSWORD env.

quarkus.datasource.db-kind = postgresql
quarkus.datasource.username = ${POSTGRES_USER}
quarkus.datasource.password = ${POSTGRES_PASSWORD}
quarkus.datasource.jdbc.url = 
jdbc:postgresql://person-db:5432/${POSTGRES_DB}

If there are any additional manifests to apply we should place them inside the src/main/kubernetes directory. This applies to, for example, the Istio configuration. So, now the only thing we need to do is to application build. Firstly go to the quarkus-person-app directory and run the following command. Then, go to the quarkus-insurance-app directory, and do the same.

$ mvn clean package

Traffic Management with Istio

There are two versions of the person-app application. So, let’s create the DestinationRule object containing two subsets v1 and v2 based on the version label.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: quarkus-person-app-dr
spec:
  host: quarkus-person-app
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2

In the next step, we need to create the VirtualService object for the quarkus-person-app service. The routing between versions can be based on the X-Version header. If that is not set Istio sends 50% to the v1 version, and 50% to the v2 version. Also, we will inject the delay into the v2 route using the Istio HTTPFaultInjection object. It adds 3 seconds delay for 50% of incoming requests.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-person-app-vs
spec:
  hosts:
    - quarkus-person-app
  http:
    - match:
        - headers:
            X-Version:
              exact: v1
      route:
        - destination:
            host: quarkus-person-app
            subset: v1
    - match:
        - headers:
            X-Version:
              exact: v2
      route:
        - destination:
            host: quarkus-person-app
            subset: v2
      fault:
        delay:
          fixedDelay: 3s
          percentage:
            value: 50
    - route:
        - destination:
            host: quarkus-person-app
            subset: v1
          weight: 50
        - destination:
            host: quarkus-person-app
            subset: v2
          weight: 50

Now, let’s create the Istio Gateway object. Replace the CLUSTER_DOMAIN variable with your cluster’s domain name:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: microservices-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - quarkus-insurance-app.apps.$CLUSTER_DOMAIN
        - quarkus-person-app.apps.$CLUSTER_DOMAIN

In order to forward traffic from the gateway, the VirtualService needs to refer to that gateway.

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: quarkus-insurance-app-vs
spec:
  hosts:
    - quarkus-insurance-app.apps.$CLUSTER_DOMAIN
  gateways:
    - microservices-gateway
  http:
    - match:
        - uri:
            prefix: "/insurance"
      rewrite:
        uri: " "
      route:
        - destination:
            host: quarkus-insurance-app
          weight: 100

Now you can call the insurance-app service:

$ curl http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/1/details

We can verify all the existing Istio objects using Kiali.

Testing Istio Tracing with Quarkus

First of all, you can generate many requests using the siege tool. There are multiple ways to run it. We can prepare the file with example requests as shown below:

http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/1
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/2
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/3
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/4
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/5
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/6
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/7
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/8
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons/9
http://quarkus-person-app.apps.${CLUSTER_DOMAIN}/person/persons
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/1/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/2/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/3/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/4/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/5/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/6/details
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/1
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/2
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/3
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/4
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/5
http://quarkus-insurance-app.apps.${CLUSTER_DOMAIN}/insurance/insurances/6

Now we need to set that file as an input for the siege command. We can also set the number of repeats (-r) and concurrent threads (-c).

$ siege -f k8s/traffic/urls.txt -i -v -r 500 -c 10 --no-parser

It takes some time before the command will finish. In the meanwhile, let’s try to send a single request to the insurance-app with the X-Version header set to v2. 50% of such requests are delayed by the Istio quarkus-person-app-vs VirtualService. Repeat the request until you have the response with 3s delay:

Here’s the access log from the Quarkus application:

Then, let’s switch to the Jaeger console. We can find the request by the guid:x-request-id tag.

Here’s the result of our search:

We have a full trace of the request including communication between Istio gateway and insurance-app, and also between the insurance-app and person-app. In order to print the details of the trace just click the record. You will see a trace timeline and requests/responses structure. You can easily verify that latency occurs somewhere between insurance-app and person-app, because the request has been processed only 4.46 ms by the person-app.

The post Distributed Tracing with Istio, Quarkus and Jaeger appeared first on Piotr's TechBlog.

This article is the second in a series of my article about Knative. After publishing the first of them, Spring Boot on Knative, you were asking me about a long application startup time after scaling to zero. That’s why I resolved this Spring Boot issue by compiling it to a native image with GraalVM. The problem with startup time seems to be an important thing in a serverless approach.

On Knative you can run any type of application – not only a function. In this article, when I’m writing “microservices”, in fact, I’m thinking about service to service communication.

Source Code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my GitHub repository. Then you should just follow my instructions

As the example of microservices in this article, I used two applications callme-service and caller-service. Both of them exposes a single endpoint, which prints a name of the application pod. The caller-service application also calls the endpoint exposed by the callme-service application.

On Kubernetes, both these applications will be deployed as Knative services in multiple revisions. We will also distribute traffic across those revisions using Knative routes. The picture visible below illustrates the architecture of our sample system.

1. Prepare Spring Boot microservices

We have two simple Spring Boot applications that expose a single REST endpoint, health checks, and run an in-memory H2 database. We use Hibernate and Lombok. Therefore, we need to include the following list of dependencies in Maven pom.xml.

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
</dependency>
<dependency>
    <groupId>org.projectlombok</groupId>
    <artifactId>lombok</artifactId>
    <version>1.18.16</version>
</dependency>

Each time we call the ping endpoint it creates an event and stores it in the H2 database. The REST endpoint returns the name of a pod and namespace inside Kubernetes and the id of the event. That method will be useful in our manual tests on the cluster.

@RestController
@RequestMapping("/callme")
public class CallmeController {

    @Value("${spring.application.name}")
    private String appName;
    @Value("${POD_NAME}")
    private String podName;
    @Value("${POD_NAMESPACE}")
    private String podNamespace;
    @Autowired
    private CallmeRepository repository;

    @GetMapping("/ping")
    public String ping() {
        Callme c = repository.save(new Callme(new Date(), podName));
        return appName + "(id=" + c.getId() + "): " + podName + " in " + podNamespace;
    }

}

Here’s our model class – Callme. The model class inside the caller-service application is pretty similar.

@Entity
@Getter
@Setter
@NoArgsConstructor
@RequiredArgsConstructor
public class Callme {

    @Id
    @GeneratedValue
    private Integer id;
    @Temporal(TemporalType.TIMESTAMP)
    @NonNull
    private Date addDate;
    @NonNull
    private String podName;

}

Also, let’s take a look at the first version of the ping method in CallerController. We will modify it later when we will discussing communication and tracing. For now, it is important to understand that this method also calls the ping method exposed by callme-service and returns the whole response.

@GetMapping("/ping")
public String ping() {
    Caller c = repository.save(new Caller(new Date(), podName));
    String callme = callme();
    return appName + "(id=" + c.getId() + "): " + podName + " in " + podNamespace
            + " is calling " + callme;
}

2. Prepare Spring Boot native image with GraalVM

Spring Native provides support for compiling Spring applications to native executables using the GraalVM native compiler. For more details about this project, you may refer to its documentation. Here’s the main class of our application.

@SpringBootApplication
public class CallmeApplication {

   public static void main(String[] args) {
      SpringApplication.run(CallmeApplication.class, args);
   }

}

Hibernate does a lot of dynamic things at runtime. So we need to get Hibernate to enhance the entities in our application at build time. We need to add the following Maven plugin to our build.

<plugin>
   <groupId>org.hibernate.orm.tooling</groupId>
   <artifactId>hibernate-enhance-maven-plugin</artifactId>
   <version>${hibernate.version}</version>
   <executions>
      <execution>
         <configuration>
            <failOnError>true</failOnError>
            <enableLazyInitialization>true</enableLazyInitialization>
            <enableDirtyTracking>true</enableDirtyTracking>
            <enableExtendedEnhancement>false</enableExtendedEnhancement>
         </configuration>
         <goals>
            <goal>enhance</goal>
         </goals>
      </execution>
   </executions>
</plugin>

In this article, I’m using the latest version of Spring Native – 0.9.0. Since Spring Native is actively developed, there are significant changes between subsequent versions. If you compare it to some other articles based on the earlier versions, we don’t have to disable proxyBeansMethods, exclude SpringDataWebAutoConfiguration, add spring-context-indexer into dependencies or create hibernate.properties. Cool! I can also use Buildpacks for building a native image.

So, now we just need to add the following dependency.

<dependency>
   <groupId>org.springframework.experimental</groupId>
   <artifactId>spring-native</artifactId>
   <version>0.9.0</version>
</dependency>

The Spring AOT plugin performs ahead-of-time transformations required to improve native image compatibility and footprint.

<plugin>
    <groupId>org.springframework.experimental</groupId>
    <artifactId>spring-aot-maven-plugin</artifactId>
    <version>${spring.native.version}</version>
    <executions>
        <execution>
            <id>test-generate</id>
            <goals>
                <goal>test-generate</goal>
            </goals>
        </execution>
        <execution>
            <id>generate</id>
            <goals>
                <goal>generate</goal>
            </goals>
        </execution>
    </executions>
</plugin>

3. Run native image on Knative with Buildpacks

Using Builpacks for creating a native image is our primary option. Although it requires a Docker daemon it works properly on every OS. However, we need to use the latest stable version of Spring Boot. In that case, it is 2.4.3. You can configure Buildpacks as well inside Maven pom.xml with the spring-boot-maven-plugin. Since we need to build and deploy the application on Kubernetes in one step, I prefer configuration in Skaffold. We use paketobuildpacks/builder:tiny as a builder image. It is also required to enable the native build option with the BP_BOOT_NATIVE_IMAGE environment variable.

apiVersion: skaffold/v2beta11
kind: Config
metadata:
  name: callme-service
build:
  artifacts:
  - image: piomin/callme-service
    buildpacks:
      builder: paketobuildpacks/builder:tiny
      env:
        - BP_BOOT_NATIVE_IMAGE=true
deploy:
  kubectl:
    manifests:
      - k8s/ksvc.yaml

Skaffold configuration refers to our Knative Service manifest. It is quite non-typical since we need to inject a pod and namespace names into the container. We also allow a maximum of 10 concurrent requests per single pod. If it is exceeded Knative scale up a number of running instances.

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: callme-service
spec:
  template:
    spec:
      containerConcurrency: 10
      containers:
      - name: callme
        image: piomin/callme-service
        ports:
          - containerPort: 8080
        env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                fieldPath: metadata.namespace

By default, Knative doesn’t allow to use Kubernetes fieldRef feature. In order to enable it, we need to update the knative-features ConfigMap in the knative-serving namespace. The required property name is kubernetes.podspec-fieldref.

kind: ConfigMap
apiVersion: v1
metadata:
  annotations:
  namespace: knative-serving
  labels:
    serving.knative.dev/release: v0.16.0
data:
  kubernetes.podspec-fieldref: enabled

Finally, we may build and deploy our Spring Boot microservices on Knative with the following command.

$ skaffold run

4. Run native image on Knative with Jib

The same as in my previous article about Knative we will build and run our applications on Kubernetes with Skaffold and Jib. Fortunately, Jib Maven Plugin has already introduced support for GraalVM “native images”. The Jib GraalVM Native Image Extension expects the native-image-maven-plugin to do the heavy lifting of generating a “native image” (with the native-image:native-image goal). Then the extension just simply copies the binary into a container image and sets it as executable.

Of course, unlike Java bytecode, a native image is not portable but platform-specific. The Native Image Maven Plugin doesn’t support cross-compilation, so the native-image should be built on the same OS as the runtime architecture. Since I build a GraalVM image of my applications on Ubuntu 20.10, I should use the same base Docker image for running containerized microservices. In that case, I chose image ubuntu:20.10 as shown below.

<plugin>
   <groupId>com.google.cloud.tools</groupId>
   <artifactId>jib-maven-plugin</artifactId>
   <version>2.8.0</version>
   <dependencies>
      <dependency>
         <groupId>com.google.cloud.tools</groupId>
         <artifactId>jib-native-image-extension-maven</artifactId>
         <version>0.1.0</version>
      </dependency>
   </dependencies>
   <configuration>
      <from>
         <image>ubuntu:20.10</image>
      </from>
      <pluginExtensions>
         <pluginExtension>
            <implementation>com.google.cloud.tools.jib.maven.extension.nativeimage.JibNativeImageExtension</implementation>
         </pluginExtension>
      </pluginExtensions>
   </configuration>
</plugin>

If you use Jib Maven Plugin you first need to build a native image. In order to build a native image of the application we also need to include a native-image-maven-plugin. Of you need to build our application using GraalVM JDK.

<plugin>
   <groupId>org.graalvm.nativeimage</groupId>
   <artifactId>native-image-maven-plugin</artifactId>
   <version>21.0.0.2</version>
   <executions>
      <execution>
         <goals>
            <goal>native-image</goal>
         </goals>
         <phase>package</phase>
      </execution>
   </executions>
</plugin>

So, the last in this section is just to run the Maven build. In my configuration, a native-image-maven-plugin needs to be activated under the native-image profile.

$ mvn clean package -Pnative-image

After the build native image of callme-service is visible inside the target directory.

The configuration of Skaffold is typical. We just need to enable Jib as a build tool.

apiVersion: skaffold/v2beta11
kind: Config
metadata:
  name: callme-service
build:
  artifacts:
  - image: piomin/callme-service
    jib: {}
deploy:
  kubectl:
    manifests:
      - k8s/ksvc.yaml

Finally, we may build and deploy our Spring Boot microservices on Knative with the following command.

$ skaffold run

5. Communication between microservices on Knative

I deployed two revisions of each application on Knative. Just for comparison, the first version of deployed applications is compiled with OpenJDK. Only the latest version is basing on the GraalVM native image. Thanks to that we may compare startup time for both revisions.

Let’s take a look at a list of revisions after deploying both versions of our applications. The traffic is splitted 60% to the latest version, and 40% to the previous version of each application.

Under the hood, Knative creates Kubernetes Services and multiple Deployments. There is always a single Deployment per each Knative Revision. Also, there are multiple services, but always one of them is per all revisions. That Service is an ExternalName service type. Assuming you still want to split traffic across multiple revisions you should use exactly that service in your communication. The name of the service is callme-service. However, we should use FQDN name with a namespace name and svc.cluster.local suffix.

We can use Spring RestTemplate for calling endpoint exposed by the callme-service. In order to guarantee tracing for the whole request path, we need to propagate Zipkin headers between the subsequent calls. For communication, we will use a service with a fully qualified internal domain name (callme-service.serverless.svc.cluster.local) as mentioned before.

@RestController
@RequestMapping("/caller")
public class CallerController {

   private RestTemplate restTemplate;

   CallerController(RestTemplate restTemplate) {
      this.restTemplate = restTemplate;
   }

   @Value("${spring.application.name}")
   private String appName;
   @Value("${POD_NAME}")
   private String podName;
   @Value("${POD_NAMESPACE}")
   private String podNamespace;
   @Autowired
   private CallerRepository repository;

   @GetMapping("/ping")
   public String ping(@RequestHeader HttpHeaders headers) {
      Caller c = repository.save(new Caller(new Date(), podName));
      String callme = callme(headers);
      return appName + "(id=" + c.getId() + "): " + podName + " in " + podNamespace
                     + " is calling " + callme;
   }

   private String callme(HttpHeaders headers) {
      MultiValueMap<String, String> map = new LinkedMultiValueMap<>();
      Set<String> headerNames = headers.keySet();
      headerNames.forEach(it -> map.put(it, headers.get(it)));
      HttpEntity httpEntity = new HttpEntity(map);
      ResponseEntity<String> entity = restTemplate
         .exchange("http://callme-service.serverless.svc.cluster.local/callme/ping",
                  HttpMethod.GET, httpEntity, String.class);
      return entity.getBody();
   }

}

In order to test the communication between our microservices we just need to invoke caller-service via Knative Route.

Let’s perform some test calls of the caller-service GET /caller/ping endpoint. We should use the URL http://caller-service-serverless.apps.cluster-d556.d556.sandbox262.opentlc.com/caller/ping.

In the first to requests caller-service calls the latest version of callme-service (compiled with GraalVM). In the third request it communicates with the older version callme-service (compiled with OpenJDK). Let’s compare the time of start for those two versions of the same application.

With GraalVM we have 0.3s instead of 5.9s. We should also keep in mind that our applications start an in-memory, embedded H2 database.

6. Configure tracing with Jaeger

In order to enable tracing for Knative, we need to update the knative-tracing ConfigMap in the knative-serving namespace. Of course, we first need to install Jaeger in our cluster.

apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-tracing
  namespace: knative-serving
spec:
  sample-rate: "1" 
  backend: zipkin 
  zipkin-endpoint: http://jaeger-collector.knative-serving.svc.cluster.local:9411/api/v2/spans 
  debug: "false"

You can also use Helm chart to install Jaeger. With this option, you need to execute the following Helm commands.

$ helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
$ helm install jaeger jaegertracing/jaeger

Knative automatically creates Zipkin span headers. The only single goal for us is to propagate HTTP headers between the caller-service and callme-service applications. In my configuration, Knative sends 100% traces to Jaeger. Let’s take a look at some traces for GET /caller/ping endpoint within Knative microservices mesh.

We can also take a look on the detailed view for every single request.

Conclusion

There are several important things you need to consider when you are running microservices on Knative. I focused on the aspects related to communication and tracing. I also showed that Spring Boot doesn’t have to start in a few seconds. With GraalVM it can start in milliseconds, so you can definitely consider it as a serverless framework. You may expect more articles about Knative soon!

The post Microservices on Knative with Spring Boot and GraalVM appeared first on Piotr's TechBlog.

The MicroProfile project breathes a new life into Java EE. Since the rise of microservices Java EE had lost its dominant position in the JVM enterprise area. As a result, application servers and EJBs have been replaced by lightweight frameworks like Spring Boot. MicroProfile is an answer to that. It defines Java EE standards for building microservices. Therefore it can be treated as a base to build more advanced frameworks like Quarkus or KumuluzEE.

If you are interested in frameworks built on top of MicroProfile, Quarkus is a good example: Quick Guide to Microservices with Quarkus on OpenShift. You can always implement your custom service discovery implementation for MicroProfile microservices. You should try with Consul: Quarkus Microservices with Consul Discovery.

Source code

If you would like to try it by yourself, you may always take a look at my source code. In order to do that you need to clone my repository sample-microprofile-microservices. Then you should go to the employee-service and department-service directories, and just follow my instructions

1. Running on WildFly

A few weeks ago WildFly has introduced the “Fat JAR” packaging feature. This feature is fully supported since WildFly 21. We can apply it during a Maven build by including wildfly-jar-maven-plugin to the pom.xml file. What is important, we don’t have to re-design an application to run it inside a bootable JAR.

In order to use the “Fat JAR” packaging feature, we need to add the package execution goal. Then we should install two features inside the configuration section. The first of them, the jaxrs-server feature, is a layer that allows us to build a typical REST application. The second of them, the microprofile-platform feature, enables MicroProfile on the WildFly server.

<profile>
   <id>bootable-jar</id>
   <activation>
      <activeByDefault>true</activeByDefault>
   </activation>
   <build>
      <finalName>${project.artifactId}</finalName>
      <plugins>
         <plugin>
            <groupId>org.wildfly.plugins</groupId>
            <artifactId>wildfly-jar-maven-plugin</artifactId>
            <version>2.0.2.Final</version>
            <executions>
               <execution>
                  <goals>
                     <goal>package</goal>
                  </goals>
               </execution>
            </executions>
            <configuration>
               <feature-pack-location>
                  wildfly@maven(org.jboss.universe:community-universe)#${version.wildfly}
               </feature-pack-location>
               <layers>
                  <layer>jaxrs-server</layer>
                  <layer>microprofile-platform</layer>
               </layers>
            </configuration>
         </plugin>
      </plugins>
   </build>
</profile>

Finally, we just need to execute the following command to build and run our “Fat JAR” application on WildFly.

$ mvn package wildfly-jar:run

If we run multiple applications on the same machine, we would have to override default HTTP and management ports. To do that we need to add the jvmArguments section inside configuration. We may insert there any number of JVM arguments. In that case, the required arguments are jboss.http.port and jboss.management.http.port.

<configuration>
   ...
   <jvmArguments>
      <jvmArgument>-Djboss.http.port=8090</jvmArgument>
      <jvmArgument>-Djboss.management.http.port=9090</jvmArgument>
   </jvmArguments>
</configuration>

2. Creating JAX-RS applications

In the first step, we will create simple REST applications with JAX-RS. WildFly provides all the required libraries, but we need to include both these artifacts for the compilation phase.

<dependency>
   <groupId>org.jboss.spec.javax.ws.rs</groupId>
   <artifactId>jboss-jaxrs-api_2.1_spec</artifactId>
   <scope>provided</scope>
</dependency>
<dependency>
   <groupId>jakarta.enterprise</groupId>
   <artifactId>jakarta.enterprise.cdi-api</artifactId>
   <scope>provided</scope>
</dependency>

Then, we should set the dependencyManagement section. We will use BOM provided by WildFly for both MicroProfile and Jakarta EE.

<dependencyManagement>
   <dependencies>
      <dependency>
         <groupId>org.wildfly.bom</groupId>
         <artifactId>wildfly-jakartaee8-with-tools</artifactId>
         <version>${version.wildfly}</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
      <dependency>
         <groupId>org.wildfly.bom</groupId>
         <artifactId>wildfly-microprofile</artifactId>
         <version>${version.wildfly}</version>
         <type>pom</type>
         <scope>import</scope>
      </dependency>
   </dependencies>
</dependencyManagement>

Here’s the JAX-RS controller inside employee-service. It uses an in-memory repository bean. It also injects a random delay to all exposed HTTP endpoints with the @Delay annotation. To clarify, I’m just setting it for future use, in order to present the metrics and fault tolerance features.

@Path("/employees")
@ApplicationScoped
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
@Delay
public class EmployeeController {

   @Inject
   EmployeeRepository repository;

   @POST
   public Employee add(Employee employee) {
      return repository.add(employee);
   }

   @GET
   @Path("/{id}")
   public Employee findById(@PathParam("id") Long id) {
      return repository.findById(id);
   }

   @GET
   public List<Employee> findAll() {
      return repository.findAll();
   }

   @GET
   @Path("/department/{departmentId}")
   public List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId) {
      return repository.findByDepartment(departmentId);
   }

   @GET
   @Path("/organization/{organizationId}")
   public List<Employee> findByOrganization(@PathParam("organizationId") Long organizationId) {
      return repository.findByOrganization(organizationId);
   }

}

Here’s a definition of the delay interceptor class. It is annotated with a base @Interceptor and custom @Delay. It injects a random delay between 0 and 1000 milliseconds to each method invoke.

@Interceptor
@Delay
public class AddDelayInterceptor {

   Random r = new Random();

   @AroundInvoke
   public Object call(InvocationContext invocationContext) throws Exception {
      Thread.sleep(r.nextInt(1000));
      System.out.println("Intercept");
      return invocationContext.proceed();
   }

}

Finally, let’s just take a look on the custom @Delay annotation.

@InterceptorBinding
@Target({METHOD, TYPE})
@Retention(RUNTIME)
public @interface Delay {
}

3. Enable metrics for MicroProfile microservices

Metrics is one of the core MicroProfile modules. Data is exposed via REST over HTTP under the /metrics base path in two different data formats for GET requests. These formats are JSON and OpenMetrics. The OpenMetrics text format is supported by Prometheus. In order to enable the MicroProfile metrics, we need to include the following dependency to Maven pom.xml.

<dependency>
   <groupId>org.eclipse.microprofile.metrics</groupId>
   <artifactId>microprofile-metrics-api</artifactId>
   <scope>provided</scope>
</dependency>

To enable the basic metrics we just need to annotate the controller class with @Timed.

@Path("/employees")
@ApplicationScoped
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
@Delay
@Timed
public class EmployeeController {
   ...
}

The /metrics endpoint is available under the management port. Firstly, let’s send some test requests, for example to the GET /employees endpoint. The application employee-service is available on http://localhost:8080/. Then let’s call the endpoint http://localhost:9990/metrics. Here’s a full list of metrics generated for the findAll method. Similar metrics would be generated for all other HTTP endpoints.

4. Generate OpenAPI specification

The REST API specification is another essential thing for all microservices. So, it is not weird that the OpenAPI module is a part of a MicroProfile core. The API specification is automatically generated after including the microprofile-openapi-api module. This module is a part microprofile-platform layer defined for wildfly-jar-maven-plugin.

After starting the application we may access OpenAPI documentation by calling http://localhost:8080/openapi endpoint. Then, we can copy the result to the Swagger editor. The graphical representation of the employee-service API is visible below.

5. Microservices inter-communication with MicroProfile REST client

The department-service calls endpoint GET /employees/department/{departmentId} from the employee-service. Then it returns a department with a list of all assigned employees.

@Getter
@Setter
@NoArgsConstructor
@AllArgsConstructor
public class Department {
   private Long id;
   private String name;
   private Long organizationId;
   private List<Employee> employees = new ArrayList<>();
}

Of course, we need to include the REST client module to the Maven dependencies.

<dependency>
   <groupId>org.eclipse.microprofile.rest.client</groupId>
   <artifactId>microprofile-rest-client-api</artifactId>
   <scope>provided</scope>
</dependency>

The MicroProfile REST module allows defining a client declaratively. We should annotate the client interface with @RegisterRestClient. The rest of the implementation is rather obvious.

@Path("/employees")
@RegisterRestClient(baseUri = "http://employee-service:8080")
public interface EmployeeClient {

   @GET
   @Path("/department/{departmentId}")
   @Produces(MediaType.APPLICATION_JSON)
   List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId);
}

Finally, we just need to inject the EmployeeClient bean to the controller class.

@Path("/departments")
@ApplicationScoped
@Produces(MediaType.APPLICATION_JSON)
@Consumes(MediaType.APPLICATION_JSON)
@Timed
public class DepartmentController {

   @Inject
   DepartmentRepository repository;
   @Inject
   EmployeeClient employeeClient;

   @POST
   public Department add(Department department) {
      return repository.add(department);
   }

   @GET
   @Path("/{id}")
   public Department findById(@PathParam("id") Long id) {
      return repository.findById(id);
   }

   @GET
   public List<Department> findAll() {
      return repository.findAll();
   }

   @GET
   @Path("/organization/{organizationId}")
   public List<Department> findByOrganization(@PathParam("organizationId") Long organizationId) {
      return repository.findByOrganization(organizationId);
   }

   @GET
   @Path("/organization/{organizationId}/with-employees")
   public List<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
      List<Department> departments = repository.findByOrganization(organizationId);
      departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
      return departments;
   }

}

The MicroProfile project does not implement service discovery patterns. There are some frameworks built on top of MicroProfile that provide such kind of implementation, for example, KumuluzEE. If you do not deploy our applications on OpenShift you may add the following entry in your /etc/hosts file to test it locally.

127.0.0.1 employee-service

Finally, let’s call endpoint GET /departments/organization/{organizationId}/with-employees. The result is visible in the picture below.

6. Java microservices fault tolerance with MicroProfile

To be honest, fault tolerance handling is my favorite feature of MicroProfile. We may configure them on the controller methods using annotations. We can choose between @Timeout, @Retry, @Fallback and @CircuitBreaker. Alternatively, it is possible to use a mix of those annotations on a single method. As you probably remember, we injected a random delay between 0 and 1000 milliseconds into all the endpoints exposed by employee-service. Now, let’s consider the method inside department-service that calls endpoint GET /employees/department/{departmentId} from employee-service. Firstly, we will annotate that method with @Timeout as shown below. The current timeout is 500 ms.

public class DepartmentController {

   @Inject
   DepartmentRepository repository;
   @Inject
   EmployeeClient employeeClient;

   ...

   @GET
   @Path("/organization/{organizationId}/with-employees")
   @Timeout(500)
   public List<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
      List<Department> departments = repository.findByOrganization(organizationId);
      departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
      return departments;
   }

}

Before calling the method, let’s create an exception mapper. If TimeoutException occurs, the department-service endpoint will return status HTTP 504 - Gateway Timeout.

@Provider
public class TimeoutExceptionMapper implements 
      ExceptionMapper<TimeoutException> {

   public Response toResponse(TimeoutException e) {
      return Response.status(Response.Status.GATEWAY_TIMEOUT).build();
   }

}

Then, we may proceed to call our test endpoint. Probably 50% of requests will finish with the result visible below.

On the other hand, we may enable a retry mechanism for such an endpoint. After that, the change for receive status HTTP 200 OK becomes much bigger than before.

@GET
@Path("/organization/{organizationId}/with-employees")
@Timeout(500)
@Retry(retryOn = TimeoutException.class)
public List<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
   List<Department> departments = repository.findByOrganization(organizationId);
   departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
   return departments;
}

7. Deploy MicroProfile microservices on OpenShift

We can easily deploy MicroProfile Java microservices on OpenShift using the JKube plugin. It is a successor of the deprecated Fabric8 Maven Plugin. Eclipse JKube is a collection of plugins and libraries that are used for building container images using Docker, JIB or S2I build strategies. It generates and deploys Kubernetes and OpenShift manifests at compile time too. So, let’s add openshift-maven-plugin to the pom.xml file.

The configuration visible below sets 2 replicas for the deployment and enforces using health checks. In addition to this, openshift-maven-plugin generates the rest of a deployment config based on Maven pom.xml structure. For example, it generates employee-service-deploymentconfig.yml, employee-service-route.yml, and employee-service-service.yml for the employee-service application.

<plugin>
   <groupId>org.eclipse.jkube</groupId>
   <artifactId>openshift-maven-plugin</artifactId>
   <version>1.0.2</version>
   <executions>
      <execution>
         <id>jkube</id>
         <goals>
            <goal>resource</goal>
            <goal>build</goal>
         </goals>
      </execution>
   </executions>
   <configuration>
      <resources>
         <replicas>2</replicas>
      </resources>
      <enricher>
         <config>
            <jkube-healthcheck-wildfly-jar>
               <enforceProbes>true</enforceProbes>
            </jkube-healthcheck-wildfly-jar>
         </config>
      </enricher>
   </configuration>
</plugin>

In order to deploy the application on OpenShift we need to run the following command.

$ mvn oc:deploy -P bootable-jar-openshift

Since the property enforceProbes has been enabled openshift-maven-plugin adds liveness and readiness probes to the DeploymentConfig. Therefore, we need to implement both these endpoints in our MicroProfile applications. MicroProfile provides a smart mechanism for creating liveness and readiness health checks. We just need to annotate the class with @Liveness or @Readiness, and implement the HealthCheck interface. Here’s the example implementation of the liveness endpoint.

@Liveness
@ApplicationScoped
public class LivenessEndpoint implements HealthCheck {
   @Override
   public HealthCheckResponse call() {
      return HealthCheckResponse.up("Server up");
   }
}

On the other hand, the implementation of the readiness probe also verifies the status of the repository bean. Of course, it is just a simple example.

@Readiness
@ApplicationScoped
public class ReadinessEndpoint implements HealthCheck {
   @Inject
   DepartmentRepository repository;

   @Override
   public HealthCheckResponse call() {
      HealthCheckResponseBuilder responseBuilder = HealthCheckResponse
         .named("Repository up");
      List<Department> departments = repository.findAll();
      if (repository != null && departments.size() > 0)
         responseBuilder.up();
      else
         responseBuilder.down();
      return responseBuilder.build();
   }
}

After deploying both employee-service and department-service application we may verify a list of DeploymentConfigs.

We can also navigate to the OpenShift console. Let’s take a look at a list of running pods. There are two instances of the employee-service and a single instance of department-service.

8. MicroProfile OpenTracing with Jaeger

Tracing is another important pattern in microservices architecture. The OpenTracing module is a part of MicroProfile specification. Besides the microprofile-opentracing-api library we also need to include the opentracing-api module.

<dependency>
   <groupId>org.eclipse.microprofile.opentracing</groupId>
   <artifactId>microprofile-opentracing-api</artifactId>
   <scope>provided</scope>
</dependency>
<dependency>
   <groupId>io.opentracing</groupId>
   <artifactId>opentracing-api</artifactId>
   <version>0.31.0</version>
</dependency>

By default, MicroProfile OpenTracing integrates the application with Jaeger. If you are testing our sample microservices on OpenShift, you may install Jaeger using an operator. Otherwise, we may just start it on the Docker container. The Jaeger UI is available on the address http://localhost:16686.

$ docker run -d --name jaeger \
-p 6831:6831/udp \
-p 16686:16686 \
jaegertracing/all-in-one:1.16.0

We don’t have to do anything more than adding the required dependencies to enable tracing. However, it is worth overriding the names of recorded operations. We may do it by annotating a particular method with @Traced and then by setting parameter operationName. The implementation of findByOrganizationWithEmployees method in the department-service is visible below.

public class DepartmentController {

   @Inject
   DepartmentRepository repository;
   @Inject
   EmployeeClient employeeClient;

   ...

   @GET
   @Path("/organization/{organizationId}/with-employees")
   @Timeout(500)
   @Retry(retryOn = TimeoutException.class)
   @Traced(operationName = "findByOrganizationWithEmployees")
   public List<Department> findByOrganizationWithEmployees(@PathParam("organizationId") Long organizationId) {
      List<Department> departments = repository.findByOrganization(organizationId);
      departments.forEach(d -> d.setEmployees(employeeClient.findByDepartment(d.getId())));
      return departments;
   }
   
}

We can also take a look at the fragment of implementation of EmployeeController.

public class EmployeeController {

   @Inject
   EmployeeRepository repository;

   ...
   
   @GET
   @Traced(operationName = "findAll")
   public List<Employee> findAll() {
      return repository.findAll();
   }

   @GET
   @Path("/department/{departmentId}")
   @Traced(operationName = "findByDepartment")
   public List<Employee> findByDepartment(@PathParam("departmentId") Long departmentId) {
      return repository.findByDepartment(departmentId);
   }
   
}

Before running the applications we should at least set the environment variable JAEGER_SERVICE_NAME. It configures the name of the application visible by Jaeger. For example, before starting the employee-service application we should set the value JAEGER_SERVICE_NAME=employee-service. Finally, let’s send some test requests to the department-service endpoint GET departments/organization/{organizationId}/with-employees.

$ curl http://localhost:8090/departments/organization/1/with-employees
$ curl http://localhost:8090/departments/organization/2/with-employees

After sending some test requests we may go to the Jaeger UI. The picture visible below shows the history of requests processed by the method findByOrganizationWithEmployees inside department-service.

As you probably remember, this method calls a method from the employee-service, and configures timeout and retries in case of failure. The picture below shows the details about a single request processed by the method findByOrganizationWithEmployees. To clarify, it has been retried once.

Conclusion

This article guides you through the most important steps of building Java microservices with MicroProfile. You may learn how to implement tracing, health checks, OpenAPI, and inter-service communication with a REST client. after reading you are able to run your MicroProfile Java microservices locally on WildFly, and moreover deploy them on OpenShift using a single maven command. Enjoy

The post Microprofile Java Microservices on WildFly appeared first on Piotr's TechBlog.