You can actually run MCP servers in many different ways. Ultimately, they are just ordinary applications whose task is to make a given tool available via an API compatible with the MCP protocol. The most popular AI IDE tools, such as Cloud Code, Codex, and Cursor, make it easy to run any MCP server. I will take a slightly unusual approach and use the support provided with Docker Desktop, namely the MCP Toolkit.

My idea for today is to build a simple Spring AI application that communicates with MCP servers for GitHub, SonarQube, and CircleCI to retrieve information about my repositories and projects hosted on those platforms. The Docker MCP Toolkit provides a single gateway that distributes incoming requests among running MCP servers. Let’s see how it works in practice!

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions. This repository contains several sample applications. The correct application for this article is in the spring-ai-mcp/external-mcp-sample-client directory.

Getting Started with Docker MCP Toolkit

First, run your Docker Desktop. You can find more than 300 popular MCP servers to run in the “Catalog” bookmark. Next, you should search for SonarQube, CircleCI, and GitHub Official servers (note that there are additional GitHub servers). To be honest, I encountered unexpected issues running the CircleCI server, so for now, I based the application on MCP communication with GitHub and SonarCloud.

Each MCP server usually requires configuration, such as your authorization token or service address. Therefore, before adding a server to Docker Toolkit, you must first configure it as described below. Only then should you click the “Add MCP server” button.

For the GitHub MCP server, in addition to entering the token itself, you must also authorize it via OAuth. Here, too, the MCP Toolkit provides graphical support. After entering the token, go to the “OAuth” tab to complete the process.

This is what your final result should look like before moving on to implementing the Spring Boot application. You have added two MCP servers, which together offer 65 tools.

To make both MCP servers available outside of Docker, you need to run the Docker MCP gateway. In the default stdio mode, the API is not exposed outside Docker. Therefore, you need to change the mode to streaming using the transport parameter, as shown below. The gateway is exposed on port 8811.

docker mcp gateway run --port 8811 --transport streaming

docker mcp gateway run --port 8811 --transport streaming

This is what it looks like after launch. Additionally, the Docker MCP gateway is secured by an API token. This will require appropriate settings on the MCP client side in the Spring AI application.

Integrate Spring AI with External MCP Clients

Prepare the MCP Client with Spring AI

Let’s move on to implementing our sample application. We need to include the Spring AI MCP client and the library that communicates with the LLM model. For me, it’s OpenAI, but you can use many other options available through Spring AI’s integration with popular chat models.

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-client-webflux</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
  </dependency>
</dependencies>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-mcp-client-webflux</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
  </dependency>
</dependencies>

Our MCP client must authenticate itself to the Docker MCP gateway using an API token. Therefore, we need to modify the Spring WebClient used by Spring AI to communicate with MCP servers. It is best to use the ExchangeFilterFunction interface to create an HTTP filter that adds the appropriate Authorization header with the bearer token to the outgoing request. The token will be injected from the application properties.

@Component
public class McpSyncClientExchangeFilterFunction implements ExchangeFilterFunction {

    @Value("${mcp.token}")
    private String token;

    @Override
    public Mono<ClientResponse> filter(ClientRequest request, 
                                       ExchangeFunction next) {

            var requestWithToken = ClientRequest.from(request)
                    .headers(headers -> headers.setBearerAuth(token))
                    .build();
            return next.exchange(requestWithToken);

    }

}

@Component
public class McpSyncClientExchangeFilterFunction implements ExchangeFilterFunction {

    @Value("${mcp.token}")
    private String token;

    @Override
    public Mono<ClientResponse> filter(ClientRequest request, 
                                       ExchangeFunction next) {

            var requestWithToken = ClientRequest.from(request)
                    .headers(headers -> headers.setBearerAuth(token))
                    .build();
            return next.exchange(requestWithToken);

    }

}

Then, let’s set the previously implemented filter for the default WebClient builder.

@SpringBootApplication
public class ExternalMcpSampleClient {

    public static void main(String[] args) {
        SpringApplication.run(ExternalMcpSampleClient.class, args);
    }

    @Bean
    WebClient.Builder webClientBuilder(McpSyncClientExchangeFilterFunction filterFunction) {
        return WebClient.builder()
                .filter(filterFunction);
    }
}

@SpringBootApplication
public class ExternalMcpSampleClient {

    public static void main(String[] args) {
        SpringApplication.run(ExternalMcpSampleClient.class, args);
    }

    @Bean
    WebClient.Builder webClientBuilder(McpSyncClientExchangeFilterFunction filterFunction) {
        return WebClient.builder()
                .filter(filterFunction);
    }
}

After that, we must configure the MCP gateway address and token in the application properties. To achieve that, we must use the spring.ai.mcp.client.streamable-http.connections property. The MCP gateway listens on port 8811. The token value will be read from the MCP_TOKEN environment variable.

spring.ai.mcp.client.streamable-http.connections:
  docker-mcp-gateway:
    url: http://localhost:8811

mcp.token: ${MCP_TOKEN}

spring.ai.mcp.client.streamable-http.connections:
  docker-mcp-gateway:
    url: http://localhost:8811

mcp.token: ${MCP_TOKEN}

Implement Application Logic with Spring AI and OpenAI Support

The concept behind the sample application is quite simple. It involves creating a @RestController per tool provided by each MCP server. For each, I will create a simple prompt to request the number of repositories or projects in my account on a given platform. Let’s start with SonCloud. Each implementation uses the Spring AI ToolCallbackProvider bean to enable the available MCP server to communicate with the LLM model.

@RestController
@RequestMapping("/sonarcloud")
public class SonarCloudController {

    private final static Logger LOG = LoggerFactory
        .getLogger(SonarCloudController.class);
    private final ChatClient chatClient;

    public SonarCloudController(ChatClient.Builder chatClientBuilder,
                                ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many projects in Sonarcloud do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

@RestController
@RequestMapping("/sonarcloud")
public class SonarCloudController {

    private final static Logger LOG = LoggerFactory
        .getLogger(SonarCloudController.class);
    private final ChatClient chatClient;

    public SonarCloudController(ChatClient.Builder chatClientBuilder,
                                ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many projects in Sonarcloud do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

Below is a very similar implementation for GitHub MCP. This controller is exposed under the /github context path.

@RestController
@RequestMapping("/github")
public class GitHubController {

    private final static Logger LOG = LoggerFactory
        .getLogger(GitHubController.class);
    private final ChatClient chatClient;

    public GitHubController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many repositories in GitHub do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

@RestController
@RequestMapping("/github")
public class GitHubController {

    private final static Logger LOG = LoggerFactory
        .getLogger(GitHubController.class);
    private final ChatClient chatClient;

    public GitHubController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many repositories in GitHub do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

Finally, there is the controller implementation for CircleCI MCP. It is available externally under the /circleci context path.

@RestController
@RequestMapping("/circleci")
public class CircleCIController {

    private final static Logger LOG = LoggerFactory
        .getLogger(CircleCIController.class);
    private final ChatClient chatClient;

    public CircleCIController(ChatClient.Builder chatClientBuilder,
                              ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many projects in CircleCI do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

@RestController
@RequestMapping("/circleci")
public class CircleCIController {

    private final static Logger LOG = LoggerFactory
        .getLogger(CircleCIController.class);
    private final ChatClient chatClient;

    public CircleCIController(ChatClient.Builder chatClientBuilder,
                              ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/count")
    String countRepositories() {
        PromptTemplate pt = new PromptTemplate("""
                How many projects in CircleCI do I have ?
                """);
        Prompt p = pt.create();
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

The last controller implementation is a bit more complex. First, I need to instruct the LLM model to generate project names in SonarQube and specify my GitHub username. This will not be part of the main prompt. Rather, it will be the system role, which guides the AI’s behavior and response style. Therefore, I’ll create the SystemPromptTemplate first. The user role prompt accepts an input parameter specifying the name of my GitHub repository. The response should combine data on the last commit in a given repository with the status of the most recent SonarQube analysis. In this case, the LLM will need to communicate with two MCP servers running with Docker MCP simultaneously.

@RestController
@RequestMapping("/global")
public class GlobalController {

    private final static Logger LOG = LoggerFactory
        .getLogger(CircleCIController.class);
    private final ChatClient chatClient;

    public GlobalController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/status/{repo}")
    String repoStatus(@PathVariable String repo) {
        SystemPromptTemplate st = new SystemPromptTemplate("""
                My username in GitHub is piomin.
                Each my project key in SonarCloud contains the prefix with my organization name and _ char.
                """);
        var stMsg = st.createMessage();

        PromptTemplate pt = new PromptTemplate("""
                When was the last commit made in my GitHub repository {repo} ?
                What is the latest analyze status in SonarCloud for that repo ?
                """);
        var usMsg = pt.createMessage(Map.of("repo", repo));

        Prompt prompt = new Prompt(List.of(usMsg, stMsg));
        return this.chatClient.prompt(prompt)
                .call()
                .content();
    }
}

@RestController
@RequestMapping("/global")
public class GlobalController {

    private final static Logger LOG = LoggerFactory
        .getLogger(CircleCIController.class);
    private final ChatClient chatClient;

    public GlobalController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultToolCallbacks(tools)
                .build();
    }

    @GetMapping("/status/{repo}")
    String repoStatus(@PathVariable String repo) {
        SystemPromptTemplate st = new SystemPromptTemplate("""
                My username in GitHub is piomin.
                Each my project key in SonarCloud contains the prefix with my organization name and _ char.
                """);
        var stMsg = st.createMessage();

        PromptTemplate pt = new PromptTemplate("""
                When was the last commit made in my GitHub repository {repo} ?
                What is the latest analyze status in SonarCloud for that repo ?
                """);
        var usMsg = pt.createMessage(Map.of("repo", repo));

        Prompt prompt = new Prompt(List.of(usMsg, stMsg));
        return this.chatClient.prompt(prompt)
                .call()
                .content();
    }
}

Before running the app, we must set two required environment variables that contain the OpenAI and Docker MCP gateway tokens.

export MCP_TOKEN=by1culxc6sctmycxtyl9xh7499mb8pctbsdb3brha1hvmm4d8l
export SPRING_AI_OPENAI_API_KEY=<YOUR_OPEN_AI_TOKEN>

export MCP_TOKEN=by1culxc6sctmycxtyl9xh7499mb8pctbsdb3brha1hvmm4d8l
export SPRING_AI_OPENAI_API_KEY=<YOUR_OPEN_AI_TOKEN>

Finally, we can run our Spring Boot app with the following command.

mvn spring-boot:run

mvn spring-boot:run

Firstly, I’m going to ask about the number of my GitHub repositories.

curl http://localhost:8080/github/count

curl http://localhost:8080/github/count

Then, I can check the number of projects in my SonarCloud account.

curl http://localhost:8080/github/sonarcloud

curl http://localhost:8080/github/sonarcloud

Finally, I can choose a specific repository and verify the last commit and the current analysis status in SonarCloud.

curl http://localhost:8080/global/status/sample-spring-boot-kafka

curl http://localhost:8080/global/status/sample-spring-boot-kafka

Here’s the LLM answer for my sample-spring-boot-kafka repository. You can perform the same exercise for your repositories and projects.

Conclusion

Spring AI, combined with the MCP client, opens a powerful path toward building truly tool-aware AI applications. By using the Docker MCP Gateway, we can easily host and manage MCP servers such as GitHub or SonarQube consistently and reproducibly, without tightly coupling them to our application runtime. Docker provides a user-friendly interface for managing MCP servers, giving users access to everything through a single MCP gateway. This approach appears to have advantages, particularly during application development.

The post Spring AI with External MCP Servers appeared first on Piotr's TechBlog.

Previously, I published several articles about Spring AI with examples of using different AI models. Therefore, I will not focus on the introduction to Spring AI. For example, you can read about integration between Spring AI and Azure AI in the following post. Please refer to the following article for a quick intro to the Spring AI project.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Prerequisites

Create the OpenShift Cluster

For this exercise, you will need a relatively large OpenShift cluster. At least one of the cluster’s nodes must have a GPU. I created a cluster on AWS with one node on a g4dn.12xlarge machine. On OpenShift, you can achieve this by creating the MachineSet object that creates nodes using the appropriate virtual machine available on AWS.

Install Required Operators

Next, install and configure several operators on the cluster. Begin with the “Node Feature Discovery” operator. On OpenShift, this operator enables automatic discovery of cluster nodes with features such as GPUs. After installing the operator, create the NodeFeatureDiscovery object. The default values set by the OpenShift console during object creation are sufficient.

The operator’s task is to mark the node with the detected GPU using the appropriate label. The label is feature.node.kubernetes.io/pci-10de.present=true. After configuring the operator, verify that the correct GPU has been detected.

$ oc get node -l feature.node.kubernetes.io/pci-10de.present=true
NAME                                        STATUS   ROLES    AGE   VERSION
ip-10-0-45-120.us-east-2.compute.internal   Ready    worker   15d   v1.31.6

Next, install the NVIDIA GPU Operator. This operator automatically installs, configures, and manages NVIDIA drivers and tools on nodes with NVIDIA graphics cards. This allows OpenShift to recognize the GPU as a resource that can be declared in pods. This will enable OpenShift to work with the “Node Feature Discovery” operator to label nodes with GPUs. The NVIDIA GPU operator uses the feature.node.kubernetes.io/pci-10de.present=true label to determine where to install the drivers. For this to happen, the ClusterPolicy object must be created. As before, you can use the default values generated by the OpenShift Console when creating this object.

The OpenShift AI feature for serving AI models requires the installation of OpenShift Serverless and OpenShift Service Mesh operators. The key solution here is KServe. KServe uses Knative to scale models on demand and integrates with Istio to secure model routing and versioning.

The final step in this phase is to install the OpenShift AI Operator and create the DataScienceCluster object. If the previous installations were successful, everything will be configured automatically after creating the DataScienceCluster object. For instance, OpenShift AI will make the Istio control plane and the Knative Serving component.

OpenShift AI creates several namespaces within a cluster. The most important is the redhat-ods-applications namespace, where most components comprising the entire solution are run.

$ oc get pod -n redhat-ods-applications
NAME                                                              READY   STATUS    RESTARTS   AGE
authorino-767bd64465-fq8bl                                        1/1     Running   0          15d
codeflare-operator-manager-5c69778b87-wxcwp                       1/1     Running   0          15d
data-science-pipelines-operator-controller-manager-6686587wcmkr   1/1     Running   0          15d
etcd-549d769449-hqzwt                                             1/1     Running   0          15d
kserve-controller-manager-85f9b8d66f-qpxbf                        1/1     Running   0          15d
kuberay-operator-8d77dcf84-qgsq5                                  1/1     Running   0          15d
kueue-controller-manager-7c895bd669-467nk                         1/1     Running   0          6h8m
modelmesh-controller-7f9dd5f848-ljlxp                             1/1     Running   0          15d
modelmesh-controller-7f9dd5f848-qqsl8                             1/1     Running   0          24d
modelmesh-controller-7f9dd5f848-txlhd                             1/1     Running   0          24d
notebook-controller-deployment-86f5b87585-p6nz5                   1/1     Running   0          15d
odh-model-controller-574ff4657-q75gr                              1/1     Running   0          15d
odh-notebook-controller-manager-9d754d5f-2ptk9                    1/1     Running   0          15d
rhods-dashboard-5b96595667-79tx6                                  2/2     Running   0          15d
rhods-dashboard-5b96595667-8m52g                                  2/2     Running   0          15d
rhods-dashboard-5b96595667-kx7p4                                  2/2     Running   0          15d
rhods-dashboard-5b96595667-nn2cf                                  2/2     Running   0          15d
rhods-dashboard-5b96595667-ttcht                                  2/2     Running   0          15d
trustyai-service-operator-controller-manager-bd9fbdb6d-kcd57      1/1     Running   0          15d

Configure and Use OpenShift AI

After installing OpenShift AI on a cluster, you can use its graphical UI. To access it, select “Red Hat OpenShift AI” from the menu at the top of the page.

After selecting the indicated option, you will be redirected to the following page. This page allows you to configure and use OpenShift AI on a cluster. The first step is to select a namespace on the cluster for the AI project. In my case, the namespace is ai.

To run an AI model on a cluster, choose how to serve it first. You can choose between a single-model serving platform and a multi-model serving platform. With the former, each model is deployed on its model server. Multiple models can be deployed on a single shared server with multi-model platforms. This article will use the first option: a single-model serving platform.

The next step is to create an acceleration profile. This profile should be created automatically after installing and configuring the NVIDIA GPU Operator. If, for some reason, it was not, you can easily create it manually. When creating this object, enter the nvidia.com/gpu value in the identifier field.

You can either click on the profile from the UI or create it using the YAML manifest.

apiVersion: dashboard.opendatahub.io/v1
kind: AcceleratorProfile
metadata:
  name: nvidia
  namespace: redhat-ods-applications
spec:
  displayName: nvidia
  enabled: true
  identifier: nvidia.com/gpu

Serve Model on OpenShift AI with vLLM

Create ServingRuntime Resource

In the previous step, we configured OpenShift AI to deploy the model with a single-model serving platform and a GPU accelerator. We will use KServe’s ModelCar functionality to deploy the model, which allows us to serve models directly from a container. This functionality is described in an article published on the Red Hat Developer blog. The article demonstrates how to build an image containing a model downloaded from the Hugging Face Hub. In turn, we will use images that have already been built and are available in the quay.io/repository/redhat-ai-services/modelcar-catalog repository. You can find ready-made images for AI models such as Granite and Llama.

To run a model on OpenShift AI in single-model serving runtime mode, you must define two objects: ServingRuntime and InferenceService. According to the OpenShift AI documentation, the ServingRuntime CR creates a serving runtime, an environment for deploying and managing a model. Here’s the ServingRuntime object that creates a runtime for the Llama 3.2 AI model. The annotation opendatahub.io/recommended-accelerators sets the name of the recommended accelerator to use with the runtime. Its value should be identical to the identifier field in the AcceleratorProfile object (1). The openshift.io/display-name annotation keeps the name with which the serving runtime is displayed (2). The spec.containers.image field indicates the runtime container image used by the serving runtime (3). This image differs depending on the type of accelerator used. Finally, the ServingRuntime object specifies that the single-model serving is used (4) and the vLLM model is supported by the runtime (5).

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  annotations:
    opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]' # (1)
    openshift.io/display-name: vLLM ServingRuntime for KServe # (2)
  labels:
    opendatahub.io/dashboard: "true"
  name: llama-32-3b-instruct
spec:
  annotations:
    prometheus.io/path: /metrics 
    prometheus.io/port: "8080" 
  containers :
    - args:
        - --port=8080
        - --model=/mnt/models 
        - --served-model-name={{.Name}} 
      command: 
        - python
        - '-m'
        - vllm.entrypoints.openai.api_server
      env:
        - name: HF_HOME
          value: /tmp/hf_home
      # (3)
      image:
quay.io/modh/vllm@sha256:0d55419f3d168fd80868a36ac89815dded9e063937a8409b7edf3529771383f3
    name: kserve-container
    ports:
      - containerPort: 8080
        protocol: TCP
  multiModel: false # (4)
  supportedModelFormats: # (5) 
    - autoSelect: true
      name: vLLM

Create InterferenceService Resource

The InferenceService CRD creates a server or inference service that processes inference queries, passes them to the model, and returns the inference output. Here’s the InferenceService object related to the previously created llama-32-3b-instruct runtime (1). It must define some vLLM parameters to successfully run the model on the existing infrastructure and enable tool calling support on the Llama 3.2 model (2). The InferenceService object specifies the image containing the Llama 3.2 model, published in the the quay.io/redhat-ai-services/modelcar-catalog:llama-3.2-3b-instruct repository (3). Alternatively, you can create your image, publish it in the custom registry, and run it on OpenShift using InferenceService CR.

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  annotations:
    openshift.io/display-name: llama-32-3b-instruct
    serving.knative.openshift.io/enablePassthrough: 'true'
    serving.kserve.io/deploymentMode: Serverless
    sidecar.istio.io/inject: 'true'
    sidecar.istio.io/rewriteAppHTTPProbers: 'true'
  name: llama-32-3b-instruct # (1)
  labels:
    opendatahub.io/dashboard: 'true'
spec:
  predictor:
    maxReplicas: 1
    minReplicas: 1
    model: # (2)
      args:
        - '--dtype=half'
        - '--max_model_len=8192'
        - '--gpu_memory_utilization=.95'
        - '--enable-auto-tool-choice'
        - '--tool_call_parser=llama3_json'
      modelFormat:
        name: vLLM
      name: ''
      resources:
        limits:
          cpu: '8'
          memory: 10Gi
          nvidia.com/gpu: '1'
        requests:
          cpu: '4'
          memory: 8Gi
          nvidia.com/gpu: '1'
      runtime: llama-32-3b-instruct
      storageUri: 'oci://quay.io/redhat-ai-services/modelcar-catalog:llama-3.2-3b-instruct' # (3)

Deploy with OpenShift AI

You can also create the same configuration using the OpenShift AI UI. The diagram below shows the settings you need for Granite 3.2.

The OpenShift AI UI lists all the models running in a given AI project. You can check the endpoint where a particular model is available. In this case, two models are running in the AI project: Llama 3.2 and Granite 3.2. Both models are available internally on the cluster and externally via the Knative Route object.

Both models are automatically exposed on the node with the GPU. You can check the GPU resource reservations on a node using the oc describe command:

A single-model serving platform runs AI models as the Knative Service. You can use the oc get ksvc command to display a list of Knative services running in the ai namespace.

$ oc get ksvc -n ai
NAME                               URL                                                                                 LATESTCREATED                            LATESTREADY                              READY   REASON
granite-32-2b-instruct-predictor   https://granite-32-2b-instruct-predictor-ai.apps.piomin.ewyw.p1.openshiftapps.com   granite-32-2b-instruct-predictor-00007   granite-32-2b-instruct-predictor-00007   True    
llama-32-3b-instruct-predictor     https://llama-32-3b-instruct-predictor-ai.apps.piomin.ewyw.p1.openshiftapps.com     llama-32-3b-instruct-predictor-00002     llama-32-3b-instruct-predictor-00002     True 

Integrate Spring AI with vLLM

Dependencies and Properties

The vLLM runtime is compatible with the OpenAI REST API. To integrate our sample Spring Boot application with a model running on vLLM, we must use the standard Spring AI OpenAI starter. The app in the spring-ai-showcase repository has more functionality than what is tested in this article. In simplified terms, the list of dependencies needed for the app to communicate with the OpenAI API and the model running on OpenShift AI is below.

<dependency>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-autoconfigure-model-openai</artifactId>
</dependency>

Although the model itself served on OpenShift AI does not require authorization with an API key the spring.ai.openai.api-key Spring AI parameter must be set. The endpoint’s address provided through the vLLM runtime must be specified in the spring.ai.openai.chat.base-url parameter. The default name of the model used must also be overwritten with the name under which the model was run on OpenShift AI. This name is for Llama 3.2 llama-32-3b-instruct. Below is a list of all the Spring Boot settings required for vLLM integration, which is available in the application-vllm.properties file.

spring.ai.openai.api-key = ${OPENAI_API_KEY:dummy}
spring.ai.openai.chat.base-url = https://llama-32-3b-instruct-ai.apps.piomin.ewyw.p1.openshiftapps.com
spring.ai.openai.chat.options.model = llama-32-3b-instruct

Implementation with Spring AI

The code below demonstrates how @RestController implements communication between the application and the target AI model. The @RestController class injects an auto-configured ChatClient.Builder to create an instance of ChatClient. The PersonController class implements a method for returning a list of persons from the GET /persons endpoint. The main goal is to generate a list of 10 objects with the fields defined in the Person class. The id field should be auto-incremented. The PromptTemplate object defines a message that will be sent to the chat model AI API. It doesn’t have to specify the exact fields that should be returned. This part is handled automatically by the Spring AI library after we invoke the entity() method on the ChatClient instance. The ParameterizedTypeReference object inside the entity method tells Spring AI to generate a list of objects.

@RestController
@RequestMapping("/persons")
public class PersonController {

    private final ChatClient chatClient;

    public PersonController(ChatClient.Builder chatClientBuilder,
                            ChatMemory chatMemory) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(
                        new PromptChatMemoryAdvisor(chatMemory),
                        new SimpleLoggerAdvisor())
                .build();
    }

    @GetMapping
    List<Person> findAll() {
        PromptTemplate pt = new PromptTemplate("""
                Return a current list of 10 persons if exists or generate a new list with random values.
                Each object should contain an auto-incremented id field.
                The age value should be a random number between 18 and 99.
                Do not include any explanations or additional text.
                Return data in RFC8259 compliant JSON format.
                """);

        return this.chatClient.prompt(pt.create())
                .call()
                .entity(new ParameterizedTypeReference<>() {});
    }

    @GetMapping("/{id}")
    Person findById(@PathVariable String id) {
        PromptTemplate pt = new PromptTemplate("""
                Find and return the object with id {id} in a current list of persons.
                """);
        Prompt p = pt.create(Map.of("id", id));
        return this.chatClient.prompt(p)
                .call()
                .entity(Person.class);
    }
}

The llama-32-3b-instruct model uses a “tool-calling” approach for API calls. You can read more about it in one of my Spring AI articles, which are available at this link. For instance, the class below implements the @Tool annotation, connecting to the database and searching it for a list of shares for individual companies. The key to using this tool is its description in the description field, which is then appropriately interpreted by the LLM model.

public class WalletTools {

    private WalletRepository walletRepository;

    public WalletTools(WalletRepository walletRepository) {
        this.walletRepository = walletRepository;
    }

    @Tool(description = "Number of shares for each company in my wallet")
    public List<Share> getNumberOfShares() {
        return (List<Share>) walletRepository.findAll();
    }
}

Then, the @Tool reference is set to the chat client when it interacts with the AI model. The AI model can call the tool’s method as required based on the tool’s description and the input prompt’s content.

@RestController
@RequestMapping("/wallet")
public class WalletController {

    private final ChatClient chatClient;
    private final StockTools stockTools;
    private final WalletTools walletTools;

    public WalletController(ChatClient.Builder chatClientBuilder,
                            StockTools stockTools,
                            WalletTools walletTools) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(new SimpleLoggerAdvisor())
                .build();
        this.stockTools = stockTools;
        this.walletTools = walletTools;
    }
    
    @GetMapping("/with-tools")
    String calculateWalletValueWithTools() {
        PromptTemplate pt = new PromptTemplate("""
        What’s the current value in dollars of my wallet based on the latest stock daily prices ?
        """);

        return this.chatClient.prompt(pt.create())
                .tools(stockTools, walletTools)
                .call()
                .content();
    }
    
}

Run Spring Boot Application

Activate the vllm profile when launching the Spring Boot application. This will cause the application to read the settings entered in the application-vllm.properties file.

mvn spring-boot:run -Dspring-boot.run.profiles=vllm

Once the application runs, you will call all three endpoints implemented in the previously discussed code snippets. These endpoints are:

• GET /persons
• GET /persons/{id}
• GET /wallet/with-tools

Once launched, the application can be accessed locally on 8080 port.

$ curl http://localhost:8080/persons
$ curl http://localhost:8080/persons/1
$ curl http://localhost:8080/wallet/with-tools

Alternatively, you can deploy the Spring Boot application on OpenShift and expose it outside the cluster with the Route object. The simplest way to achieve that is through the odo CLI tool. You can find more details about odo in the following post. To deploy the app with odo run the following command:

odo dev

After that, the application should be deployed in the selected namespace and available for testing on the 20001 local port, thanks to the port-forwarding feature.

Here’s the example output:

Final Thoughts

This article demonstrates the simplest way to integrate a Java application with an AI model running on OpenShift via an OpenAI-compliant interface. Preparing and exposing such a model to OpenShift AI requires several steps, such as installing and configuring Kubernetes operators. However, KServe’s ModelCar approach standardizes the entire process, making AI models available as containers.

The post OpenShift AI with vLLM and Spring AI appeared first on Piotr's TechBlog.

This is the eighth part of my series of articles about Spring Boot and AI. It is worth reading the following posts before proceeding with the current one. Here’s a list of articles about Spring AI on my blog with a short description:

1. https://piotrminkowski.com/2025/01/28/getting-started-with-spring-ai-and-chat-model: The first tutorial introduces the Spring AI project and its support for building applications based on chat models like OpenAI or Mistral AI.
2. https://piotrminkowski.com/2025/01/30/getting-started-with-spring-ai-function-calling: The second tutorial shows Spring AI support for Java function calling with the OpenAI chat model.
3. https://piotrminkowski.com/2025/02/24/using-rag-and-vector-store-with-spring-ai: The third tutorial shows Spring AI support for RAG (Retrieval Augmented Generation) and vector store.
4. https://piotrminkowski.com/2025/03/04/spring-ai-with-multimodality-and-images: The fourth tutorial shows Spring AI support for a multimodality feature and image generation
5. https://piotrminkowski.com/2025/03/10/using-ollama-with-spring-ai: The fifth tutorial shows Spring AI support for interactions with AI models run with Ollama
6. https://piotrminkowski.com/2025/03/13/tool-calling-with-spring-ai: The sixth tutorial shows Spring AI support for the Tool Calling feature.
7. https://piotrminkowski.com/2025/03/17/using-model-context-protocol-mcp-with-spring-ai: The seventh tutorial shows Spring AI support for MCP (Model Context Protocol) in Spring Boot server-side and client-side applications

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Enable and Configure Azure OpenAI

You need to begin the exercise by creating an instance of the Azure OpenAI service. The most crucial element here is the service’s name since it is part of the exposed Open AI endpoint. My service’s name is piomin-azure-openai.

The Azure OpenAI service should be exposed without restrictions to allow easy access to the Spring AI app.

After creating the service, go to its main page in the Azure Portal. It provides information about API keys and an endpoint URL. Also, you have to deploy an Azure OpenAI model to start making API calls from your Spring AI app.

Copy the key and the endpoint URL and save them for later usage.

You must create a new deployment with an AI model in the Azure AI Foundry portal. There are several available options. The Spring AI Azure OpenAI starter by default uses the gpt-4o model. If you choose another AI model, you will have to set its name in the spring.ai.azure.openai.chat.options.deployment-name Spring AI property. After selecting the preferred model, click the “Confirm” button.

Finally, you can deploy the model on the Azure AI Foundry portal. Choose the most suitable deployment type for your needs.

Azure allows us to deploy multiple models. You can verify a list of model deployments here:

That’s all on the Azure Portal side. Now it’s time for the implementation part in the application source code.

Enable Azure OpenAI for Spring AI

Spring AI provides the Spring Boot starter for the Azure OpenAI Chat Client. You must add the following dependency to your Maven pom.xml file. Since the sample Spring Boot application is portable across various AI models, it includes the Azure OpenAI starter only if the azure-ai profile is active. Otherwise, it uses the spring-ai-openai-spring-boot-starter library.

<profile>
  <id>azure-ai</id>
  <dependencies>
    <dependency>
      <groupId>org.springframework.ai</groupId>
      <artifactId>spring-ai-azure-openai-spring-boot-starter</artifactId>
    </dependency>
  </dependencies>
</profile>

It’s time to use the key you previously copied from the Azure OpenAI service page. Let’s export it as the AZURE_OPENAI_API_KEY environment variable.

export AZURE_OPENAI_API_KEY=<YOUR_AZURE_OPENAI_API_KEY>

Here are the application properties dedicated to the azure-ai Spring Boot profile. The previously exported AZURE_OPENAI_API_KEY environment variable is set as the spring.ai.azure.openai.api-key property. You also must set the OpenAI service endpoint. This address depends on your Azure OpenAI service name.

spring.ai.azure.openai.api-key = ${AZURE_OPENAI_API_KEY}
spring.ai.azure.openai.endpoint = https://piomin-azure-openai.openai.azure.com/

To run the application and connect to your instance of the Azure OpenAI service, you must activate the azure-ai Maven profile and the Spring Boot profile under the same name. Here’s the required command:

mvn spring-boot:run -Pazure-ai -Dspring-boot.run.profiles=azure-ai

Test Spring AI Features with Azure OpenAI

I described several Spring AI features in the previous articles from this series. In each section, I will briefly mention the tested feature with a fragment of the sample source code. Please refer to my previous posts for more details about each feature and its sample implementation.

Chat Client with Memory and Structured Output

Here’s the @RestController containing endpoints we will use in these tests.

@RestController
@RequestMapping("/persons")
public class PersonController {

    private final ChatClient chatClient;

    public PersonController(ChatClient.Builder chatClientBuilder,
                            ChatMemory chatMemory) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(
                        new PromptChatMemoryAdvisor(chatMemory),
                        new SimpleLoggerAdvisor())
                .build();
    }

    @GetMapping
    List<Person> findAll() {
        PromptTemplate pt = new PromptTemplate("""
                Return a current list of 10 persons if exists or generate a new list with random values.
                Each object should contain an auto-incremented id field.
                The age value should be a random number between 18 and 99.
                Do not include any explanations or additional text.
                Return data in RFC8259 compliant JSON format.
                """);

        return this.chatClient.prompt(pt.create())
                .call()
                .entity(new ParameterizedTypeReference<>() {});
    }

    @GetMapping("/{id}")
    Person findById(@PathVariable String id) {
        PromptTemplate pt = new PromptTemplate("""
                Find and return the object with id {id} in a current list of persons.
                """);
        Prompt p = pt.create(Map.of("id", id));
        return this.chatClient.prompt(p)
                .call()
                .entity(Person.class);
    }
}

First, you must call the endpoint that generates a list of ten persons from different countries. Then choose one person by ID to pick it up from the chat memory. Here are the results.

The interesting part happens in the background. Here’s a fragment of advice context added to the prompt by Spring AI.

Tool Calling

Here’s the @RestController containing endpoints we will use in these tests. There are two tools injected into the chat client: StockTools and WalletTools. These tools interact with a local H2 database to get a sample stock wallet structure and with the stock online API to load the latest share prices.

@RestController
@RequestMapping("/wallet")
public class WalletController {

    private final ChatClient chatClient;
    private final StockTools stockTools;
    private final WalletTools walletTools;

    public WalletController(ChatClient.Builder chatClientBuilder,
                            StockTools stockTools,
                            WalletTools walletTools) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(new SimpleLoggerAdvisor())
                .build();
        this.stockTools = stockTools;
        this.walletTools = walletTools;
    }

    @GetMapping("/with-tools")
    String calculateWalletValueWithTools() {
        PromptTemplate pt = new PromptTemplate("""
        What’s the current value in dollars of my wallet based on the latest stock daily prices ?
        """);

        return this.chatClient.prompt(pt.create())
                .tools(stockTools, walletTools)
                .call()
                .content();
    }

    @GetMapping("/highest-day/{days}")
    String calculateHighestWalletValue(@PathVariable int days) {
        PromptTemplate pt = new PromptTemplate("""
        On which day during last {days} days my wallet had the highest value in dollars based on the historical daily stock prices ?
        """);

        return this.chatClient.prompt(pt.create(Map.of("days", days)))
                .tools(stockTools, walletTools)
                .call()
                .content();
    }
}

You must have your API key for the Twelvedata service to run these tests. Don’t forget to export it as the STOCK_API_KEY environment variable before running the app.

export STOCK_API_KEY=<YOUR_STOCK_API_KEY>

The GET /wallet/with-tools endpoint calculates the current stock wallet value in dollars.

The GET /wallet/highest-day/{days} computes the value of the stock wallet for a given period in days and identifies the day with the highest value.

Multimodality and Images

Here’s a part of the @RestController responsible for describing image content and generating a new image with a given item.

@RestController
@RequestMapping("/images")
public class ImageController {

    private final static Logger LOG = LoggerFactory.getLogger(ImageController.class);
    private final ObjectMapper mapper = new ObjectMapper();

    private final ChatClient chatClient;
    private ImageModel imageModel;

    public ImageController(ChatClient.Builder chatClientBuilder,
                           Optional<ImageModel> imageModel) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(new SimpleLoggerAdvisor())
                .build();
        imageModel.ifPresent(model -> this.imageModel = model);
    }
        
    @GetMapping("/describe/{image}")
    List<Item> describeImage(@PathVariable String image) {
        Media media = Media.builder()
                .id(image)
                .mimeType(MimeTypeUtils.IMAGE_PNG)
                .data(new ClassPathResource("images/" + image + ".png"))
                .build();
        UserMessage um = new UserMessage("""
        List all items you see on the image and define their category.
        Return items inside the JSON array in RFC8259 compliant JSON format.
        """, media);
        return this.chatClient.prompt(new Prompt(um))
                .call()
                .entity(new ParameterizedTypeReference<>() {});
    }
    
    @GetMapping(value = "/generate/{object}", produces = MediaType.IMAGE_PNG_VALUE)
    byte[] generate(@PathVariable String object) throws IOException, NotSupportedException {
        if (imageModel == null)
            throw new NotSupportedException("Image model is not supported");
        ImageResponse ir = imageModel.call(new ImagePrompt("Generate an image with " + object, ImageOptionsBuilder.builder()
                .height(1024)
                .width(1024)
                .N(1)
                .responseFormat("url")
                .build()));
        String url = ir.getResult().getOutput().getUrl();
        UrlResource resource = new UrlResource(url);
        LOG.info("Generated URL: {}", url);
        dynamicImages.add(Media.builder()
                .id(UUID.randomUUID().toString())
                .mimeType(MimeTypeUtils.IMAGE_PNG)
                .data(url)
                .build());
        return resource.getContentAsByteArray();
    }
    
}

The GET /images/describe/{image} returns a structured list of items identified on a given image. It also categorizes each detected item. In this case, there are two available categories: fruits and vegetables.

By the way, here’s the image described above.

The image generation feature requires a dedicated model on Azure AI. The DALL-E 2 and DALL-E 3 models on Azure support a text-to-image feature.

The application must be aware of the model name. That’s why you must add a new property to your application properties with the following value.

spring.ai.azure.openai.image.options.deployment-name = dall-e-3

Then you must restart the application. After that, you can generate an image by calling the GET /images/generate/{object} endpoint. Here’s the result for the pineapple.

Enable Azure CosmosDB Vector Store

Dependency

By default, the sample Spring Boot application uses Pinecone vector store. However, SpringAI supports two services available on Azure: Azure AI Search and CosmosDB. Let’s choose CosmosDB as the vector store. You must add the following dependency to your Maven pom.xml file:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-azure-cosmos-db-store-spring-boot-starter</artifactId>
</dependency>

Configuration on Azure

Then, you must create an instance of CosmosDB in your Azure account. The name of my instance is piomin-ai-cosmos.

Once it is created, you will obtain its address and API key. To do that, go to the “Settings -> Keys” menu and save both values visible below.

Then, you have to create a dedicated database and container for your application. To do that, go to the “Data Explorer” tab and provide names for the database and container ID. You must also set the partition key.

All previously provided values must be set in the application properties. Export your CosmosDB API key as the AZURE_VECTORSTORE_API_KEY environment variable.

spring.ai.vectorstore.cosmosdb.endpoint = https://piomin-ai-cosmos.documents.azure.com:443/
spring.ai.vectorstore.cosmosdb.key = ${AZURE_VECTORSTORE_API_KEY}
spring.ai.vectorstore.cosmosdb.databaseName = spring-ai
spring.ai.vectorstore.cosmosdb.containerName = spring-ai
spring.ai.vectorstore.cosmosdb.partitionKeyPath = /id

Unfortunately, there are still some issues with the Azure CosmosDB support in the Spring AI M6 milestone version. I see that they were fixed in the SNAPSHOT version. So, if you want to test it by yourself, you will have to switch from milestones to snapshots.

<properties>
  <java.version>21</java.version>
  <spring-ai.version>1.0.0-SNAPSHOT</spring-ai.version>
</properties>
  
<repositories>
  <repository>
    <name>Central Portal Snapshots</name>
    <id>central-portal-snapshots</id>
    <url>https://central.sonatype.com/repository/maven-snapshots/</url>
    <releases>
      <enabled>false</enabled>
    </releases>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
  <repository>
    <id>spring-snapshots</id>
    <name>Spring Snapshots</name>
    <url>https://repo.spring.io/snapshot</url>
    <releases>
      <enabled>false</enabled>
    </releases>
    <snapshots>
      <enabled>true</enabled>
    </snapshots>
  </repository>
</repositories>

Run and Test the Application

After those changes, you can start the application with the following command:

mvn spring-boot:run -Pazure-ai -Dspring-boot.run.profiles=azure-ai

Once the application is running, you can test the following @RestController that offers RAG functionality. The GET /stocks/load-data endpoint obtains stock prices of given companies and puts them in the vector store. The GET /stocks/v2/most-growth-trend uses the RetrievalAugmentationAdvisor instance to retrieve the most suitable data and include it in the user query.

@RestController
@RequestMapping("/stocks")
public class StockController {

    private final ObjectMapper mapper = new ObjectMapper();
    private final static Logger LOG = LoggerFactory.getLogger(StockController.class);
    private final ChatClient chatClient;
    private final RewriteQueryTransformer.Builder rqtBuilder;
    private final RestTemplate restTemplate;
    private final VectorStore store;

    @Value("${STOCK_API_KEY:none}")
    private String apiKey;

    public StockController(ChatClient.Builder chatClientBuilder,
                           VectorStore store,
                           RestTemplate restTemplate) {
        this.chatClient = chatClientBuilder
                .defaultAdvisors(new SimpleLoggerAdvisor())
                .build();
        this.rqtBuilder = RewriteQueryTransformer.builder()
                .chatClientBuilder(chatClientBuilder);
        this.store = store;
        this.restTemplate = restTemplate;
    }

    @GetMapping("/load-data")
    void load() throws JsonProcessingException {
        final List<String> companies = List.of("AAPL", "MSFT", "GOOG", "AMZN", "META", "NVDA");
        for (String company : companies) {
            StockData data = restTemplate.getForObject("https://api.twelvedata.com/time_series?symbol={0}&interval=1day&outputsize=10&apikey={1}",
                    StockData.class,
                    company,
                    apiKey);
            if (data != null && data.getValues() != null) {
                var list = data.getValues().stream().map(DailyStockData::getClose).toList();
                var doc = Document.builder()
                        .id(company)
                        .text(mapper.writeValueAsString(new Stock(company, list)))
                        .build();
                store.add(List.of(doc));
                LOG.info("Document added: {}", company);
            }
        }
    }

    @RequestMapping("/v2/most-growth-trend")
    String getBestTrendV2() {
        PromptTemplate pt = new PromptTemplate("""
                {query}.
                Which {target} is the most % growth?
                The 0 element in the prices table is the latest price, while the last element is the oldest price.
                """);

        Prompt p = pt.create(Map.of("query", "Find the most growth trends", "target", "share"));

        Advisor retrievalAugmentationAdvisor = RetrievalAugmentationAdvisor.builder()
                .documentRetriever(VectorStoreDocumentRetriever.builder()
                        .similarityThreshold(0.7)
                        .topK(3)
                        .vectorStore(store)
                        .build())
                .queryTransformers(rqtBuilder.promptTemplate(pt).build())
                .build();

        return this.chatClient.prompt(p)
                .advisors(retrievalAugmentationAdvisor)
                .call()
                .content();
    }

}

Finally, you can call the following two endpoints.

$ curl http://localhost:8080/stocks/load-data
$ curl http://localhost:8080/stocks/v2/most-growth-trend

Final Thoughts

This exercise shows how to modify an existing Spring Boot AI application to integrate it with the Azure OpenAI service. It also gives a recipe on how to include Azure CosmosDB as a vector store for RAG scenarios and similarity searches.

The post Spring AI with Azure OpenAI appeared first on Piotr's TechBlog.

This is the seventh part of my series of articles about Spring Boot and AI. It is worth reading the following posts before proceeding with the current one. Please pay special attention to the last article from the list about the tool calling feature since we will implement it in our sample client and server apps using MCP.

1. https://piotrminkowski.com/2025/01/28/getting-started-with-spring-ai-and-chat-model: The first tutorial introduces the Spring AI project and its support for building applications based on chat models like OpenAI or Mistral AI.
2. https://piotrminkowski.com/2025/01/30/getting-started-with-spring-ai-function-calling: The second tutorial shows Spring AI support for Java function calling with the OpenAI chat model.
3. https://piotrminkowski.com/2025/02/24/using-rag-and-vector-store-with-spring-ai: The third tutorial shows Spring AI support for RAG (Retrieval Augmented Generation) and vector store.
4. https://piotrminkowski.com/2025/03/04/spring-ai-with-multimodality-and-images: The fourth tutorial shows Spring AI support for a multimodality feature and image generation
5. https://piotrminkowski.com/2025/03/10/using-ollama-with-spring-ai: The fifth tutorial shows Spring AI support for interactions with AI models run with Ollama
6. https://piotrminkowski.com/2025/03/13/tool-calling-with-spring-ai: The sixth tutorial show Spring AI for the Tool Calling feature.

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Motivation for MCP with Spring AI

MCP introduces an interesting concept for applications interacting with AI models. With MCP the application can provide specific tools/functions for several other services, which need to use data exposed by that application. Additionally, it can expose prompt templates and resources. Thanks to that, we don’t need to implement AI tools/functions inside every client service but integrate them with the application that exposes tools over MCP.

The best way to analyze the MCP concept is through an example. Let’s consider an application that connects to a database and exposes data through REST endpoints. If we want to use that data in our AI application we should implement and register AI tools that retrieve data by connecting such the REST endpoints. So, each client-side application that needs data from the source service would have to implement its own set of AI tools locally. Here comes the MCP concept. The source service defines and exposes AI tools/functions in the standardized form. All other apps that need to provide data to AI models can load and use a predefined set of tools.

The following diagram illustrates our scenario. Two Spring Boot applications act as MCP servers. They connect to the database and use Spring AI MCP Server support to expose @Tool methods to the MCP client-side app. The client-side app communicates with the OpenAI model. It includes the tools exposed by the server-side apps in the user query to the AI model. The person-mcp-service app provides @Tool methods for searching persons in the database table. The account-mcp-service is doing the same for the persons’ accounts.

Build MCP Server App with Spring AI

Let’s begin with the implementation of applications that act as MCP servers. They both run and use an in-memory H2 database. To interact with a database we include the Spring Data JPA module. Spring AI allows us to switch between three transport types: STDIO, Spring MVC, and Spring WebFlux. MCP Server with Spring WebFlux supports Server-Sent Events (SSE) and an optional STDIO transport. Here’s a list of required Maven dependencies:

<dependencies>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-mcp-server-webflux-spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-jpa</artifactId>
  </dependency>
  <dependency>
    <groupId>com.h2database</groupId>
    <artifactId>h2</artifactId>
    <scope>runtime</scope>
  </dependency>
</dependencies>

Create the Person MCP Server

Here’s an @Entity class for interacting with the person table:

@Entity
public class Person {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String firstName;
    private String lastName;
    private int age;
    private String nationality;
    @Enumerated(EnumType.STRING)
    private Gender gender;
    
    // ... getters and setters
    
}

The Spring Data Repository interface contains a single method for searching persons by their nationality:

public interface PersonRepository extends CrudRepository<Person, Long> {
    List<Person> findByNationality(String nationality);
}

The PersonTools @Service bean contains two Spring AI @Tool methods. It injects the PersonRepository bean to interact with the H2 database. The getPersonById method returns a single person with a specific ID field, while the getPersonsByNationality returns a list of all persons with a given nationality.

@Service
public class PersonTools {

    private PersonRepository personRepository;

    public PersonTools(PersonRepository personRepository) {
        this.personRepository = personRepository;
    }

    @Tool(description = "Find person by ID")
    public Person getPersonById(
            @ToolParam(description = "Person ID") Long id) {
        return personRepository.findById(id).orElse(null);
    }

    @Tool(description = "Find all persons by nationality")
    public List<Person> getPersonsByNationality(
            @ToolParam(description = "Nationality") String nationality) {
        return personRepository.findByNationality(nationality);
    }
    
}

Once we define @Tool methods, we must register them within the Spring AI MCP server. We can use the ToolCallbackProvider bean for that. More specifically, the MethodToolCallbackProvider class provides a builder that creates an instance of the ToolCallbackProvider class with a list of references to objects with @Tool methods.

@SpringBootApplication
public class PersonMCPServer {

    public static void main(String[] args) {
        SpringApplication.run(PersonMCPServer.class, args);
    }

    @Bean
    public ToolCallbackProvider tools(PersonTools personTools) {
        return MethodToolCallbackProvider.builder()
                .toolObjects(personTools)
                .build();
    }

}

Finally, we must provide configuration properties. The person-mcp-server app will listen on the 8060 port. We should also set the name and version of the MCP server embedded in our application.

spring:
  ai:
    mcp:
      server:
        name: person-mcp-server
        version: 1.0.0
  jpa:
    database-platform: H2
    generate-ddl: true
    hibernate:
      ddl-auto: create-drop

logging.level.org.springframework.ai: DEBUG

server.port: 8060

That’s all. We can start the application.

$ cd spring-ai-mcp/person-mcp-service
$ mvn spring-boot:run

Create the Account MCP Server

Then, we will do very similar things in the second application that acts as an MCP server. Here’s the @Entity class for interacting with the account table:

@Entity
public class Account {

    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long id;
    private String number;
    private int balance;
    private Long personId;
    
    // ... getters and setters
    
}

The Spring Data Repository interface contains a single method for searching accounts belonging to a given person:

public interface AccountRepository extends CrudRepository<Account, Long> {
    List<Account> findByPersonId(Long personId);
}

The AccountTools @Service bean contains a single Spring AI @Tool method. It injects the AccountRepository bean to interact with the H2 database. The getAccountsByPersonId method returns a list of accounts owned by the person with a specified ID field value.

@Service
public class AccountTools {

    private AccountRepository accountRepository;

    public AccountTools(AccountRepository accountRepository) {
        this.accountRepository = accountRepository;
    }

    @Tool(description = "Find all accounts by person ID")
    public List<Account> getAccountsByPersonId(
            @ToolParam(description = "Person ID") Long personId) {
        return accountRepository.findByPersonId(personId);
    }
}

Of course, the account-mcp-server application will use ToolCallbackProvider to register @Tool methods defined inside the AccountTools class.

@SpringBootApplication
public class AccountMCPService {

    public static void main(String[] args) {
        SpringApplication.run(AccountMCPService.class, args);
    }

    @Bean
    public ToolCallbackProvider tools(AccountTools accountTools) {
        return MethodToolCallbackProvider.builder()
                .toolObjects(accountTools)
                .build();
    }
    
}

Here are the application configuration properties. The account-mcp-server app will listen on the 8040 port.

spring:
  ai:
    mcp:
      server:
        name: account-mcp-server
        version: 1.0.0
  jpa:
    database-platform: H2
    generate-ddl: true
    hibernate:
      ddl-auto: create-drop

logging.level.org.springframework.ai: DEBUG

server.port: 8040

Let’s run the second server-side app:

$ cd spring-ai-mcp/account-mcp-service
$ mvn spring-boot:run

Once we start the application, we should see the log indicating how many tools were registered in the MCP server.

Build MCP Client App with Spring AI

Implementation

We will create a single client-side application. However, we can imagine an architecture where many applications consume tools exposed by one MCP server. Our application interacts with the OpenAI chat model, so we must include the Spring AI OpenAI starter. For the MCP Client starter, we can choose between two dependencies: Standard MCP client and Spring WebFlux client. Spring team recommends using the WebFlux-based SSE connection with the spring-ai-mcp-client-webflux-spring-boot-starter. Finally, we include the Spring Web starter to expose the REST endpoint. However, you can use Spring WebFlux starter to expose them reactively.

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-mcp-client-webflux-spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
  </dependency>
</dependencies>

Our MCP client connects with two MCP servers. We must provide the following connection settings in the application.yml file.

spring.ai.mcp.client.sse.connections:
  person-mcp-server:
    url: http://localhost:8060
  account-mcp-server:
    url: http://localhost:8040

Our sample Spring Boot application contains to @RestControllers, which expose HTTP endpoints. The PersonController class defines two endpoints for searching and counting persons by nationality. The MCP Client Boot Starter automatically configures tool callbacks that integrate with Spring AI’s tool execution framework. Thanks to that we can use the ToolCallbackProvider instance to provide default tools to the ChatClient bean. Then, we can perform the standard steps to interact with the AI model with Spring AI ChatClient. However, the client will use tools exposed by both sample MCP servers.

@RestController
@RequestMapping("/persons")
public class PersonController {

    private final static Logger LOG = LoggerFactory
        .getLogger(PersonController.class);
    private final ChatClient chatClient;

    public PersonController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultTools(tools)
                .build();
    }

    @GetMapping("/nationality/{nationality}")
    String findByNationality(@PathVariable String nationality) {

        PromptTemplate pt = new PromptTemplate("""
                Find persons with {nationality} nationality.
                """);
        Prompt p = pt.create(Map.of("nationality", nationality));
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

    @GetMapping("/count-by-nationality/{nationality}")
    String countByNationality(@PathVariable String nationality) {
        PromptTemplate pt = new PromptTemplate("""
                How many persons come from {nationality} ?
                """);
        Prompt p = pt.create(Map.of("nationality", nationality));
        return this.chatClient.prompt(p)
                .call()
                .content();
    }
}

Let’s switch to the second @RestController. The AccountController class defines two endpoints for searching accounts by person ID. The GET /accounts/count-by-person-id/{personId} returns the number of accounts belonging to a given person. The GET /accounts/balance-by-person-id/{personId} is slightly more complex. It counts the total balance in all person’s accounts. However, it must also return the person’s name and nationality, which means that it must call the getPersonById tool method exposed by the person-mcp-server app after calling the tool for searching accounts by person ID.

@RestController
@RequestMapping("/accounts")
public class AccountController {

    private final static Logger LOG = LoggerFactory.getLogger(PersonController.class);
    private final ChatClient chatClient;

    public AccountController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools) {
        this.chatClient = chatClientBuilder
                .defaultTools(tools)
                .build();
    }

    @GetMapping("/count-by-person-id/{personId}")
    String countByPersonId(@PathVariable String personId) {
        PromptTemplate pt = new PromptTemplate("""
                How many accounts has person with {personId} ID ?
                """);
        Prompt p = pt.create(Map.of("personId", personId));
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

    @GetMapping("/balance-by-person-id/{personId}")
    String balanceByPersonId(@PathVariable String personId) {
        PromptTemplate pt = new PromptTemplate("""
                How many accounts has person with {personId} ID ?
                Return person name, nationality and a total balance on his/her accounts.
                """);
        Prompt p = pt.create(Map.of("personId", personId));
        return this.chatClient.prompt(p)
                .call()
                .content();
    }

}

Running the Application

Before starting the client-side app we must export the OpenAI token as the SPRING_AI_OPENAI_API_KEY environment variable.

export SPRING_AI_OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Then go to the sample-client directory and run the app with the following command:

$ cd spring-ai-mcp/sample-client
$ mvn spring-boot:run

Once we start the application, we can switch to the logs. As you see, the sample-client app receives responses with tools from both person-mcp-server and account-mcp-server apps.

Testing MCP with Spring Boot

Both server-side applications load data from the import.sql scripts on startup. Spring Data JPA automatically imports data from such scripts. Our MCP client application listens on the 8080 port. Let’s call the first endpoint to get a list of persons from Germany:

curl http://localhost:8080/persons/nationality/Germany

Here’s the response from the OpenAI model:

We can also call the endpoint that counts the number with a given nationality.

curl http://localhost:8080/persons/count-by-nationality/Germany

As the final test, we can call the GET /accounts/balance-by-person-id/{personId} endpoint that interacts with tools exposed by both MCP server-side apps. It requires an AI model to combine data from person and account sources.

Exposing Prompts with MCP

We can also expose prompts and resources with the Spring AI MCP server support. To register and expose prompts we need to define the list of SyncPromptRegistration objects. It contains the name of the prompt, a list of input arguments, and a text content.

@SpringBootApplication
public class PersonMCPServer {

    public static void main(String[] args) {
        SpringApplication.run(PersonMCPServer.class, args);
    }

    @Bean
    public ToolCallbackProvider tools(PersonTools personTools) {
        return MethodToolCallbackProvider.builder()
                .toolObjects(personTools)
                .build();
    }

    @Bean
    public List<McpServerFeatures.SyncPromptRegistration> prompts() {
        var prompt = new McpSchema.Prompt("persons-by-nationality", "Get persons by nationality",
                List.of(new McpSchema.PromptArgument("nationality", "Person nationality", true)));

        var promptRegistration = new McpServerFeatures.SyncPromptRegistration(prompt, getPromptRequest -> {
            String argument = (String) getPromptRequest.arguments().get("nationality");
            var userMessage = new McpSchema.PromptMessage(McpSchema.Role.USER,
                    new McpSchema.TextContent("How many persons come from " + argument + " ?"));
            return new McpSchema.GetPromptResult("Count persons by nationality", List.of(userMessage));
        });

        return List.of(promptRegistration);
    }
}

After startup, the application prints information about a list of registered prompts in the logs.

There is no built-in Spring AI support for loading prompts using the MCP client. However, Spring AI MCP support is under active development so we may expect some new features soon. For now, Spring AI provides the auto-configured instance of McpSyncClient. We can use it to search the prompt in the list of prompts received from the server. Then, we can prepare the PromptTemplate instance using the registered content and create the Prompt by filling the template with the input parameters.

@RestController
@RequestMapping("/persons")
public class PersonController {

    private final static Logger LOG = LoggerFactory
        .getLogger(PersonController.class);
    private final ChatClient chatClient;
    private final List<McpSyncClient> mcpSyncClients;

    public PersonController(ChatClient.Builder chatClientBuilder,
                            ToolCallbackProvider tools,
                            List<McpSyncClient> mcpSyncClients) {
        this.chatClient = chatClientBuilder
                .defaultTools(tools)
                .build();
        this.mcpSyncClients = mcpSyncClients;
    }

    // ... other endpoints
    
    @GetMapping("/count-by-nationality-from-client/{nationality}")
    String countByNationalityFromClient(@PathVariable String nationality) {
        return this.chatClient
                .prompt(loadPromptByName("persons-by-nationality", nationality))
                .call()
                .content();
    }

    Prompt loadPromptByName(String name, String nationality) {
        McpSchema.GetPromptRequest r = new McpSchema
            .GetPromptRequest(name, Map.of("nationality", nationality));
        var client = mcpSyncClients.stream()
                .filter(c -> c.getServerInfo().name().equals("person-mcp-server"))
                .findFirst();
        if (client.isPresent()) {
            var content = (McpSchema.TextContent) client.get() 
                .getPrompt(r)
                .messages()
                .getFirst()
                .content();
            PromptTemplate pt = new PromptTemplate(content.text());
            Prompt p = pt.create(Map.of("nationality", nationality));
            LOG.info("Prompt: {}", p);
            return p;
        } else return null;
    }
}

Final Thoughts

Model Context Protocol is an important initiative in the AI world. It allows us to avoid reinventing the wheel for each new data source. A unified protocol streamlines integration, minimizing development time and complexity. As businesses expand their AI toolsets, MCP enables seamless connectivity across multiple systems without the burden of excessive custom code. Spring AI introduced the initial version of MCP support recently. It seems promising. With Spring AI Client and Server starters, we may implement a distributed architecture, where several different apps use the AI tools exposed by a single service.

The post Using Model Context Protocol (MCP) with Spring AI appeared first on Piotr's TechBlog.

This is the sixth part of my series of articles about Spring Boot and AI. It is worth reading the following posts before proceeding with the current one. Please pay special attention to the second article. I will refer to it often in this article.

1. https://piotrminkowski.com/2025/01/28/getting-started-with-spring-ai-and-chat-model: The first tutorial introduces the Spring AI project and its support for building applications based on chat models like OpenAI or Mistral AI.
2. https://piotrminkowski.com/2025/01/30/getting-started-with-spring-ai-function-calling: The second tutorial shows Spring AI support for Java function calling with the OpenAI chat model.
3. https://piotrminkowski.com/2025/02/24/using-rag-and-vector-store-with-spring-ai: The third tutorial shows Spring AI support for RAG (Retrieval Augmented Generation) and vector store.
4. https://piotrminkowski.com/2025/03/04/spring-ai-with-multimodality-and-images: The fourth tutorial shows Spring AI support for a multimodality feature and image generation
5. https://piotrminkowski.com/2025/03/10/using-ollama-with-spring-ai: The fifth tutorial shows Spring AI supports for interactions with AI models run with Ollama

Source Code

Feel free to use my source code if you’d like to try it out yourself. To do that, you must clone my sample GitHub repository. Then you should only follow my instructions.

Motivation for Tool Calling in Spring AI

The tool calling feature helps us solve a common AI model challenge related to internal or live data sources. If we want to augment a model with such data our applications must allow it to interact with a set of APIs or tools. In our case, the internal database (H2) contains information about the structure of our stock wallet. The sample Spring Boot application asks an AI model about the total value of the wallet based on daily stock prices or the highest value for the last few days. The model must retrieve the structure of our stock wallet and the latest stock prices. We will do the same exercise as for a function calling feature. It will be enhanced with additional scenarios I’ll describe later.

Use the Calling Tools Feature in Spring AI

Create WalletTools

Let’s begin with the WalletTools implementation, which is responsible for interaction with a database. We can compare it to the previous implementation based on Spring functions available in the pl.piomin.services.functions.stock.WalletService class. It defines a single method annotated with @Tool. The important element is the right description that must inform the model what that method does. The method returns the number of shares for each company in our portfolio retrieved from the database through the Spring Data @Repository.

public class WalletTools {

    private WalletRepository walletRepository;

    public WalletTools(WalletRepository walletRepository) {
        this.walletRepository = walletRepository;
    }

    @Tool(description = "Number of shares for each company in my wallet")
    public List<Share> getNumberOfShares() {
        return (List<Share>) walletRepository.findAll();
    }
}

We can register the WalletTools class as a Spring @Bean in the application main class.