Building a Java API connecting to LLMs with Spring AI and Ollama local models

Introduction

In the rapidly evolving world of AI, developers often need to integrate multiple AI providers into their applications. Whether you're using local models with Ollama, cloud services like OpenAI, or planning to add Anthropic or Google's Gemini, having a unified interface to manage these providers is crucial.

In this tutorial, we'll build a flexible, extensible AI backend using Spring Boot and Spring AI that can seamlessly switch between different AI providers. We'll implement a clean architecture that makes it easy to add new providers without changing existing code.

What We'll Build

We're going to create a REST API that:

• Supports multiple AI providers through a unified interface

• Allows dynamic provider and model selection per request

• Implements a registry pattern for provider management

• Provides proper error handling and validation

• Uses Spring AI for simplified AI integration

Here's what our architecture will look like:

Prerequisites

Before we begin, make sure you have:

• Java 21 or higher installed

• Maven installed

• Ollama installed and running (for local AI models)

• Your favorite IDE (IntelliJ IDEA, VS Code, etc.)

Step 1: Project Setup

Let's start by creating a new Spring Boot project. You can use Spring Initializr or create it manually.

1.1 Create the Project Structure

mkdir ai-backends-java
cd ai-backends-java

1.2 Create the pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" 
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
         https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.5.6</version>
        <relativePath/>
    </parent>

    <groupId>com.aibackends</groupId>
    <artifactId>ai</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>ai</name>
    <description>AIBackends Java</description>

    <properties>
        <java.version>21</java.version>
        <spring-ai.version>1.0.2</spring-ai.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-starter-model-ollama</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>
</project>

1.3 Create the Main Application Class

Create the directory structure and main class:

mkdir -p src/main/java/com/aibackends/ai

// src/main/java/com/aibackends/ai/AiApplication.java
package com.aibackends.ai;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;

@SpringBootApplication
public class AiApplication {
    public static void main(String[] args) {
        SpringApplication.run(AiApplication.class, args);
    }
}

Step 2: Define the Provider Architecture

Now let's create the core architecture that will allow us to support multiple AI providers.

2.1 Create the Provider Interface

First, we'll define an interface that all AI providers must implement:

// src/main/java/com/aibackends/ai/provider/ChatProvider.java
package com.aibackends.ai.provider;

/**
 * Interface for AI chat providers
 */
public interface ChatProvider {

    /**
     * Get a chat response from the AI provider
     * 
     * @param message The user's message
     * @param model The model to use (provider-specific)
     * @return The AI's response
     */
    String getChatResponse(String message, String model);

    /**
     * Get the provider type
     * 
     * @return The provider type enum
     */
    ProviderType getProviderType();

    /**
     * Check if the provider supports a specific model
     * 
     * @param model The model name to check
     * @return true if the model is supported
     */
    boolean supportsModel(String model);

    /**
     * Get the default model for this provider
     * 
     * @return The default model name
     */
    String getDefaultModel();
}

2.2 Create the Provider Type Enum

This enum will represent all supported providers:

// src/main/java/com/aibackends/ai/provider/ProviderType.java
package com.aibackends.ai.provider;

/**
 * Enum representing supported AI providers
 */
public enum ProviderType {
    OLLAMA("ollama"),
    ANTHROPIC("anthropic"),
    GEMINI("gemini");

    private final String value;

    ProviderType(String value) {
        this.value = value;
    }

    public String getValue() {
        return value;
    }

    public static ProviderType fromValue(String value) {
        for (ProviderType type : ProviderType.values()) {
            if (type.value.equalsIgnoreCase(value)) {
                return type;
            }
        }
        throw new IllegalArgumentException("Unknown provider type: " + value);
    }
}

2.3 Create the Provider Registry

The registry will manage all available providers and allow us to retrieve them dynamically:

// src/main/java/com/aibackends/ai/provider/ChatProviderRegistry.java
package com.aibackends.ai.provider;

import org.springframework.stereotype.Component;
import java.util.HashMap;
import java.util.List;
import java.util.Map;
import java.util.Optional;

@Component
public class ChatProviderRegistry {

    private final Map<ProviderType, ChatProvider> providers = new HashMap<>();

    public ChatProviderRegistry(List<ChatProvider> chatProviders) {
        // Register all available providers
        for (ChatProvider provider : chatProviders) {
            providers.put(provider.getProviderType(), provider);
        }
    }

    /**
     * Get a chat provider by type
     * 
     * @param providerType The provider type
     * @return The chat provider
     * @throws IllegalArgumentException if provider not found
     */
    public ChatProvider getProvider(ProviderType providerType) {
        return Optional.ofNullable(providers.get(providerType))
                .orElseThrow(() -> new IllegalArgumentException(
                        "Provider not available: " + providerType));
    }

    /**
     * Get a chat provider by string value
     * 
     * @param provider The provider name
     * @return The chat provider
     */
    public ChatProvider getProvider(String provider) {
        ProviderType providerType = ProviderType.fromValue(provider);
        return getProvider(providerType);
    }

    /**
     * Check if a provider is available
     * 
     * @param providerType The provider type
     * @return true if available
     */
    public boolean isProviderAvailable(ProviderType providerType) {
        return providers.containsKey(providerType);
    }

    /**
     * Get all available provider types
     * 
     * @return List of available provider types
     */
    public List<ProviderType> getAvailableProviders() {
        return providers.keySet().stream().toList();
    }
}

Step 3: Implement the Ollama Provider

Now let's implement our first AI provider - Ollama, which runs AI models locally.

3.1 Create the Ollama Service

// src/main/java/com/aibackends/ai/service/OllamaChatService.java
package com.aibackends.ai.service;

import com.aibackends.ai.provider.ChatProvider;
import com.aibackends.ai.provider.ProviderType;
import org.springframework.ai.chat.client.ChatClient;
import org.springframework.ai.ollama.OllamaChatModel;
import org.springframework.ai.ollama.api.OllamaOptions;
import org.springframework.stereotype.Service;

import java.util.List;

@Service
public class OllamaChatService implements ChatProvider {

    private final OllamaChatModel ollamaChatModel;
    private final List<String> supportedModels = List.of(
            "llama3.2", "llama3.1", "llama3", "llama2",
            "mistral", "mixtral", "codellama", "gemma",
            "phi3", "qwen2.5", "deepseek-coder-v2"
    );

    public OllamaChatService(OllamaChatModel ollamaChatModel) {
        this.ollamaChatModel = ollamaChatModel;
    }

    @Override
    public String getChatResponse(String message, String model) {
        // Set the model in options
        String modelToUse = model != null ? model : getDefaultModel();
        OllamaOptions options = OllamaOptions.builder()
                .model(modelToUse)
                .build();

        // Create a new ChatClient with the specified model
        var chatClient = ChatClient.builder(ollamaChatModel)
                .defaultOptions(options)
                .build();

        return chatClient.prompt()
                .user(message)
                .call()
                .content();
    }

    @Override
    public ProviderType getProviderType() {
        return ProviderType.OLLAMA;
    }

    @Override
    public boolean supportsModel(String model) {
        return supportedModels.stream()
                .anyMatch(m -> m.equalsIgnoreCase(model));
    }

    @Override
    public String getDefaultModel() {
        return "llama3.2";
    }
}

3.2 Create Configuration

// src/main/java/com/aibackends/ai/config/OllamaConfig.java
package com.aibackends.ai.config;

import org.springframework.context.annotation.Configuration;

@Configuration
public class OllamaConfig {
    // Spring AI auto-configuration handles the Ollama beans
    // No manual configuration needed when using spring-ai-starter-model-ollama
}

Step 4: Create the REST API

Now let's create the REST controller that will expose our AI services.

4.1 Create the Controller

// src/main/java/com/aibackends/ai/controller/AIController.java
package com.aibackends.ai.controller;

import com.aibackends.ai.provider.ChatProvider;
import com.aibackends.ai.provider.ChatProviderRegistry;
import com.aibackends.ai.provider.ProviderType;
import org.springframework.http.ResponseEntity;
import org.springframework.web.bind.annotation.*;

import java.util.List;

@RestController
@RequestMapping("/api")
public class AIController {

    private final ChatProviderRegistry providerRegistry;

    public AIController(ChatProviderRegistry providerRegistry) {
        this.providerRegistry = providerRegistry;
    }

    @PostMapping("/chat")
    public ResponseEntity<?> chat(@RequestBody ChatRequest request) {
        try {
            // Validate request
            if (request.message() == null || request.message().isBlank()) {
                return ResponseEntity.badRequest()
                        .body(new ErrorResponse("Message cannot be empty"));
            }

            if (request.provider() == null || request.provider().isBlank()) {
                return ResponseEntity.badRequest()
                        .body(new ErrorResponse("Provider must be specified"));
            }

            // Get the provider
            ChatProvider chatProvider = providerRegistry.getProvider(request.provider());

            // Validate model if specified
            if (request.model() != null && !request.model().isBlank() 
                    && !chatProvider.supportsModel(request.model())) {
                return ResponseEntity.badRequest()
                        .body(new ErrorResponse("Model '" + request.model() + 
                                "' is not supported by provider '" + request.provider() + "'"));
            }

            // Get the response
            String response = chatProvider.getChatResponse(
                    request.message(), 
                    request.model()
            );

            return ResponseEntity.ok(new ChatResponse(
                    response,
                    request.provider(),
                    request.model() != null ? request.model() : chatProvider.getDefaultModel()
            ));

        } catch (IllegalArgumentException e) {
            return ResponseEntity.badRequest()
                    .body(new ErrorResponse(e.getMessage()));
        } catch (Exception e) {
            return ResponseEntity.internalServerError()
                    .body(new ErrorResponse("Internal server error: " + e.getMessage()));
        }
    }

    @GetMapping("/providers")
    public ResponseEntity<ProvidersResponse> getProviders() {
        List<ProviderInfo> providers = providerRegistry.getAvailableProviders().stream()
                .map(providerType -> {
                    ChatProvider provider = providerRegistry.getProvider(providerType);
                    return new ProviderInfo(
                            providerType.getValue(),
                            provider.getDefaultModel()
                    );
                })
                .toList();

        return ResponseEntity.ok(new ProvidersResponse(providers));
    }

    @GetMapping("/providers/{provider}/models")
    public ResponseEntity<?> getProviderModels(@PathVariable String provider) {
        try {
            ChatProvider chatProvider = providerRegistry.getProvider(provider);

            // For now, return a basic response. In a real implementation,
            // each provider would have a method to list available models
            return ResponseEntity.ok(new ModelsResponse(
                    provider,
                    List.of(chatProvider.getDefaultModel()),
                    chatProvider.getDefaultModel()
            ));

        } catch (IllegalArgumentException e) {
            return ResponseEntity.badRequest()
                    .body(new ErrorResponse(e.getMessage()));
        }
    }

    // Request/Response DTOs
    public record ChatRequest(
            String message,
            String provider,
            String model
    ) {}

    public record ChatResponse(
            String response,
            String provider,
            String model
    ) {}

    public record ErrorResponse(String error) {}

    public record ProvidersResponse(List<ProviderInfo> providers) {}

    public record ProviderInfo(
            String name,
            String defaultModel
    ) {}

    public record ModelsResponse(
            String provider,
            List<String> models,
            String defaultModel
    ) {}
}

Step 5: Configure the Application

Create the application configuration file:

# src/main/resources/application.properties
server.port=8085

spring.application.name=ai-backends

# Ollama configuration
spring.ai.ollama.base-url=http://localhost:11434
spring.ai.ollama.chat.model=llama3.2

# Logging
logging.level.com.aibackends=DEBUG

Step 6: Test the Application

6.1 Start Ollama

First, make sure Ollama is running and has a model installed:

# Install Ollama (if not already installed)
# Visit https://ollama.ai for installation instructions

# Pull a model
ollama pull llama3.2

# Start Ollama (usually starts automatically)
ollama serve

6.2 Run the Application

./mvnw spring-boot:run

6.3 Test the Endpoints

List Available Providers

curl http://localhost:8085/api/providers | jq .

Response:

{
  "providers": [
    {
      "name": "ollama",
      "defaultModel": "llama3.2"
    }
  ]
}

Send a Chat Request

curl -X POST http://localhost:8085/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "message": "Hello! What is 2 + 2?",
    "provider": "ollama",
    "model": "llama3.2"
  }' | jq .

Response:

{
  "response": "The answer to 2 + 2 is 4.",
  "provider": "ollama",
  "model": "llama3.2"
}

Step 7: Adding New Providers

The beauty of this architecture is how easy it is to add new providers. Let's see how you would add OpenAI support:

7.1 Add the Dependency

Add to your pom.xml:

<dependency>
    <groupId>org.springframework.ai</groupId>
    <artifactId>spring-ai-starter-model-openai</artifactId>
</dependency>

7.2 Create the Provider Implementation

@Service
@ConditionalOnProperty(name = "spring.ai.openai.api-key")
public class OpenAIChatService implements ChatProvider {

    private final OpenAiChatModel openAiChatModel;

    public OpenAIChatService(OpenAiChatModel openAiChatModel) {
        this.openAiChatModel = openAiChatModel;
    }

    @Override
    public String getChatResponse(String message, String model) {
        // Implementation similar to Ollama
    }

    @Override
    public ProviderType getProviderType() {
        return ProviderType.OPENAI;
    }

    // ... other methods
}

7.3 Add to Provider Type Enum

public enum ProviderType {
    OLLAMA("ollama"),
    OPENAI("openai"),  // Add this
    ANTHROPIC("anthropic"),
    GEMINI("gemini");
    // ... rest of the enum
}

7.4 Configure in application.properties

# OpenAI configuration
spring.ai.openai.api-key=your-api-key-here
spring.ai.openai.chat.options.model=gpt-3.5-turbo

That's it! The provider will automatically be registered and available through the API.

Advanced Features

Error Handling

Our implementation includes comprehensive error handling:

• Validation for empty messages

• Provider validation

• Model validation

• Graceful handling of provider errors

Model Selection

Each request can specify a different model:

{
  "message": "Write a poem",
  "provider": "ollama",
  "model": "mistral"
}

Provider Discovery

The /api/providers endpoint allows clients to discover available providers dynamically.

Best Practices

1. Interface Segregation: The ChatProvider interface is focused and specific

2. Dependency Injection: Spring manages all dependencies automatically

3. Error Handling: All errors are handled gracefully with appropriate HTTP status codes

4. Extensibility: New providers can be added without modifying existing code

5. Configuration: Each provider can be configured independently

Conclusion

We've built a flexible, extensible AI backend that can work with multiple AI providers. The architecture we've implemented makes it easy to:

• Add new AI providers without changing existing code

• Switch between providers dynamically

• Handle errors gracefully

• Validate requests properly

• Discover available providers and models

This approach gives you the flexibility to use local models for development and privacy-sensitive applications while being able to switch to cloud providers for production or when more powerful models are needed.

Happy coding! 🚀