Configuring LLM providers

OpenAI gpt-4o is configured as a default LLM provider for kagent (the default-model-config in the kagent namespace). The ModelConfig resource is used for configuring models.

The following providers are currently supported:

• OpenAI
• Azure OpenAI
• Anthropic (note that there's an active issue with tool calling)

Configuring Anthropic

1. Create a Kubernetes Secret that stores the API key, replace <your_api_key> with an actual API key:

export ANTHROPIC_API_KEY=<your_api_key>
kubectl create secret generic kagent-anthropic -n kagent --from-literal ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY

2. Create a ModelConfig resource that references the secret and key name, and specify the Anthropic model you want to use:

apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: claude-model-config
  namespace: kagent
spec:
  apiKeySecretName: kagent-anthropic
  apiKeySecretKey: ANTHROPIC_API_KEY
  model: claude-3-sonnet-20240229
  provider: Anthropic
  anthropic: {}

3. Apply the above resource to the cluster.

Once the resource is applied, you can select the model from the Model dropdown in the UI when creating or updating agents.

Configuring Azure OpenAI

1. Create a Kubernetes Secret that stores the API key, replace <your_api_key> with an actual API key:

export AZURE_OPENAI_API_KEY=<your_api_key>
kubectl create secret generic kagent-azureopenai -n kagent --from-literal AZURE_OPENAI_API_KEY=$AZURE_OPENAI_API_KEY

2. Create a ModelConfig resource that references the secret and key name, and specify the additional information that's required for the Azure OpenAI - that's the deployment name, version and the Azure AD token. You can get these values from Azure.

apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: azureopenai-model-config
  namespace: kagent
spec:
  apiKeySecretName: kagent-azureopenai
  apiKeySecretKey: AZURE_OPENAI_API_KEY
  model: gpt-4o-mini
  provider: AzureOpenAI
  azureOpenAI:
    azureEndpoint: "https://{yourendpointname}.openai.azure.com/"
    apiVersion: "2025-03-01-preview"
    azureDeployment: "gpt-4o-mini"
    azureAdToken: <azure_ad_token_value>

3. Apply the above resource to the cluster.

Once the resource is applied, you can select the model from the Model dropdown in the UI when creating or updating agents.

Configuring Ollama

Ollama allows you to run LLMs locally on your computer or in a Kubernetes cluster. Configuring Ollama in kagent follows the same pattern as for other providers.

Let's give an example of how to run Ollama on a Kubernetes cluster:

1. Create a namespace for Ollama deployment and service:

kubectl create ns ollama

2. Create the deployment and service:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ollama
  namespace: ollama
spec:
  selector:
    matchLabels:
      name: ollama
  template:
    metadata:
      labels:
        name: ollama
    spec:
      containers:
      - name: ollama
        image: ollama/ollama:latest
        ports:
        - name: http
          containerPort: 11434
          protocol: TCP
---
apiVersion: v1
kind: Service
metadata:
  name: ollama
  namespace: ollama
spec:
  type: ClusterIP
  selector:
    name: ollama
  ports:
  - port: 80
    name: http
    targetPort: http
    protocol: TCP

You can run kubectl get pod -n ollama and wait until the pod has started.

Once the pod has started, you can port-forward to the Ollama service and use ollama run [model-name] to download/run the model. You can download Ollama binary here.

As kagent relies on calling tools, sure you're using a model that allows function calling.

Let's assume we've downloaded the llama3 model, you can then use the following ModelConfig to configure the model:

apiVersion: kagent.dev/v1alpha1
kind: ModelConfig
metadata:
  name: llama3-model-config
  namespace: kagent
spec:
  apiKeySecretKey: OPENAI_API_KEY
  apiKeySecretName: kagent-openai
  model: llama3
  provider: Ollama
  ollama:
    host: http://ollama.ollama.svc.cluster.local