For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Bring your own OpenAI-compatible model to kagent.

You can bring your own model from an OpenAI API-compatible LLM provider. The example integrates with Cohere AI.

1. Save your API key from the OpenAI-compatible provider as an environment variable. For example, navigate to the Cohere AI dashboard.

   export PROVIDER_API_KEY=Dgs...

2. Create a Kubernetes secret that stores your API key. Make sure to create the secret in the same namespace as you plan to create your agent, such as kagent.

   kubectl create secret generic kagent-my-provider -n kagent --from-literal PROVIDER_API_KEY=$PROVIDER_API_KEY

3. Create a ModelConfig resource.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: my-provider-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: ${PROVIDER_API_KEY}
     model: command-a-03-2025
     provider: OpenAI
     openAI:
       baseUrl: "https://api.cohere.ai/compatibility/v1"
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   apiKeySecret	The name of the Kubernetes secret that stores your API key.
   apiKeySecretKey	The key in the secret that stores your API key.
   model	The OpenAI API-compatible model to use. For more information about the model, consult your LLM provider’s documentation. For example, you might use command-a-03-2025 for Cohere AI.
   provider	To use an OpenAI API-compatible model, set the provider to OpenAI.
   openAI	Additional provider details. For available settings, consult your LLM provider’s documentation. At the least, you must configure the baseUrl setting to point to the endpoint of your LLM provider.
   baseUrl	The base URL of your LLM provider. Note that the LLM provider might have a special base URL for OpenAI compatibility, such as "https://api.cohere.ai/compatibility/v1" for Cohere AI.

Good job! You added a model to kagent. Next, you can create or update an agent to use this model.

Self-hosted vLLM behind an OpenAI-compatible gateway

A common self-hosted pattern places an OpenAI-compatible gateway such as Bifrost or LiteLLM in front of a vLLM server (kagent → gateway → vLLM). Configure the gateway as an OpenAI-compatible provider, exactly as shown above, with two extra things to get right.

Enable tool calling in vLLM

kagent sends a tools array with tool_choice: "auto" on every request. kagent’s runtime registers a built-in ask_user tool on every agent, so a tools array is always sent, even when you configure no tools yourself. The vLLM backend must be launched with automatic tool choice enabled, or every agent turn fails.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The correct --tool-call-parser depends on your model family. For example, Qwen2.5 uses hermes and Llama 3.1 uses llama3_json. Parser names change across vLLM releases, so check the vLLM tool calling docs for your model’s current parser name.

Use the gateway’s model identifier

Set spec.model to the identifier your gateway routes on (often provider-prefixed, such as vllm/Qwen/Qwen2.5-7B-Instruct), which can differ from the bare model name vLLM serves internally. Point openAI.baseUrl at the gateway (LiteLLM defaults to port 4000, Bifrost to 8080).

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: qwen-vllm-via-gateway
  namespace: kagent
spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: ${PROVIDER_API_KEY}
  provider: OpenAI
  model: vllm/Qwen/Qwen2.5-7B-Instruct
  openAI:
    baseUrl: http://litellm.kagent.svc.cluster.local:4000/v1

Troubleshooting: provider API error (status 400)

If every agent message fails with a generic status 400, the most common cause is that vLLM started without --enable-auto-tool-choice and a matching --tool-call-parser. Because kagent always sends tool_choice: "auto", vLLM rejects the request until automatic tool choice is enabled. Restart vLLM with the flags above and retry.

TLS Configuration

To secure communication to LLMs with your own custom certificates, configure the TLS CA details in the ModelConfig. Then, your agents communicate with the LLM with those custom certificates. This feature is useful for internal or company-managed LLM servers.

Note: TLS configuration only supports OpenAI-compatible providers.

Use Case: Custom CA Certificates

Configure the CA certificate of your LLM server in the ModelConfig resource.

1. Create a Secret with the relevant CA certificate in the same namespace as the ModelConfig.

   kubectl -n kagent create secret generic llm-certs \
     --from-file=ca.crt=ca.crt

2. Create a ModelConfig resource with TLS configuration that references the CA certificate Secret.

   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: internal-llm-model-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: ${PROVIDER_API_KEY}
     provider: OpenAI
     model: ${MODEL_NAME}
     openAI:
       baseUrl: ${COMPATIBLE_PROVIDER_URL}
     tls:
       caCertSecretRef: llm-certs
       caCertSecretKey: ca.crt

Use Case: Insecure Communication

Warning: Insecure communication is for demo purposes only. Do not use insecure communication in production environments.

For development or testing scenarios where you need to disable TLS verification, configure the ModelConfig to skip certificate verification.

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: internal-llm-model-config
  namespace: kagent
spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: ${PROVIDER_API_KEY}
  provider: OpenAI
  model: ${MODEL_NAME}
  openAI:
    baseUrl: ${COMPATIBLE_PROVIDER_URL}
  tls:
    disableVerify: true

TLS Configuration Settings

Review the following table to understand the TLS configuration options. For more information, see the API docs.