For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

Configure kagent to use any provider that serves the OpenAI API, including a self-hosted gateway in front of your own models.

Many providers serve the OpenAI API rather than an API of their own. To use one, set provider: OpenAI and point openAI.baseUrl at the provider’s endpoint. This is the same mechanism behind the xAI and Amazon Bedrock OpenAI-compatible paths.

Create the ModelConfig

The following example uses Cohere, which serves an OpenAI-compatible endpoint.

1. Save the API key from your provider as an environment variable.

   export PROVIDER_API_KEY=<your_api_key>

2. Create a Kubernetes Secret that stores the API key. Create it in the same namespace as the AgentTemplates that use it, such as kagent.

   kubectl create secret generic kagent-my-provider -n kagent --from-literal PROVIDER_API_KEY=$PROVIDER_API_KEY

3. Create a ModelConfig that points at your provider’s endpoint.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha3
   kind: ModelConfig
   metadata:
     name: my-provider-model-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: PROVIDER_API_KEY
     model: command-a-03-2025
     provider: OpenAI
     openAI:
       baseUrl: https://api.cohere.ai/compatibility/v1
   EOF

   Field	Description
   apiKeySecret	The name of the Kubernetes Secret that stores the API key, in the same namespace as this ModelConfig.
   apiKeySecretKey	The key within that Secret that holds the API key.
   model	The model identifier that your provider routes on. Consult your provider’s documentation.
   provider	The provider to use, OpenAI.
   openAI.baseUrl	Your provider’s OpenAI-compatible endpoint. Providers often serve this at a dedicated path, such as /compatibility/v1.

For the rest of the settings that the openAI block accepts, see OpenAI. Not every compatible provider honors every setting, so check your provider’s documentation before setting one. For every openAI field, including its type, default, and validation rules, see the API reference.

Self-hosted vLLM behind a gateway

A common self-hosted pattern puts an OpenAI-compatible gateway such as Bifrost or LiteLLM in front of a vLLM server, so that requests travel from kagent to the gateway to vLLM. Configure the gateway as an OpenAI-compatible provider in the same way, with two extra things to get right.

Enable tool calling in vLLM

kagent sends a tools array with tool_choice: "auto" on every request. kagent’s runtime registers a built-in ask_user tool on every agent, so that array is sent even when you bind no tools yourself. Launch vLLM with automatic tool choice enabled, or every agent turn fails.

vllm serve Qwen/Qwen2.5-7B-Instruct \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

The correct --tool-call-parser depends on your model family. Qwen2.5 uses hermes and Llama 3.1 uses llama3_json. Parser names change across vLLM releases, so check the vLLM tool calling docs for the current name for your model.

Use the gateway’s model identifier

Set model to the identifier that your gateway routes on, which is often provider-prefixed and can differ from the bare model name that vLLM serves internally. Point openAI.baseUrl at the gateway. LiteLLM defaults to port 4000, and Bifrost to 8080.

spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: PROVIDER_API_KEY
  model: vllm/Qwen/Qwen2.5-7B-Instruct
  provider: OpenAI
  openAI:
    baseUrl: http://litellm.kagent.svc.cluster.local:4000/v1

Troubleshooting a 400 from the provider

When every agent message fails with a generic provider API error (status 400), the most common cause is a vLLM server started without --enable-auto-tool-choice and a matching --tool-call-parser. Because kagent always sends tool_choice: "auto", vLLM rejects the request until automatic tool choice is enabled. Restart vLLM with both flags and try again.

TLS

A provider on your own network may present a certificate that the agent does not already trust. The tls block adjusts how the agent verifies it.

Turning verification off does not mount anything, so it does compile.

spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: PROVIDER_API_KEY
  model: command-a-03-2025
  provider: OpenAI
  openAI:
    baseUrl: https://llm.internal.example.com/v1
  tls:
    disableVerify: true

Use the ModelConfig

Reference the ModelConfig by name from an AgentTemplate in the same namespace.

spec:
  modelConfig:
    name: my-provider-model-config

Next steps