BYO OpenAI-compatible model#

You can bring your own model from an OpenAI API-compatible LLM provider. The example integrates with Cohere AI.

1. Save your API key from the OpenAI-compatible provider as an environment variable. For example, navigate to the Cohere AI dashboard.

   export PROVIDER_API_KEY=Dgs...

2. Create a Kubernetes secret that stores your API key. Make sure to create the secret in the same namespace as you plan to create your agent, such as kagent.

   kubectl create secret generic kagent-my-provider -n kagent --from-literal PROVIDER_API_KEY=$PROVIDER_API_KEY

3. Create a ModelConfig resource.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: my-provider-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: PROVIDER_API_KEY
     model: command-a-03-2025
     provider: OpenAI
     openAI:
       baseUrl: "https://api.cohere.ai/compatibility/v1"
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   apiKeySecret	The name of the Kubernetes secret that stores your API key.
   apiKeySecretKey	The key in the secret that stores your API key.
   model	The OpenAI API-compatible model to use. For more information about the model, consult your LLM provider's documentation. For example, you might use command-a-03-2025 for Cohere AI.
   provider	To use an OpenAI API-compatible model, set the provider to OpenAI.
   openAI	Additional provider details. For available settings, consult your LLM provider's documentation. At the least, you must configure the baseUrl setting to point to the endpoint of your LLM provider.
   baseUrl	The base URL of your LLM provider. Note that the LLM provider might have a special base URL for OpenAI compatibility, such as "https://api.cohere.ai/compatibility/v1" for Cohere AI.

Good job! You added a model to kagent. Next, you can create or update an agent to use this model.