BYO OpenAI-compatible model#

You can bring your own model from an OpenAI API-compatible LLM provider. The example integrates with Cohere AI.

1. Save your API key from the OpenAI-compatible provider as an environment variable. For example, navigate to the Cohere AI dashboard.

   export PROVIDER_API_KEY=Dgs...

2. Create a Kubernetes secret that stores your API key. Make sure to create the secret in the same namespace as you plan to create your agent, such as kagent.

   kubectl create secret generic kagent-my-provider -n kagent --from-literal PROVIDER_API_KEY=$PROVIDER_API_KEY

3. Create a ModelConfig resource.

   kubectl apply -f - <<EOF
   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: my-provider-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: ${PROVIDER_API_KEY}
     model: command-a-03-2025
     provider: OpenAI
     openAI:
       baseUrl: "https://api.cohere.ai/compatibility/v1"
   EOF

   Review the following table to understand this configuration. For more information, see the API docs.

   Setting	Description
   apiKeySecret	The name of the Kubernetes secret that stores your API key.
   apiKeySecretKey	The key in the secret that stores your API key.
   model	The OpenAI API-compatible model to use. For more information about the model, consult your LLM provider's documentation. For example, you might use command-a-03-2025 for Cohere AI.
   provider	To use an OpenAI API-compatible model, set the provider to OpenAI.
   openAI	Additional provider details. For available settings, consult your LLM provider's documentation. At the least, you must configure the baseUrl setting to point to the endpoint of your LLM provider.
   baseUrl	The base URL of your LLM provider. Note that the LLM provider might have a special base URL for OpenAI compatibility, such as "https://api.cohere.ai/compatibility/v1" for Cohere AI.

Good job! You added a model to kagent. Next, you can create or update an agent to use this model.

TLS Configuration#

To secure communication to LLMs with your own custom certificates, configure the TLS CA details in the ModelConfig. Then, your agents communicate with the LLM with those custom certificates. This feature is useful for internal or company-managed LLM servers.

Note: TLS configuration only supports OpenAI-compatible providers.

Use Case: Custom CA Certificates#

Configure the CA certificate of your LLM server in the ModelConfig resource.

1. Create a Secret with the relevant CA certificate in the same namespace as the ModelConfig.

   kubectl -n kagent create secret generic llm-certs \
     --from-file=ca.crt=ca.crt

2. Create a ModelConfig resource with TLS configuration that references the CA certificate Secret.

   apiVersion: kagent.dev/v1alpha2
   kind: ModelConfig
   metadata:
     name: internal-llm-model-config
     namespace: kagent
   spec:
     apiKeySecret: kagent-my-provider
     apiKeySecretKey: ${PROVIDER_API_KEY}
     provider: OpenAI
     model: ${MODEL_NAME}
     openAI:
       baseUrl: ${COMPATIBLE_PROVIDER_URL}
     tls:
       caCertSecretRef: llm-certs
       caCertSecretKey: ca.crt

Use Case: Insecure Communication#

Warning: Insecure communication is for demo purposes only. Do not use insecure communication in production environments.

For development or testing scenarios where you need to disable TLS verification, configure the ModelConfig to skip certificate verification.

apiVersion: kagent.dev/v1alpha2
kind: ModelConfig
metadata:
  name: internal-llm-model-config
  namespace: kagent
spec:
  apiKeySecret: kagent-my-provider
  apiKeySecretKey: ${PROVIDER_API_KEY}
  provider: OpenAI
  model: ${MODEL_NAME}
  openAI:
    baseUrl: ${COMPATIBLE_PROVIDER_URL}
  tls:
    disableVerify: true

TLS Configuration Settings#

Review the following table to understand the TLS configuration options. For more information, see the API docs.