> For the complete documentation index, see [llms.txt](/docs/llms.txt). Markdown versions of all docs pages are available by appending .md to any docs URL.

# Lightweight OTel stack

Deploy a small observability stack based on OpenTelemetry (OTel) that collects traces and metrics from both kagent and Agent Substrate. The stack includes the following components:

- **Traces**: Distributed tracing with [Jaeger](https://github.com/jaegertracing/jaeger).
- **Metrics**: Time-series metrics collection with [Prometheus](https://github.com/prometheus/prometheus).
- **Collection**: Telemetry collection and routing with the [OpenTelemetry Collector](https://github.com/open-telemetry/opentelemetry-collector).
- **Logs**: No logging backend. The collector prints the logs that it receives to its own output.

This stack follows the setup that Agent Substrate uses for its own local testing, and it needs less memory than the [OTel stack](https://kagent.dev/docs/kagent/1.x/observability/otel-stack/) with Grafana.

## About the stack

The stack has no logging backend. The collector prints the logs that it receives to its own output, which is enough to confirm that logs arrive. To store and query logs, such as the Actor state changes that Agent Substrate records, use the [OTel stack](https://kagent.dev/docs/kagent/1.x/observability/otel-stack/), which adds Loki.

```mermaid
flowchart LR
    subgraph sources["Your cluster"]
        kagent["kagent controller<br/>and agent runtimes"]
        substrate["Agent Substrate"]
    end
    collector["OTel collector"]
    subgraph backends["Backends"]
        jaeger["Jaeger"]
        prom["Prometheus"]
    end
    kagent -->|traces, logs| collector
    substrate -->|traces, logs, metrics| collector
    kagent -->|metrics scrape| prom
    collector --> jaeger
    collector -->|metrics scrape| prom
    classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px
    classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px
    class sources,backends boundary
    class kagent,substrate,collector,jaeger,prom inner
```

For what kagent and Agent Substrate each send, see [About the stack](https://kagent.dev/docs/kagent/1.x/observability/otel-stack#about-the-stack) in the OTel stack guide.

## Before you begin

1. [Install kagent](https://kagent.dev/docs/kagent/1.x/setup/installation/).
2. [Create your first agent](https://kagent.dev/docs/kagent/1.x/get-started/your-first-agent/), so that you have an Agent to send requests to. That guide also installs the kagent CLI. The steps on this page need the 1.0.0-alpha17 CLI, because the CLIs of other releases, newer ones included, do not have the Session commands that these steps use. To check your version, run `kagent version`.
3. Install [`jq`](https://jqlang.org/download/), to read the Session ID and revision out of the CLI's JSON output.

## Install Jaeger

Install [Jaeger](https://www.jaegertracing.io/) in all-in-one mode, which stores traces in memory and does not need other components.

```bash
helm upgrade --install jaeger jaeger \
  --repo https://jaegertracing.github.io/helm-charts \
  --version 4.4.7 \
  --namespace telemetry --create-namespace \
  --values - <<EOF
provisionDataStore:
  cassandra: false
allInOne:
  enabled: true
storage:
  type: memory
agent:
  enabled: false
collector:
  enabled: false
query:
  enabled: false
EOF
```

## Install Prometheus

Install the [Prometheus](https://github.com/prometheus-community/helm-charts/tree/main/charts/prometheus) server on its own, without the exporters and Alertmanager that the chart includes by default. The values configure two scrape jobs, one for the metrics that the collector serves and one for the kagent controller.

1. Install the chart.
   
   ```bash
   helm upgrade --install prometheus prometheus \
     --repo https://prometheus-community.github.io/helm-charts \
     --version 29.33.0 \
     --namespace telemetry \
     --values - <<EOF
   alertmanager:
     enabled: false
   kube-state-metrics:
     enabled: false
   prometheus-node-exporter:
     enabled: false
   prometheus-pushgateway:
     enabled: false
   server:
     persistentVolume:
       enabled: false
   extraScrapeConfigs: |
     - job_name: otel-collector
       scrape_interval: 15s
       static_configs:
       - targets: ["otel-collector.telemetry.svc.cluster.local:8889"]
     - job_name: kagent-controller
       scrape_interval: 15s
       scheme: https
       authorization:
         credentials_file: /var/run/secrets/kubernetes.io/serviceaccount/token
       tls_config:
         insecure_skip_verify: true
       static_configs:
       - targets: ["kagent-controller-metrics.kagent.svc.cluster.local:8443"]
   EOF
   ```
   
   The kagent controller serves its metrics over HTTPS with a self-signed certificate, and it accepts a scrape only from a ServiceAccount that holds its metrics reader role. The `kagent-controller` job therefore skips certificate verification and sends the Prometheus ServiceAccount token.
   
   The `kagent-controller` target reads `DOWN` until you turn on the controller metrics in [Send kagent telemetry to the collector](#send-kagent-telemetry-to-the-collector), because the controller metrics Service does not exist before then.
2. Verify that Prometheus is running.
   
   ```bash
   kubectl get pods -n telemetry -l app.kubernetes.io/name=prometheus
   ```

## Install the OTel collector

Install a collector that receives OTLP on ports `4317` for gRPC and `4318` for HTTP. The collector sends traces to Jaeger, serves metrics on port `8889`, and prints logs to its own output. The configuration uses the `contrib` collector image, because the Prometheus exporter is not part of the core image. For production, start from the reference Collector values in the kagent source at `examples/observability/collector-values.yaml`. The smaller collector in this guide omits the persistent queue and content-scrubbing processors.

```bash
helm upgrade --install otel-collector opentelemetry-collector \
  --repo https://open-telemetry.github.io/opentelemetry-helm-charts \
  --version 0.172.0 \
  --namespace telemetry \
  --values - <<EOF
fullnameOverride: otel-collector
mode: deployment
image:
  repository: otel/opentelemetry-collector-contrib
ports:
  metrics-export:
    enabled: true
    containerPort: 8889
    servicePort: 8889
    protocol: TCP
config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
  processors:
    batch: {}
    delta_to_cumulative: {}
    transform/harness_native_genai:
      error_mode: ignore
      trace_statements:
        - context: span
          conditions:
            - resource.attributes["kagent.genai.producer"] == "runtime" and not span.attributes["llm_request.context"] == "standalone"
          statements:
            - delete_matching_keys(span.attributes, "^gen_ai\\.(operation\\.name|system|request\\.model|response\\.finish_reasons|usage\\..*)$")
  connectors:
    signal_to_metrics:
      spans:
        - name: gen_ai.invoke_agent.duration
          description: The duration of one agent invocation, derived from its span.
          unit: s
          conditions:
            - span.attributes["gen_ai.operation.name"] == "invoke_agent" and instrumentation_scope.name != "github.com/kagent-dev/kagent/go/adk/pkg/a2a/server" and instrumentation_scope.name != "kagent.core.a2a"
          histogram:
            buckets: [0.5, 1, 2, 5, 10, 30, 60, 120, 300]
            value: Seconds(span.end_time - span.start_time)
          attributes:
            - key: gen_ai.agent.name
              optional: true
            - key: gen_ai.request.model
              optional: true
            - key: error.type
              optional: true
          include_resource_attributes:
            - key: service.name
            - key: service.namespace
              optional: true
            - key: service.version
              optional: true
            - key: kagent.runtime
              optional: true
            - key: gen_ai.main_agent.id
              optional: true
  exporters:
    debug:
      verbosity: basic
    otlp_grpc/jaeger:
      endpoint: jaeger.telemetry.svc.cluster.local:4317
      tls:
        insecure: true
    prometheus:
      endpoint: 0.0.0.0:8889
      resource_to_telemetry_conversion:
        enabled: true
  service:
    pipelines:
      traces:
        receivers: [otlp]
        processors: [transform/harness_native_genai, batch]
        exporters: [signal_to_metrics, otlp_grpc/jaeger]
      logs:
        receivers: [otlp]
        processors: [batch]
        exporters: [debug]
      metrics:
        receivers: [otlp, signal_to_metrics]
        processors: [delta_to_cumulative, batch]
        exporters: [prometheus]
EOF
```

The `signal_to_metrics` connector derives the `gen_ai.invoke_agent.duration` metric from the `invoke_agent` spans that the `kagent` runtime emits. That runtime does not record the metric itself.

The connector reads the spans before any sampling in the collector. Its conditions skip the two A2A server instrumentation scopes. A span from those scopes carries the `invoke_agent` operation only when that layer also records the metric, so skipping the scopes prevents a double count.

The connector emits delta histograms, and the Prometheus exporter accepts only cumulative ones. The `delta_to_cumulative` processor converts them. Without it, the exporter logs `Dropped misaligned histogram datapoint` and keeps one point per series, so the duration queries return almost nothing.

## Send kagent telemetry to the collector

Turn on the kagent trace, log, and metrics exporters, and point them at the collector. Also turn on the controller's metrics endpoint, and allow Prometheus to scrape it.

1. Save the current revision of your Agent. A later step uses it to tell when kagent recompiles the Agent with the new settings. The command first waits for any recompile that is still in progress, such as one from an earlier Helm upgrade, so that it saves a finished revision.
   
   ```bash
   for i in $(seq 1 60); do
     REVISIONS=$(kubectl get agent my-first-agent -n kagent \
       -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}')
     [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break
     sleep 5
   done
   export OLD_REVISION=${REVISIONS#* }
   echo "Current revision: $OLD_REVISION"
   ```
2. Upgrade the kagent Helm release to export traces, logs, and metrics to the collector, and to serve controller metrics.
   
   ```bash
   helm upgrade kagent \
     oci://ghcr.io/kagent-dev/kagent/helm/kagent \
     --version 1.0.0-alpha17 \
     --namespace kagent \
     --reuse-values \
     --values - <<EOF
   config:
     telemetry:
       traces:
         endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
         protocol: grpc
       logs:
         endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
         protocol: grpc
       metrics:
         endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
         protocol: grpc
     runtimeDefaults:
       telemetry:
         traces:
           endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
           protocol: grpc
         logs:
           endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
           protocol: grpc
         metrics:
           endpoint: http://otel-collector.telemetry.svc.cluster.local:4317
           protocol: grpc
   controller:
     metrics:
       enabled: true
   EOF
   ```
3. Allow Prometheus to scrape the controller. Bind the metrics reader role that the kagent chart creates to the Prometheus ServiceAccount.
   
   ```bash
   kubectl create clusterrolebinding kagent-metrics-reader-prometheus \
     --clusterrole=kagent-metrics-reader \
     --serviceaccount=telemetry:prometheus-server
   ```
4. Wait for the controller to roll out.
   
   ```bash
   kubectl rollout status deployment/kagent-controller -n kagent --timeout=300s
   ```
5. Wait for kagent to recompile the Agent. The controller rebuilds each Agent after the controller restarts. A Session that you create before the rebuild finishes starts from the previous revision, without the new settings. The following command prints `Recompiled` when the new revision is ready.
   
   ```bash
   for i in $(seq 1 60); do
     [ "$(kubectl get agent my-first-agent -n kagent \
       -o jsonpath='{.status.latestSuccessfulRevision}')" != "$OLD_REVISION" ] \
       && echo "Recompiled" && break
     sleep 5
   done
   ```
   
   If the command finishes without printing `Recompiled`, the upgrade did not change the settings that kagent compiles into the Agent. Either the settings were already in place, or the chart did not recognize the `otel` keys. Helm accepts a key that a chart does not define without an error, so check that you upgraded to version 1.0.0-alpha17 of the chart, which uses the keys on this page.

## Send Agent Substrate telemetry to the collector

Upgrade the Agent Substrate Helm release. A single `otel.endpoint` setting turns on traces, logs, and metrics for every Agent Substrate component. Agent Substrate keeps 1% of its traces by default, so the command keeps every trace instead. Lower `samplingRatio` again for production.

```bash
helm upgrade substrate \
  oci://ghcr.io/kagent-dev/substrate/helm/substrate \
  --version 0.4.0-alpha1 \
  --namespace ate-system \
  --reuse-values \
  --set otel.endpoint=http://otel-collector.telemetry.svc.cluster.local:4317 \
  --set otel.traces.samplingRatio=1.0 \
  --wait --timeout 10m
```

## Send a request

Create a Session that picks up the new telemetry settings, and send it a request to produce traces and metrics.

1. Create a new Session. A Session keeps the runtime configuration that it was created with, so only a new Session exports traces.
   
   ```bash
   kagent agent session create --agent my-first-agent
   ```
2. Confirm that the Session runs the Agent's current revision. The command waits until kagent finishes compiling the Agent, then compares that revision with the one that the Session started from. If the command prints `Outdated`, the Session was created from an earlier revision, and exports without the new settings. Create another Session, and run the command again.
   
   ```bash
   for i in $(seq 1 60); do
     REVISIONS=$(kubectl get agent my-first-agent -n kagent \
       -o jsonpath='{.status.desiredRevision} {.status.latestSuccessfulRevision}')
     [ "${REVISIONS% *}" = "${REVISIONS#* }" ] && break
     sleep 5
   done
   SESSION_REVISION=$(kagent agent session get $SESSION_ID -o json | jq -r '.session.preparedRevision')
   [ "$SESSION_REVISION" = "${REVISIONS#* }" ] && echo "Current" || echo "Outdated"
   ```
3. Send a request to produce telemetry.
   
   ```bash
   export SESSION_ID=$(kagent agent session create --agent my-first-agent -o json | jq -r '.session.id')
   kagent agent invoke --session $SESSION_ID --task "What is 2+2?"
   ```

## Review the telemetry

Check each signal in turn: traces in Jaeger, metrics in Prometheus, and logs in the collector's own output.

1. Review the traces in Jaeger.
   
   1. Forward the Jaeger query port, and leave the command running.
      
      ```bash
      kubectl port-forward -n telemetry svc/jaeger 16686:16686
      ```
   2. In your browser, open Jaeger at [http://localhost:16686](http://localhost:16686).
   3. From the **Service** list, select `my-first-agent`, and click **Find Traces**. To see Agent Substrate's own work, select one of its services instead: `ateapi`, `atelet`, `atecontroller`, or `ateom-gvisor`. For what each service reports, see [Agent Substrate traces](https://kagent.dev/docs/kagent/1.x/observability/tracing#agent-substrate-traces).
2. Review the metrics in Prometheus.
   
   1. Forward the Prometheus port, and leave the command running.
      
      ```bash
      kubectl port-forward -n telemetry svc/prometheus-server 9090:80
      ```
   2. In your browser, open Prometheus at [http://localhost:9090/targets](http://localhost:9090/targets), and confirm that the `otel-collector` and `kagent-controller` targets are `UP`.
   3. Open the **Query** page, and run a query such as `ate_workerpool_ready_workers`. For more metrics, see [Metrics](https://kagent.dev/docs/kagent/1.x/observability/metrics/).
3. Check that the collector receives logs. Each line reports a batch of log records from one source.
   
   ```bash
   kubectl logs -n telemetry deploy/otel-collector | grep '"otelcol.signal": "logs"'
   ```

## Clean up

1. Turn off telemetry export in kagent and Agent Substrate.
   
   ```bash
   helm upgrade kagent \
     oci://ghcr.io/kagent-dev/kagent/helm/kagent \
     --version 1.0.0-alpha17 \
     --namespace kagent --reuse-values \
     --set config.telemetry=null \
     --set config.runtimeDefaults.telemetry=null \
     --set controller.metrics.enabled=false
   helm upgrade substrate \
     oci://ghcr.io/kagent-dev/substrate/helm/substrate \
     --version 0.4.0-alpha1 \
     --namespace ate-system --reuse-values \
     --set otel.endpoint="" \
     --set otel.traces.samplingRatio=0.01
   ```
2. Remove the role binding, the collector, the backends, and the `telemetry` namespace.
   
   ```bash
   kubectl delete clusterrolebinding kagent-metrics-reader-prometheus
   helm uninstall otel-collector prometheus jaeger -n telemetry
   kubectl delete namespace telemetry
   ```

## Next steps

[Tracing](https://kagent.dev/docs/kagent/1.x/observability/tracing/)

Read the spans and attributes of an agent request.

[Metrics](https://kagent.dev/docs/kagent/1.x/observability/metrics/)

Review the metrics that kagent and Agent Substrate report.

[OTel stack](https://kagent.dev/docs/kagent/1.x/observability/otel-stack/)

Add Loki and Grafana to store logs and view every signal in one place.