Tracing

Enable OpenTelemetry tracing for kagent, then read a trace that runs from the controller through to the Actor that executed your agent.

A trace records one agent request as a tree of timed spans, so you can see where a slow or failed request spent its time and which model and tool calls it made along the way. In kagent 1.0 a single request crosses two processes, the controller and the ActorActorThe sandboxed unit of compute, provided by Agent Substrate, that runs an AgentInstance's conversation loop. Every AgentInstance is backed by one.Learn more that runs the agent, and a trace ties both halves together.

About trace coverage

Tracing spans two processes, and the link between them is a W3C Trace Context header that the controller passes to the Actor. The following diagram traces one request through both.

    flowchart LR
    caller["Caller"]
    subgraph controllerproc["kagent controller"]
        grpc["gRPC API"]
        gateway["A2A gateway"]
    end
    subgraph actorproc["Actor"]
        runtime["Agent runtime"]
    end
    %% Cross-subgraph edges are declared outside every subgraph block, because
    %% mermaid assigns a node to the subgraph that first references it.
    caller --> grpc
    grpc --> gateway
    gateway -->|traceparent| runtime
    classDef boundary fill:#a78bfa26,stroke:#a78bfa,stroke-width:2px
    classDef inner fill:#80808033,stroke:#9ca3af,stroke-width:1px
    class controllerproc,actorproc boundary
    class grpc,gateway,runtime inner

A caller reaches the gRPC API on the kagent controller, which starts the trace. The controller hands the request to its A2A gateway, which opens an A2AA2AThe Agent-to-Agent protocol, which callers and other agents use to talk to an AgentInstance. The conversation's context identifier is the AgentInstance ID, so a second message on the same ID continues the same conversation.Learn more (Agent-to-Agent) connection to the AgentInstance’s Actor and injects a traceparent header into that call. The agent runtime inside the Actor reads the header and continues the same trace, so the model and tool spans it produces hang off the controller’s spans rather than starting a trace of their own.

Important

The controller passes its tracing configuration to the kagent, codex, and claude runtimes. Each of the three exports on its own instrumentation, so the span names in this page describe the kagent runtime and do not carry over to the other two. An agent on the byo runtime receives no tracing configuration, and its half of the trace is missing. For the available runtimes, see Choose a runtime.

Note

A byo image that implements OTel itself reads the exporter variables from the Harness spec.env, which the controller leaves alone for this runtime. Its spans still do not reach a collector inside the cluster, because kagent adds the collector to an Actor’s egress allowlist only for the runtimes it configures, and no field adds a host to that list by hand. For more information, see Networking and egress control.

Both processes report themselves as separate OpenTelemetry (OTel) services. A tracing backend uses these service names to group the spans.

The controller reports as kagent-controller in the kagent service namespace. Its spans also carry the pod, node, and namespace that the controller runs on.
Each agent runtime reports as its own service, named for the AgentTemplateAgentTemplateA Kubernetes custom resource defining what an agent does: its model, system prompt, tools, skills, and plugins. It runs only once a Harness accepts it.Learn more and HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: its runtime, workload image, WorkerPool and snapshot storage, and which AgentTemplates it accepts.Learn more pair it was compiled from, with hyphens replaced by underscores. The my-first-agent template on the my-first-harness Harness reports as my_first_agent_my_first_harness.

Note

A service per template and Harness pair is a change from kagent 0.x, where every agent reported under one kagent service. A backend that you filter by service now shows one entry for each pair, and adding an agent adds a service.

Spans

The kagent runtime creates the same spans for every agent, and most span names describe the operation rather than the agent. The invoke_agent span is the exception, because its name carries the service name of the agent that ran. To narrow a search to one agent, filter by service name rather than by span name. The following spans appear in nesting order, from the span that accepts the request down to the model and tool calls that serve it.

Span	When it is created
`POST /lf.a2a.v1.A2AService/SendMessage`	Once per request, as the root of the runtime’s half of the trace. The runtime creates it when it accepts the A2A call from the controller.
`invocation`	Once per request, as the parent of the agent’s own work.
`invoke_agent <agent>`	Once per request, named for the AgentTemplate and Harness pair that serves it.
`generate_content <model>`	Once per model call, named for the model that was called.
`execute_tool <tool>`	Once per tool call, named for the tool that was called.
`execute_tool (merged)`	Once per model turn that calls more than one tool, as the parent of that turn’s `execute_tool` spans. A turn that calls a single tool creates no merged span.

Correlation attributes

A trace tells you which request you are looking at through attributes on its spans, not through the span names. The runtime stamps the following four attributes onto its root span and copies them onto every descendant span. A search on any one of these attributes returns the whole subtree rather than a single span.

Attribute	Value
`gen_ai.task.id`	The A2A task ID, which identifies one turn of a conversation.
`gen_ai.conversation.id`	The A2A context ID, which identifies the conversation and is stable across its turns.
`kagent.app_name`	The AgentTemplate, as `<namespace>__NS__<name>` with hyphens replaced by underscores.
`kagent.user_id`	The authenticated caller, or `A2A_USER_<context-id>` for an unauthenticated one.

The runtime also adds each scalar value in the A2A message’s metadata as an a2a.message.metadata.<key> attribute, so a client can tag a request and search for it later. Unlike the four correlation attributes, these tags stay on the invocation span alone, so a search on one returns that span instead of the whole subtree.

Warning

Spans for a model call carry the full serialized request and response as the gcp.vertex.agent.llm_request and gcp.vertex.agent.llm_response attributes. Prompts and replies therefore reach your tracing backend. Payloads larger than 32 KiB are truncated to a prefix. To keep this content out of traces, set OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT to false in the Harness spec.env. Note that the same variable has the opposite default for audit logging, where content is withheld until you set it to true. For more information, see Audit prompts. The codex and claude runtimes do not read this variable. On those two runtimes, the otel.captureSensitiveContent Helm setting controls the same content, as described in the agent harness telemetry content settings.

Before you begin

Install kagent.
Create your first agent, so that you have an AgentInstanceAgentInstanceA running, conversational pairing of a Harness and an AgentTemplate. Unlike the two, it is not a Kubernetes resource: kagent's gRPC API creates it and its database tracks it.Learn more to send a request to.

Install Jaeger

Install a backend that accepts OpenTelemetry Protocol (OTLP) traces. The following steps install Jaeger in all-in-one mode, which stores traces in memory and needs no other components.

Create a jaeger.yaml configuration file.

cat << 'EOF' > jaeger.yaml
provisionDataStore:
  cassandra: false
allInOne:
  enabled: true
storage:
  type: memory
agent:
  enabled: false
collector:
  enabled: false
query:
  enabled: false
EOF

Install Jaeger.

helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm upgrade --install jaeger jaegertracing/jaeger \
  --namespace jaeger --create-namespace \
  --history-max 3 \
  --values jaeger.yaml \
  --version 4.4.7

Enable tracing

Tracing is off by default. Turning it on is a Helm change, because the controller reads its tracing configuration from the environment and passes it to the agent runtimes it starts.

Get your current Helm values for kagent.

helm get values kagent -n kagent -o yaml > values.yaml

Add the tracing settings to the values file, pointing the exporter at Jaeger.

otel:
  tracing:
    enabled: true
    exporter:
      otlp:
        endpoint: http://jaeger.jaeger.svc.cluster.local:4317
        protocol: grpc
        timeout: 15000
        insecure: true

Review the following table to understand this configuration.

Field	Description
`enabled`	Whether to export traces at all. Defaults to `false`.
`exporter.otlp.endpoint`	The OTLP endpoint to export to. Empty by default, which leaves the exporter on the OTel default of `localhost:4317`.
`exporter.otlp.protocol`	`grpc` or `http/protobuf`. Defaults to `grpc`, which matches the port `4317` in the example endpoint. Point `http/protobuf` at port `4318` instead.
`exporter.otlp.timeout`	The export timeout in milliseconds. Defaults to `15000`.
`exporter.otlp.insecure`	Whether to skip Transport Layer Security (TLS) for the exporter connection. Defaults to `true`.

Upgrade the kagent Helm release.

helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha1 \
  --namespace kagent \
  --values values.yaml

Create a new AgentInstance, so that its Actor starts from a runtime that has the tracing configuration.
```
kagent create agent-instance --harness my-first-harness --agent-template my-first-agent
```

Review a trace

Send a request to the AgentInstance to produce a trace.

export INSTANCE_ID=$(kagent get agent-instance -o json \
  | jq -r '[.agentInstances[] | select(.agentTemplate.name == "my-first-agent")] | sort_by(.createdAt) | last | .id')
kagent invoke --agent-instance $INSTANCE_ID --task "What is 2+2?"

Forward the Jaeger query port, and leave the command running.
```
kubectl port-forward -n jaeger svc/jaeger 16686:16686
```
In your browser, open the Jaeger user interface at http://localhost:16686.
From the Service dropdown, select my_first_agent_my_first_harness, the service that the AgentTemplate and Harness pair reports as. Selecting kagent-controller instead returns the same traces from the controller’s side.
Leave Operation on all, or select invocation to start from the agent’s own work rather than from the A2A call that carries it, and click Find Traces.
Click a trace to open it. The span tree shows the controller’s gRPC and gateway spans, followed by the runtime’s POST /lf.a2a.v1.A2AService/SendMessage span, and finally the invocation, invoke_agent, generate_content, and execute_tool spans.
To narrow a search to one conversation, put a correlation attribute in the Tags field, such as gen_ai.conversation.id=<context-id>.

Traces from a suspended Actor

Agent Substrate checkpointsCheckpointA durable pin on the snapshot that an AgentInstance most recently suspended to, and a record of how far its transcript had advanced. Not a new state: tagging copies the snapshot so that Agent Substrate does not collect it, and a second AgentInstance can be forked from it.Learn more an Actor as soon as the response body closes, which is sooner than a batching span exporter normally sends its buffer. Spans still in the buffer at that moment freeze inside the snapshotSnapshotThe stored state that an Actor suspends to, held in object storage. Resuming restores the Actor from its most recent snapshot, which is what makes suspending idle agents cheap.Learn more and reach the backend only when the session next resumes, or never at all for a conversation’s last message.

To avoid losing them, the controller sets KAGENT_PRE_RESPONSE_TRACE_FLUSH to true on the kagent and codex runtimes, and the runtime flushes its span buffer before each response completes. The flush waits up to three seconds, which you can change with KAGENT_TRACE_FLUSH_TIMEOUT_MS in the Harness spec.env. The claude runtime gets no such flush, so its spans arrive on its exporter’s own schedule and a conversation’s last turn can lose them.

This behavior allows a kagent trace to arrive promptly rather than on the exporter’s own schedule. To understand what suspension does to an Actor, see Suspend and resume.

Turn tracing off

Disable tracing in the kagent Helm release.

helm upgrade kagent \
  oci://ghcr.io/kagent-dev/kagent/helm/kagent \
  --version 1.0.0-alpha1 \
  --namespace kagent --reuse-values \
  --set otel.tracing.enabled=false

Create a new AgentInstance to pick up the change, because an existing Actor keeps the configuration it started with.

Remove Jaeger.

helm uninstall jaeger -n jaeger
kubectl delete namespace jaeger

Next steps

Audit promptsExport every prompt and reply as a log event for security and compliance review. Suspend and resumeLearn what happens to an Actor between the turns of a conversation.

Launch the UI Audit prompts