Agent Substrate

Watch an agent’s Actor suspend between turns, pin its state with a checkpoint, and fork that checkpoint into a second agent that continues the conversation.

Agent Substrate runs every agent as an Actor: a sandboxed unit of compute that holds a WorkerWorkerA pre-started, sandboxed pod that hosts at most one Actor at a time. Actors are multiplexed across a pool of Workers over time.Learn more only while a turn is in progress, and whose state you can pin and branch. This example follows one agent through all three behaviors.

The Actor that these steps follow is also the isolation boundary. Every Actor runs in its own gVisorgVisorA user-space kernel that isolates a workload from the host kernel by intercepting its system calls. kagent compiles every ActorTemplate to the gvisor sandbox class.Learn more sandbox rather than sharing one with its neighbors. This isolation allows a model to safely run tools and execute commands. For what the sandbox blocks, see Sandboxing.

Before you begin

Checkpoints and forksForkA second AgentInstance created from a checkpoint, continuing the conversation from the point that the checkpoint pinned. A fork inherits the checkpoint's revision, so later edits to the AgentTemplate do not change what it runs, and new turns append only to the fork, leaving the original's history untouched.Learn more have no kagent CLI commands yet, so this example calls CheckpointService with grpcurl. The steps also assume that you have already sent your agent at least one message, because a checkpoint needs a completed turn to pin.

Install kagent, and confirm that your installation sets controller.grpc.reflection=true. Reflection lets grpcurl discover the controller’s methods without a local copy of kagent’s protocol buffer definitions.
Create your first agent, then save the AgentInstance’s ID to an environment variable. To find the ID, run kagent get agent-instance to list your AgentInstances and copy the value from the ID column.
```
export INSTANCE_ID=<your-agent-instance-id>
```
Install grpcurl.
Port-forward the controller’s gRPC port, and leave the command running.
```
kubectl port-forward -n kagent svc/kagent-controller 8083:8083
```

Watch the Actor suspend between turns

List the Actors in your namespace’s atespaceAtespaceThe isolation boundary that an Actor belongs to, and the first half of its identity. Global-scoped in Agent Substrate, not a Kubernetes namespace.Learn more. An AgentInstance’s Actor name is formatted ai-<agent-instance-id>.
```
kubectl ate get actors --atespace kagent
```
Between turns, the Actor reports ACTOR_STATE_SUSPENDED and holds no Worker, so the ATEOM POD column reads <none> and the ATEOM IP column is blank. The VERSION column is the Actor record’s revision counter, which increases each time the record is updated. Example output:
```
ATESPACE   NAME                                      TEMPLATE                                              STATE                   ATEOM POD   ATEOM IP   VERSION   AGE
kagent     ai-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10   kagent/my-first-agent-my-first-harness-5f2b3c1a9e8d   ACTOR_STATE_SUSPENDED   <none>                 4         11m
```
Send the agent another message. Nothing in the command acknowledges that the Actor was suspended, because resuming is automatic.
```
kagent invoke --agent-instance $INSTANCE_ID --task "Summarize this conversation so far."
```
From a second terminal, list the Actors again while the turn is still running.
```
kubectl ate get actors --atespace kagent
```
The same Actor now reports ACTOR_STATE_RUNNING, names the Worker pod that it resumed onto, and carries a higher VERSION. Example output:
```
ATESPACE   NAME                                      TEMPLATE                                              STATE                 ATEOM POD                                ATEOM IP      VERSION   AGE
kagent     ai-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10   kagent/my-first-agent-my-first-harness-5f2b3c1a9e8d   ACTOR_STATE_RUNNING   kagent/kagent-default-7c9f8b6d54-x2n4p   10.244.1.37   6         12m
```
If the listing already reads ACTOR_STATE_SUSPENDED, the turn finished before the command ran. Repeat steps 2 and 3 to catch the Actor mid-turn. A turn is short, and the two transitions on either side of one, ACTOR_STATE_RESUMING and ACTOR_STATE_SUSPENDING, pass quickly enough that a single listing rarely catches them.

After the turn finishes, list the Actors again.

kubectl ate get actors --atespace kagent

The Actor is back to ACTOR_STATE_SUSPENDED and holds no Worker again, at a higher VERSION than the listing in step 1. The NAME and AGE columns confirm that this is the same Actor throughout, rather than a new one per turn. Example output:

ATESPACE   NAME                                      TEMPLATE                                              STATE                   ATEOM POD   ATEOM IP   VERSION   AGE
kagent     ai-0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10   kagent/my-first-agent-my-first-harness-5f2b3c1a9e8d   ACTOR_STATE_SUSPENDED   <none>                 8         13m

Check the AgentInstance while its Actor is suspended.

kagent get agent-instance

The AgentInstance reports READY, even though the Actor that runs it holds no Worker. Example output:

+--------------------------------------+----------------+------------------+-------+----------------------+
| ID                                   | AGENT TEMPLATE | HARNESS          | STATE | CREATED              |
+--------------------------------------+----------------+------------------+-------+----------------------+
| 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | my-first-harness | READY | 2026-08-31T15:02:10Z |
+--------------------------------------+----------------+------------------+-------+----------------------+

The AgentInstance stays READY throughout all steps. A suspended agent remains listed and readable because suspension is a property of the Actor underneath the conversation, not of the conversation itself. For the full cycle, see Suspend and resume.

Note

Two objects report state on this page, and each interface names its states differently. kubectl ate get actors reports the Actor’s state in full, such as ACTOR_STATE_SUSPENDED, because the command prints the Agent Substrate enum name. The kagent CLI trims the prefix from the AgentInstance’s state and prints READY, and the same value reaches you as AGENT_INSTANCE_STATE_READY in a grpcurl response. Checkpoints have no CLI command yet, so the next section calls the API directly and reads the checkpoint’s state in full, as CHECKPOINT_STATE_READY.

Pin the conversation with a checkpoint

Each suspend writes a snapshotSnapshotThe stored state that an Actor suspends to, held in object storage. Resuming restores the Actor from its most recent snapshot, which is what makes suspending idle agents cheap.Learn more, and Agent Substrate is free to collect that snapshot once a newer one supersedes it. A checkpointCheckpointA durable pin on the snapshot that an AgentInstance most recently suspended to, and a record of how far its transcript had advanced. Not a new state: tagging copies the snapshot so that Agent Substrate does not collect it, and a second AgentInstance can be forked from it.Learn more pins a snapshot so that you can come back to it.

Create a checkpoint. The checkpoint records the snapshot that it pinned and how far the transcriptTranscriptThe record of an AgentInstance's conversation, held server-side and append-only. It survives the Actor suspending between turns, and a resumed runtime cannot shrink it. had advanced. The requestId field is a required idempotency key of 1 to 128 characters, so reusing it returns the same checkpoint rather than creating a second one.
```
grpcurl -plaintext \
  -d '{"agentInstanceId":"'"$INSTANCE_ID"'","requestId":"'"$(uuidgen)"'"}' \
  localhost:8083 kagent.api.v1alpha1.CheckpointService/CreateCheckpoint
```
Example output:
```
{
  "checkpoint": {
    "id": "0198c3e2-8a41-7d05-b6c2-1f4e9a7b3c58",
    "agentInstanceId": "0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10",
    "headTaskId": "0198c3d9-b7e3-7a24-8f10-6c2d5e8a1b47",
    "historySequence": "4",
    "state": "CHECKPOINT_STATE_READY",
    "createdAt": "2026-08-31T15:12:44Z"
  }
}
```
Note
A checkpoint captures a turn boundary, so two conditions must hold: the AgentInstance must be READY with no lifecycle operation in flight, and at least one turn must have reached a quiescent state. A request that fails either one reports AgentInstance has no quiescent turn boundary. Send the request again once the turn finishes.
Save the checkpoint’s id to fork from it in the next section.
```
export CHECKPOINT_ID=<your-checkpoint-id>
```

List the checkpoints on the AgentInstance at any time. Omit limit for the default page of 50, up to a maximum of 100.

grpcurl -plaintext \
  -d '{"agentInstanceId":"'"$INSTANCE_ID"'","page":{"limit":50}}' \
  localhost:8083 kagent.api.v1alpha1.CheckpointService/ListCheckpoints

Underneath, the checkpoint attaches a TagTagAn Agent Substrate resource that gives one snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. It also acts as a retention pin: Agent Substrate does not collect a snapshot while a tag names it, and only the tag's visibility scope can change afterward.Learn more named checkpoint-<checkpoint-id> to the snapshot, and Agent Substrate does not collect a snapshot while a tag names it. You can see the tag by running kubectl ate get tags --atespace kagent.

Fork the conversation into a second agent

Forking creates a second AgentInstance from the pinned snapshot, running the revision that the checkpoint was taken on. The fork continues the conversation from the checkpoint: it inherits the transcriptTranscriptThe record of an AgentInstance's conversation, held server-side and append-only. It survives the Actor suspending between turns, and a resumed runtime cannot shrink it. up to that point, along with the Actor’s durable state.

Note

The two branches share everything up to the checkpoint and nothing after it. New turns append only to the AgentInstance that received them, so the original’s history and the checkpoint itself stay unchanged no matter what the fork goes on to do. Each branch runs its own Actor.

Fork the checkpoint.

grpcurl -plaintext \
  -d '{"checkpointId":"'"$CHECKPOINT_ID"'","requestId":"'"$(uuidgen)"'"}' \
  localhost:8083 kagent.api.v1alpha1.CheckpointService/ForkAgentInstance

The response carries a new AgentInstance with its own ID. Example output:

{
  "agentInstance": {
    "id": "0198c3e5-1d62-7f38-a904-8b3c7e2f5d16",
    "harness": {
      "namespace": "kagent",
      "name": "my-first-harness"
    },
    "agentTemplate": {
      "namespace": "kagent",
      "name": "my-first-agent"
    },
    "state": "AGENT_INSTANCE_STATE_READY",
    "contextId": "ce5a10b8-7789-4ba7-8395-e60a339de763"
  }
}

The contextId is the fork’s link to the conversation it inherited. It matches the source AgentInstance’s contextId, while the two id values differ, so the branches address one shared conversation from separate instances.

Save the fork’s ID, then ask it about a turn that happened before the checkpoint.
```
export FORK_ID=<your-fork-agent-instance-id>
kagent invoke --agent-instance $FORK_ID --task "What did I ask you first?"
```
The fork answers from the conversation it inherited. That is the difference between a fork and a new AgentInstance that happens to use the same AgentTemplate. Send the original a different question and the two diverge from here.

List your AgentInstances to verify that both appear as separate AgentInstances.

kagent get agent-instance

The two rows share an AgentTemplate and a Harness, and differ in their IDs and creation times. Each one has its own Actor, named ai-<agent-instance-id>. Example output:

+--------------------------------------+----------------+------------------+-------+----------------------+
| ID                                   | AGENT TEMPLATE | HARNESS          | STATE | CREATED              |
+--------------------------------------+----------------+------------------+-------+----------------------+
| 0198c3d7-4f2a-7b61-9c3e-5d8f7a2b4e10 | my-first-agent | my-first-harness | READY | 2026-08-31T15:02:10Z |
| 0198c3e5-1d62-7f38-a904-8b3c7e2f5d16 | my-first-agent | my-first-harness | READY | 2026-08-31T15:14:02Z |
+--------------------------------------+----------------+------------------+-------+----------------------+

A fork runs the compiled revisionRevisionThe compiled, immutable output of one Harness and AgentTemplate pairing, identified by a content digest. An AgentInstance runs the revision it was created from for its whole life, so editing either resource affects only instances created afterward. that its checkpoint was taken on, not whatever revision the AgentTemplate resolves to now. Editing the AgentTemplate after checkpointing does not change what a fork of that checkpoint runs. That stability is the point of a checkpoint: it holds a known-good configuration you can return to, rather than tracking the template as it moves on.

Note

A checkpoint can only be forked when its snapshot captured durable data alone. kagent compiles every ActorTemplate to take a Data-scope snapshot on commit, so a checkpoint taken on a suspended AgentInstance is forkable. A checkpoint whose snapshot also captured process state is rejected with Checkpoint includes process state and cannot be forked, because process memory belongs to the one Actor that produced it.

Clean up

Delete the checkpoint. Deleting removes the Tag and releases the pin, and Agent Substrate can collect the snapshot whenever no tag names it.

grpcurl -plaintext \
  -d '{"checkpointId":"'"$CHECKPOINT_ID"'"}' \
  localhost:8083 kagent.api.v1alpha1.CheckpointService/DeleteCheckpoint

Delete the fork. A fork is an AgentInstance in its own right, so deleting the checkpoint that it started from does not remove it.
```
kagent delete agent-instance $FORK_ID
```

Next steps

Suspend and resumeUnderstand the snapshot cycle that checkpoints pin. SandboxingSee what the sandbox around each Actor isolates. IdentitySee who owns an AgentInstance and the checkpoints taken on it.

Use agents from an MCP client