For the complete documentation index, see llms.txt. Markdown versions of all docs pages are available by appending .md to any docs URL.

See how Agent Substrate runs, suspends, and resumes the Actors behind every AgentInstance.

The kagent architecture page established that every AgentInstanceAgentInstanceA running, conversational pairing of a Harness and an AgentTemplate. Unlike the two, it is not a Kubernetes resource: kagent's gRPC API creates it and its database tracks it.Learn more runs on an Actor. This page explains what an Actor is built from and what it runs on: the ActorTemplate that it is created from, the compute that hosts it, the atespace that identifies it, the sandbox that isolates it, and the snapshotSnapshotThe stored state that an Actor suspends to, held in object storage. Resuming restores the Actor from its most recent snapshot, which is what makes suspending idle agents cheap.Learn more cycle that lets it suspend when idle and resume on demand.

ActorTemplate

Every Actor is created from an ActorTemplate, the compiled definition that the kagent controller produces from a HarnessHarnessA Kubernetes custom resource defining how an agent is allowed to run: its runtime, workload image, WorkerPool and snapshot storage, and which AgentTemplates it accepts.Learn more and AgentTemplateAgentTemplateA Kubernetes custom resource defining what an agent does: its model, system prompt, tools, skills, and plugins. It runs only once a Harness accepts it.Learn more pair.

Substrate adds enforcement. It rejects any change to an ActorTemplate’s spec after it is created, so immutability is a property of the resource itself rather than a convention that the controller follows. That immutability requires the controller to create a new ActorTemplate for every compiled revisionRevisionThe compiled, immutable output of one Harness and AgentTemplate pairing, identified by a content digest. An AgentInstance runs the revision it was created from for its whole life, so editing either resource affects only instances created afterward. instead of editing an existing one, and allows the controller to safely reclaim an old ActorTemplate once no AgentInstance references it.

Workers and WorkerPools

An Actor needs somewhere to run. Each Actor runs on a Worker: a pre-started, sandboxed pod that hosts at most one Actor at a time. Instead of starting a new pod each time an AgentInstance needs an Actor, Substrate schedules that Actor onto a Worker that is already running and waiting.

Workers come from a WorkerPool, a Kubernetes custom resource that an operator provisions before any Harness can create AgentInstances. A WorkerPool declares how many Workers to keep running and which sandbox technology those Workers use.

An operator never creates a Worker directly. Substrate manages them, keeping enough ready in each WorkerPool so that an Actor can start or resume on one immediately, without waiting on the Kubernetes scheduler to place a new Pod.

Atespaces

An atespace is the isolation boundary that an Actor belongs to, and the first half of its identity. Agent Substrate addresses an Actor by its atespace and its name together, so the same Actor name can exist in two atespaces without colliding. Despite the resemblance, an atespace is a global-scoped Agent Substrate resource rather than a Kubernetes namespace.

kagent names each atespace after the Kubernetes namespace of the AgentInstance whose Actor it holds, and creates that atespace on demand the first time an AgentInstance in the namespace needs an Actor. The Actor’s own name comes from the AgentInstance’s identifier. An AgentInstance in the kagent namespace therefore runs on an Actor that Agent Substrate addresses within the kagent atespace. Both halves of that identity appear in the address that traffic uses to reach the Actor, which Sandboxing covers.

Sandboxing

Because an Actor often runs a model-directed agent that calls tools and executes commands, Substrate runs each Actor in an isolated sandbox rather than a plain container. A WorkerPool’s sandboxClass field selects the sandbox technology for its Workers: gVisor or a micro-VM technology such as Kata Containers. Both technologies isolate an Actor from its Worker’s host kernel, and both support suspend and resume operations.

kagent compiles every ActorTemplate to the gvisor class, so a kagent agent runs in a gVisorgVisorA user-space kernel that isolates a workload from the host kernel by intercepting its system calls. kagent compiles every ActorTemplate to the gvisor sandbox class.Learn more sandbox today and the micro-VM class is a Substrate capability that kagent does not yet select. Keep a WorkerPool that backs kagent Harnesses on gvisor. For what each class isolates, see Sandboxing.

Suspend, snapshot, and resume

Substrate’s density model rests on one fact about agent workloads: an Actor spends most of its time idle, waiting on a person or a large language model (LLM) to respond, not actively computing. Substrate exploits that by suspending idle Actors and reclaiming their Worker, then resuming them on demand when traffic arrives. Suspending and resuming allows a WorkerPool to run far more Actors than it has Workers for at any given moment.

The following diagram traces an Actor through one suspend-and-resume cycle, and shows the second path that opens up once the resulting snapshot is tagged.

    flowchart LR
    pool["WorkerPool"] --> worker1["Worker"]
    worker1 -->|hosts| actor["Actor<br>(running)"]
    actor -->|suspend| snapshot["ActorSnapshot<br>(immutable)"]
    snapshot -->|resume| worker2["Any free Worker<br>in the pool"]
    snapshot -->|pinned by| tag["ActorSnapshotTag<br>(retention pin)"]
    tag -->|seeds| newactor["New Actor"]
  

A WorkerPool keeps Workers running and ready, and one Worker hosts the Actor while its conversation is active. Suspending that Actor writes its full state to an immutable ActorSnapshot and frees the Worker that it was running on.

The diagram forks at that snapshot, because a snapshot serves two purposes.

• Resume restores the same Actor onto any free Worker in the pool, which is not necessarily the Worker that it ran on before. Because the snapshot captures the Actor’s full state, the conversation continues from where it left off. Every idle agent takes this path.
• A TagTagAn Agent Substrate resource that gives one snapshot a stable, human-meaningful name, so callers do not need to track Substrate's internal snapshot identity. It also acts as a retention pin: Agent Substrate does not collect a snapshot while a tag names it, and only the tag's visibility scope can change afterward.Learn more pins that snapshot, and a New Actor can be seeded from the tag at the moment that it is created. Resuming an existing Actor never goes through a tag.

A tag gives a snapshot a stable, human-meaningful name, so callers do not need to track Substrate’s internal snapshot identity. A tag names one snapshot permanently, and only its visibility scope can change afterward. A tag also acts as a retention pin, so Substrate does not delete a snapshot while a tag still names it.

For example, an agent partway through a long incident investigation reaches a state worth keeping. Creating a checkpoint tags the snapshot that the agent most recently suspended to, which holds that one snapshot in place while the agent carries on and writes newer ones. Without the tag, Substrate collects that snapshot once a newer one supersedes it.

Substrate’s own target for this cycle is 100 milliseconds at the ninety-fifth percentile, measured from the moment traffic arrives for a suspended Actor to the moment that Actor can receive it.