Skip to Content
DevelopSnapshotting

User-Defined Snapshotting

Golem recovers agent state by replaying the oplog — the log of all operations performed by the agent. For long-running or CPU-heavy agents, this replay can become slow. Golem 1.5 introduces user-defined snapshotting, allowing agents to opt in to periodic snapshots so that recovery only needs to replay entries recorded after the last snapshot.

Snapshotting also enables manual agent updates between incompatible versions — see Updating Agents for details.

Go snapshotting is not supported in this release. Leave snapshotting disabled for Go agents and use oplog replay for recovery. There is no go/snapshotting template. The Go API examples below are reference material, not a supported recovery or update workflow.

Automatic snapshotting is optional. Without it, recovery starts from the authoritative recovery baseline, if one exists: the snapshot of the last successful manual update or snapshot-assisted automatic update. Otherwise it replays the full oplog.

Snapshotting is for durable agents only. An ephemeral agent takes no snapshots: a restart of an ephemeral agent replays only its initialization and never loads a snapshot. Each SDK refuses an ephemeral agent definition that enables snapshotting, and golem build refuses a component with such an agent type. The error names the agent. Remove the snapshotting option or make the agent durable.

Default snapshotting implementation

Each language SDK can generate JSON saving and loading when the complete agent state has an explicit JSON-deserialization contract. Automatic loading is not generated for arbitrary state: use custom snapshotting when your SDK cannot deserialize the complete state.

These counters use the default JSON implementation, without custom byte encoding. Each example requests a snapshot after every successful invocation to make the behavior easy to try. This is an example schedule, not the server’s default snapshotting policy; see Configuring snapshot-based recovery for other schedules.

Provide a state schema in the snapshotting configuration and do not provide a custom snapshot: { save, load } block. The SDK serializes and restores the schema-defined state as JSON:

import { defineAgent, method } from '@golemcloud/golem-ts-sdk'; import { z } from 'zod'; const Counter = defineAgent({ name: 'Counter', id: { name: z.string() }, snapshotting: { state: z.object({ value: z.number() }), policy: { everyNInvocations: 1 }, }, methods: { increment: method({ input: {}, returns: z.number() }), }, }); Counter.implement({ init: () => ({ value: 0 }), methods: { increment() { this.value += 1; return this.value; }, }, });

Only the schema-defined value is saved. On restoration, the SDK decodes it into fresh state without calling init. A bare enabled policy without state requires a custom save/load pair.

Evolving the state schema

During a manual update, the source revision saves a snapshot, and the target revision loads it. With the default implementation, the target revision decodes the saved JSON using its own state schema or deserializer. It does not run normal initialization first, so values supplied only by init or a constructor do not fill missing snapshot fields.

The target revision must be able to decode the snapshots of the agents you update:

With Zod, give a new field a .default(...) value or make it .optional(). A new required field without a decoding default makes loading fail when the saved JSON does not contain it:

snapshotting: { state: z.object({ value: z.number(), label: z.string().default(''), lastUpdated: z.number().optional(), }), policy: { everyNInvocations: 1 }, }

This schema can load an earlier snapshot containing only value. The restored label is ''; lastUpdated may be absent, so agent methods must handle that case.

For a rename or type change, the loader must accept the earlier representation and convert it to the complete target state. Use custom snapshotting when the default decoder cannot do this, subject to the SDK’s format restrictions described above. A version marker can help a custom loader select the conversion, but the marker alone does not make an earlier snapshot loadable. Test loading a snapshot saved by the source revision, not just a save/load round-trip in the target revision.

If decoding fails during a manual update, the update fails and the agent stays on its source revision. Inspect the update history:

golem agent get <AGENT_ID>

The history shows Failed update to ... with details such as Manual update failed to load snapshot: ... followed by the loader error. A typed TypeScript schema mismatch includes snapshot state does not match its declared schema. Fix the loader or schema in a new revision, deploy it, and request the update again. See Updating agents.

The same decoding rule applies when an automatic update uses a snapshot from the source revision. A decoding failure during a snapshot-assisted automatic update has UPDATE_SNAPSHOT_INCOMPATIBLE; see Failure codes.

Custom snapshotting

You can implement custom saving and restoration to control exactly how agent state is serialized. Saving runs on the live instance. Loading is a separate factory that returns the complete state or instance; the SDK never runs the normal agent initialization path first and then hands that instance to the loader. The following example shows a counter agent that saves its count as raw bytes:

Provide a snapshot: { save, load } block on .implement(...). save() has the agent state as this. load(bytes, context) has no this and returns a complete fresh state object:

import { z } from 'zod'; import { defineAgent, method } from '@golemcloud/golem-ts-sdk'; export const Counter = defineAgent({ name: 'Counter', id: { name: z.string() }, snapshotting: { everyNInvocations: 1 }, methods: { increment: method({ input: {}, returns: z.number() }), }, }); export const CounterImpl = Counter.implement({ init: () => ({ value: 0 }), methods: { increment() { this.value += 1; return this.value; }, }, snapshot: { save() { const snapshot = new Uint8Array(8); new DataView(snapshot.buffer).setFloat64(0, this.value); return snapshot; }, load(bytes, _context) { const value = new DataView( bytes.buffer, bytes.byteOffset, bytes.byteLength, ).getFloat64(0); return { value }; }, }, });

Multipart snapshots: JSON state and named bytes

Use explicit multipart snapshots when the live instance contains binary resources that should not be serialized into JSON. Save a small schema-backed state projection separately from a dynamic map of named {bytes, contentType} parts. This is an additional mode: the simpler JSON, byte-only, and existing SQLite snapshot modes remain available.

These examples save a revision counter and an in-memory binary index. The loaders construct fresh live state; ordinary initialization is skipped.

Select multipart.state in the definition. Both save and load are required, and may be async. The live state returned by init is independent of the saved schema.

import { Snapshot } from '@golemcloud/golem-ts-sdk'; // In defineAgent({ ... }): snapshotting: { multipart: { state: z.object({ revision: z.number() }) }, policy: { everyNInvocations: 10 }, } // In .implement({ ... }), with live { revision, index }: snapshot: { async save() { return { state: { revision: this.revision }, parts: new Map([ ['index', { bytes: this.index, contentType: 'application/octet-stream' }], ]), }; }, async load(saved, _context) { return { revision: saved.state.revision, index: Snapshot.requirePart(saved.parts, 'index', 'application/octet-stream'), }; }, }

Standard Schema validates the input on save, but the SDK persists that input JSON, not a transformed validation result. On load, the schema validates stored JSON and the loader receives its output type. There is no reverse encoding. Do not combine state and multipart configuration. Explicit multipart does not automatically discover SQLite databases and rejects managed db: parts.

Part and envelope rules

  • Names are case-sensitive and match [A-Za-z0-9_][A-Za-z0-9_.-]*. Logical names state and __proto__ are valid. Parts have no automatic resource semantics; applications decide what they mean and whether to accept extra parts.
  • Content types must be bare ASCII MIME type/subtype strings, such as application/octet-stream. They are normalized to lowercase; parameters such as ; charset=utf-8 are rejected. requirePart checks both presence and content type, returning the SDK’s normal snapshot error on failure.
  • Multipart is emitted even when there are no binary parts. Binary-only applications use JSON null state. Empty bytes and trailing CR, LF, or CRLF are preserved exactly.
  • The SDK owns the principal envelope. On the wire, one state part contains UTF-8 JSON {version:1, principal, state}, user names become part:<name>, and managed SQLite images use db:<name>. Do not construct these prefixes or envelope metadata yourself.
  • The SDK validates framing, metadata, names, duplicate structural fields, and supported namespaces before invoking user restoration. Changing snapshot modes does not enable fallback parsing or historical-format migration.

Snapshots are whole-buffer operations. Copies of state, part bytes, encoded payloads, and restored resources can amplify peak memory usage. The host’s 64 KiB inline/blob-spill setting is a storage threshold, not a maximum snapshot size. Save and load hooks are unpersisted lifecycle operations and may be retried; async hooks do not make external effects exactly-once.

How snapshot restoration runs

Snapshot loading is a specially supported SDK lifecycle operation, not an agent method. The SDKs keep it separate from normal initialization and install the returned state or instance only after restoration succeeds. Restore context exposes the agent identity and restored principal, plus the phantom ID and fresh configuration where applicable.

Golem runs snapshot loading in read-only mode and writes nothing from it to the oplog. Decoding, local computation, fresh randomness, configuration access, and other permitted reads can be used to construct the restored value. Durable host calls classified as writes, and outgoing HTTP or agent RPC calls, are rejected before they take effect. This restriction is not a general filesystem write sandbox: local filesystem writes are not blocked by it.

If loading fails, partial state is discarded:

  • A manual update fails and the agent remains on its previous component version. This is also true when the agent exits during the load. When an interrupt stops the load, the update stays pending, and the next start loads the snapshot again. See Updating agents.
  • A snapshot-assisted automatic update fails with UPDATE_SNAPSHOT_INCOMPATIBLE, and the agent remains on its previous component version. Golem does not reject the snapshot record. See Updating agents.
  • If an automatic snapshot is rejected by its loader, or loading or subsequent replay fails deterministically, Golem rejects only that snapshot record and recreates the component. The next start uses the previous usable automatic snapshot, if Golem keeps one and it is not the rejected record. When no usable automatic snapshot is left, Golem starts from the authoritative recovery baseline, or from the beginning if none exists. Transient failures and resource-limit failures can instead propagate or cause recovery to retry.
  • Failure to load the authoritative recovery baseline of an already completed manual update or snapshot-assisted update cannot fall back to pre-update history.

Configuring snapshot-based recovery

Choose when Golem takes automatic snapshots in the agent definition. Disabling automatic snapshot creation does not discard the authoritative recovery baseline of the last successful update, if one exists.

Pass the snapshotting option to defineAgent(...):

export const Counter = defineAgent({ name: 'Counter', id: { name: z.string() }, snapshotting: { state: z.object({ value: z.number() }), policy: { periodicSeconds: 5 }, }, methods: { increment: method({ input: {}, returns: z.number() }), }, });

Set snapshotting.policy to one of these values when using a state schema:

PolicyBehavior
'disabled'Disables automatic snapshot creation. Omitting snapshotting also disables it.
'default'Uses the server’s default schedule, which may be disabled.
{ everyNInvocations: 10 }Takes a snapshot every 10 successful invocations.
{ periodicSeconds: 5 }Schedules snapshots at safe invocation boundaries after each 5-second interval.

For custom byte snapshots, pass the policy directly as snapshotting instead of wrapping it in { state, policy }.

Agent files and snapshots

To include the agent’s files in snapshot-based recovery:

  1. Run the agent on executors with filesystem snapshots enabled. The local golem server run command enables them by default on macOS. Standalone executors and local servers on other platforms disable them by default.
  2. Configure the agent’s snapshotting policy and application state saving and loading as described above. Enabling executor snapshots alone does not schedule snapshots for an agent.
  3. Flush file output streams before each invocation returns. Your snapshot saver must not modify files: Golem does not run it again during replay.

Golem captures and restores the files together with the application snapshot. You do not need to include file contents in the application snapshot or write extra snapshot code for files. See Recovery and storage for capture, confirmation, restoration, failures, retention, and cold-start costs. For update-specific file conflicts, see Updating agents.

Without filesystem snapshots, automatic snapshots work only while the files are reconstructible from untouched read-only initial files. Otherwise Golem skips the snapshot and recovers from an older usable restore point or by replay, which rebuilds the files without data loss but can take longer. A snapshot-based manual update then fails with UPDATE_NEEDS_FILESYSTEM_SNAPSHOTS. A successful update’s authoritative recovery baseline is not optional: if it requires a filesystem snapshot, the agent cannot start without it. See Recovery without filesystem snapshots.

Typed state snapshotting saves the contents of top-level in-memory and temporary DatabaseSync fields. For a file-backed database it saves only the path; filesystem snapshots restore the file. Commit or roll back transactions before returning. Closed databases and fields holding StatementSync, Session, or SQLTagStore cannot be saved. SQLite replay also has page-cache limits. See the custom snapshot guide for the full rules.

Observability

When using the default (JSON-based) snapshotting implementation, snapshot entries recorded in the oplog contain the serialized JSON state of the agent. This makes it easy to inspect agent state at any snapshot point for debugging purposes.

Snapshotting is also used when updating agents across incompatible versions, allowing Golem to capture the agent’s state before applying the update.