Deployment

To run Golem in production, choose one of the deployment modes and go through the production checklist. This page describes the services that the deployments are made of.

Golem Services are distributed as Docker images. They are available in Golem Services Docker Hub . Each Docker image is built for the following architectures:

• linux/amd64
• linux/arm64

The services are using the following storage backends:

• Relational Database (RDB) - PostgreSQL  or SQLite (for running locally)
• Redis  - optional, for the worker executor’s storage or the worker service’s sessions
• Blob storage - Shared File System or S3
• etcd  - optional, only for the shard manager’s distributed mode

User and Permissions

Every image runs its service as the user golem, uid and gid 65534, with the home directory /home/golem. The default configuration keeps local data below /data, which belongs to that user: SQLite databases in /data and file system blob storage in /data/blob_storage. The golemservices/golem image keeps its data in /var/lib/golem, which also belongs to that user. The services need no Linux capability.

A path that a service writes must belong to uid 65534, or be writable by it:

• A new Docker named volume mounted at /data, /data/blob_storage or /var/lib/golem takes the owner of that directory in the image, so it needs no change.
• A bind mount takes the owner of the host directory. Give the directory to the user before the first start, for example with chown -R 65534:65534 <host directory>.
• In Kubernetes, set runAsUser, runAsGroup and fsGroup to 65534 in the pod securityContext. The kubelet then makes a volume that supports ownership management, such as an emptyDir or a persistent volume, writable by group 65534. A hostPath volume keeps the owner of the host directory.

Run the services with no capabilities and without privilege escalation: --cap-drop ALL --security-opt no-new-privileges with Docker, capabilities.drop: [ALL] and allowPrivilegeEscalation: false in Kubernetes. Only xfs-quotad, a separate process that ManagedXfs agent filesystem storage needs, holds a capability, CAP_SYS_ADMIN. Its image, golemservices/golem-xfs-quotad, also runs as uid 65534; see Run xfs-quotad.

Golem Services

The following sections provide a description of each service.

Golem

The golem  executable runs every service in one process with golem server run. The golemservices/golem image runs it, and serves the management API, the HTTP APIs and MCP on one port, 9881. Configure it with a TOML file passed as --config, or with GOLEM__<SECTION>__<FIELD> environment variables; see the configuration  and the environment variables . The image sets these variables:

Set GOLEM__ROUTER__MANAGEMENT_HOST to the host name of the management API. Requests for that host reach the management API, requests for every other host reach the HTTP API and MCP deployments, and no environment can register the host as a domain. Behind a reverse proxy, add the proxy’s address to GOLEM__WORKER_SERVICE__ROUTE_RESOLVER__TRUSTED_INGRESS_ADDRESSES, so the router takes the host from X-Forwarded-Host.

The router listens on all interfaces, so the server refuses to start until the admin token (GOLEM__REGISTRY_SERVICE__INITIAL_ACCOUNTS__ROOT__TOKEN), the webhook keys (GOLEM__WORKER_SERVICE__WEBHOOK_CALLBACK_HANDLER__HMAC_KEY and GOLEM__WORKER_EXECUTOR__AGENT_WEBHOOKS_SERVICE__HMAC_KEY, the same value) and the invocation session token key (GOLEM__WORKER_SERVICE__INVOCATION_SESSION_TOKENS__ACTIVE_KEY__KEY) have secret values. The error lists the settings that still have their public default values.

At every start, the registry service provisions the built-in plugins and tools. It reads each of them from its cache, builtin-artifacts in the data directory, and downloads from the golem-builtins  releases on GitHub each one that the cache does not have or that does not match its checksum. The downloads take about a minute. Without outgoing HTTPS connections to GitHub, a start that must download fails. Keep /var/lib/golem on a persistent volume, so that only the first start downloads. Without a persistent volume, for example in a container without a volume or in a task without storage, every start downloads again.

The image arguments are the arguments of golem; the default is server run. With GOLEM_LITESTREAM=true, Litestream  replicates the SQLite databases of the data directory. Use it where the volume itself can be lost, for example an ephemeral disk or a disk of one host. In this mode the worker executor keeps its key-value storage and its indexed storage in one SQLite database each, key-value-storage.db and indexed-storage.db, instead of one database per agent. The server then has a fixed set of databases: these two, registry.db, shard_manager.db, scheduler.db and gateway-sessions.db. No database file is ever deleted: deleting an agent deletes rows, and Litestream replicates that like any other write. Before the server starts, the image restores each of these databases that the data directory does not have from the replica. Turn Litestream on only for a new data directory: in this mode the server does not read the databases per agent in kv-store, which it writes without Litestream.

Litestream takes the data directory from GOLEM__DATA_DIR, so do not set data_dir in the --config file, and it sets the key-value storage and the indexed storage of the worker executor, so do not set them either. Litestream replicates only the SQLite databases: keep the blobs in S3, for example in one bucket with namespace_prefixes = true. Litestream does not replicate the cache of the built-in plugins and tools, so a restore to a new volume downloads them again. Exactly one server may write to a replica.

See also: docker image 

Registry Service

golem-registry-service  manages accounts, applications, environments, components, plugins, tools, policies, and related registry data in the relational database and blob storage.

See also: configuration , environment variables , docker image 

Worker Service

golem-worker-service  provides APIs and HTTP gateway functionality for agents and acts as a routing service for worker executors. It retrieves gateway definitions from the registry service, uses blob storage for initial agent files, and stores gateway sessions in Redis, SQLite, or PostgreSQL.

To use PostgreSQL for gateway sessions, set gateway_session_storage.type = "Postgres" in the worker service configuration. Under gateway_session_storage.config, set host, port, database, username, password, max_connections, pending_login_expiration, and cleanup_interval. The optional schema defaults to golem_worker_service_session_store, separate from other services’ migration tables. For the single executable, nest these settings under worker_service; supply the password through GOLEM__WORKER_SERVICE__GATEWAY_SESSION_STORAGE__CONFIG__PASSWORD rather than checking it into a configuration file.

See also: configuration , environment variables , docker image 

Long-lived connections

See HTTP Gateway for proxy idle timeouts, WebSocket Pings and Pong deadlines, MCP keep-alives, and Durable Streams SSE behavior. The same page covers trusted ingress and shared HTTP API and MCP ports.

MCP with several replicas

The worker service keeps each MCP session as a record in the MCP session store: mcp.session_storage, or the gateway session store when that setting is not set. See MCP sessions. Every replica reads the same store, so any replica serves any request of a session, and adding, removing or restarting a replica does not end the sessions. With more than one replica, the store must be Redis (6.2 or later) or Postgres, shared by all replicas. A SQLite store serves one process only.

A replica keeps the running tool calls and the response streams of a session in its own memory. So the proxy or load balancer in front of the worker service must send all requests that carry the same Mcp-Session-Id header to the same replica. Use consistent hashing on the Mcp-Session-Id header. initialize carries no session ID and can go to any replica. Cookie-based affinity does not work, because MCP clients do not keep cookies. Without affinity, sessions still work, but notifications/cancelled does not reach the running call, a call on another replica ends only at its next check of the session after a DELETE, and a client cannot resume a cut response stream with Last-Event-ID.

When the number of replicas changes, consistent hashing sends some sessions to another replica. These sessions continue on the new replica, but the calls and streams that the old replica holds stay on the old replica:

• notifications/cancelled for a call that runs on the old replica has no effect, and the call runs to its end.
• DELETE ends the session. A call that runs on the old replica ends at its next check of the session. A running call checks its session every mcp.session_idle_timeout / 3, and at least once a minute. A GET stream that is open on the old replica stays open until its next check of the session, at most mcp.session_idle_timeout later.
• A client cannot resume a response stream that the old replica holds.

Golem Router sends the requests that carry Mcp-Session-Id to a replica chosen by a consistent hash of that header, so a deployment behind it needs no more configuration. Behind another proxy or load balancer, configure the hashing there.

Worker Executor

golem-worker-executor  is responsible for running the agents that belong to assigned shards. The service keeps each agent’s oplog in indexed storage - PostgreSQL, SQLite or Redis - and uses key-value and blob storage alongside it.

See also: configuration , environment variables , docker image 

Shard Manager

golem-shard-manager  assigns shards to worker executors and rebalances them when executors join or leave. Run exactly one instance with PostgreSQL or SQLite persistence. Use etcd persistence to run multiple replicas with one elected leader.

See Shard Manager for configuration, failover, health probes, and storage recovery precautions.

See also: configuration , environment variables , docker image 

Component Compilation Service

golem-component-compilation-service  warms the compiled-component cache in Blob storage. It is an optional optimisation: if precompilation is skipped or fails, the executor compiles and caches the component on first use. Pending requests are held in memory and are lost on restart. The service compiles one component at a time because Wasmtime already parallelises compilation internally.

The service rejects an enqueue immediately when its request queue is full. The registry still completes the component upload. The registry’s component_compilation.config.enqueue_timeout bounds the entire enqueue attempt, including connection establishment and retries; its default is 1s. A timeout skips cache warming, not the component upload. A timed-out request might already have reached the compilation service and can still be compiled. Saturation is not retried inside the upload request.

Both capacities must be positive. Their environment variables are GOLEM__COMPILE_WORKER__QUEUE_CAPACITY and GOLEM__COMPILE_WORKER__UPLOAD_QUEUE_CAPACITY. The registry timeout is GOLEM__COMPONENT_COMPILATION__CONFIG__ENQUEUE_TIMEOUT.

Choose queue capacities

The defaults are starting values, not measured recommendations for golem-dev or golem-prod. Measure each environment with its actual CPU and memory limits and Blob storage backend before selecting values:

1. Use representative small and large components and realistic deployment bursts. For cold-compilation measurements, confirm compiled-blob cache misses and disable or start with a cold Wasmtime filesystem cache (engine.enable_fs_cache). Fresh component revisions alone do not bypass the filesystem cache. Measure warm-cache workloads separately.
2. Observe component_compilation_time_seconds and the pending-request queue. Measure sustained successful compilations and artifact uploads per second over wall-clock time. Download time and upload backpressure affect throughput but are outside the histogram. Choose how much cache-warming delay is useful: validate the time from admission until the artifact is available against the expected first invocation. Use measured throughput and that delay to choose the request capacity. A larger queue does not increase compilation throughput.
3. Exercise slow Blob storage uploads. Observe component_compilation_upload_queue_length and process/pod peak memory with the proposed capacity. The upload queue retains compiled components, which can be much larger than request metadata. In addition to the queue, allow for one component being uploaded, one being processed or waiting for upload-queue space, compilation working memory, download buffers, artifact serialization buffers, Blob storage upload buffers, and runtime overhead. Leave headroom below the pod memory limit.
4. Run a burst that exceeds the request capacity. Confirm that component_compilation_enqueue_total{outcome="rejected"} increases in the registry and that uploads remain responsive. Exercise an unavailable or unresponsive compilation service and confirm the configured enqueue budget bounds the extra upload delay.
5. Record the component mix, resource limits, compilation durations, sustained throughput, time until artifacts are available, queue peaks, peak memory, enqueue outcomes, and upload latency with the chosen golem-dev and golem-prod settings in the deployment configuration review.

Use process or container memory measurements, such as the platform’s working-set/RSS metrics. Do not interpret RSS differences between compilations as precise per-component allocation measurements: allocators, shared compiler threads, and retained artifacts affect them. The compilation-duration histogram measures successful Wasmtime component-build calls. It excludes request-queue wait, component download, compiled-blob cache hits, and artifact upload, but includes Wasmtime filesystem-cache loads when that cache is enabled. The queue gauges exclude in-flight work; a flat queue does not prove progress. Check compilation logs and duration samples as well.

See also: configuration , environment variables , docker image 

Golem Router

golem-router  is an Envoy  proxy that serves the services on one port, routed by host. Requests for the management host reach the registry service and the management API of the worker service. Requests for every other host reach the custom request port of the worker service, which serves the HTTP API and MCP deployments by domain. Point every name at the router in DNS. The image runs Envoy as uid and gid 65534.

The container arguments are added to the arguments of Envoy, for example --log-level debug. A --concurrency argument replaces GOLEM_ROUTER_CONCURRENCY.

Envoy starts one worker thread for each CPU of the host, not for each CPU of the container limit, and each worker thread opens its own connections to the services. Set GOLEM_ROUTER_CONCURRENCY to the CPU limit of the container.

With GOLEM_ROUTER_STATS_PORT, the router serves /stats/prometheus on that port and answers every other path with 404. The Envoy admin interface listens on 127.0.0.1:19000 in the container and is not reachable from outside it. Do not publish the metrics port to the internet; scrape it from inside the deployment.

The router sends HTTP/2 without TLS to the services, so a request that the client cancels is also cancelled in the service. It sends the WebSockets of the management API, /connect and the invocation session (/v1/agents/invoke-agent-session), to the worker service over HTTP/1.1. HTTP API deployments do not serve WebSockets, so the router answers a WebSocket upgrade for any other host with 403. When a host name resolves to several replicas, the router sends a request that carries Mcp-Session-Id to the replica chosen by a consistent hash (Maglev) of that header, so all requests of an MCP session reach one replica; see MCP with several replicas. It sends every other request to the replica with the fewest open requests. A request has no time limit, so invocations that wait for a result, server-sent events, MCP sessions and websockets stay open for as long as they send data; GOLEM_ROUTER_STREAM_IDLE_TIMEOUT ends a silent one. The router does not limit the size of a request body; the services apply their own limits. It accepts header names with an underscore and keeps an encoded slash in the path. It resolves the service host names again every GOLEM_ROUTER_DNS_REFRESH_RATE, so it follows a service that restarts with a new address, and it starts before the services are resolvable. The router answers /metrics on the management host with 404, so the metrics of the services do not reach the clients; scrape the services directly.

The router keeps the original Host header, with its port, and sets X-Forwarded-Host to the same value. It appends the address of the client to X-Forwarded-For, and sets X-Forwarded-Proto to its own scheme, http. Where TLS ends at a proxy in front of the router, set GOLEM_ROUTER_TRUSTED_HOPS to 1, so that the router keeps the X-Forwarded-Proto of that proxy.

Configure worker-service to trust the router’s address and reserve the management host in the registry. See HTTP Gateway for the forwarding-header contract and domain reservation.

See also: configuration , docker image 

Request Limits

Every path into an agent has the same payload limit of 32 MiB: the gRPC message from the worker service to the worker executor, the invocation session WebSocket message, and the REST and MCP request body.

A request that declares a larger Content-Length gets 413 before the service reads its body. A request without Content-Length gets 413 when its body passes the limit.

The invocation session and the agent event stream (connect) answer a message or frame above their limit with close code 1009. They do not read the payload of a frame whose header declares a larger size.

The HTTP/2 header list limit is above the 60 KiB that Envoy accepts by default, so a gateway in front of Golem decides first. The HTTP/2 settings apply to every HTTP port of the service: for the worker service, port, custom_request_port and mcp_port.

For environment variables, add the path to GOLEM__, for example GOLEM__BODY_LIMITS__API. For golem server run, nest the settings under worker_service or registry_service, for example GOLEM__WORKER_SERVICE__BODY_LIMITS__MCP. The HTTP/2 settings of the golem server run listener are router.http_server, for example GOLEM__ROUTER__HTTP_SERVER__HTTP2_MAX_HEADER_LIST_SIZE.

Network Addresses

Every service binds all of its listeners to bind_address (GOLEM__BIND_ADDRESS). The default 0.0.0.0 accepts IPv4 connections only. Set it to :: on a network that needs IPv6, such as a Fly.io private network or an IPv6-only Railway environment: the service then accepts both IPv6 and IPv4 connections, whatever the operating system’s default for IPv6 sockets is. On a host whose kernel has no IPv6 support, binding :: fails, and the service stops at startup with the address in the error.

The shard manager identifies a worker executor by the address it reaches it at, and by default takes that address from the source IP of the executor’s registration. Where the network translates source addresses, that IP is not one the shard manager and the worker services can dial. Set grpc.advertised_ip (GOLEM__GRPC__ADVERTISED_IP) on each executor to the IP address it is reachable at - on Fly.io, the machine’s FLY_PRIVATE_IP - and the shard manager uses it for the shard lease and for the quota leases of that executor. Every executor needs an address of its own: two executors that advertise the same address replace each other’s registration. An advertised IP address must be of a family the executor’s bind_address accepts - an advertised IPv6 address needs an IPv6 bind_address, usually :: - and an address that names no single host - unspecified, multicast, broadcast or IPv6 link-local - is refused at startup.

Deployment Variants

For deployment variants, see the following sections: