Metrics
Overview
Every Golem service exposes Prometheus metrics on its HTTP interface. The metrics are available at the /metrics endpoint for scraping.
Beside the service-specific metrics each service exports a single counter called version_info in which the labels hold service-specific information.
The currently available labels are:
| Label | Description |
|---|---|
version | The version of the service |
The shard lease metrics take their label values from a fixed set and never from an agent, project or environment identifier. A label value is exported at 0 before its first event, so a query reads 0 instead of no data. To find a single agent, use the logs at the site of the refusal.
Service-specific metrics
Component Service
Component service exposes the following Prometheus metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
poem_request_count | counter | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Counts every request on the HTTP interface |
poem_errors_count | counter | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Counts failed requests on the HTTP interface |
poem_request_duration_ms | histogram | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Measures the duration of handling requests on the HTTP interface |
api_success_seconds | histogram | api, api_type | Measures the duration of successfully serving API requests (both HTTP and gRPC) |
api_failure_seconds | histogram | api, api_type, error | Measures the duration of failed API requests (both HTTP and gRPC) |
grpc_api_active_streams | gauge | Number of open incoming gRPC streams |
Component Compilation Service
Component compilation service exposes the following Prometheus metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
cache_size | gauge | cache | Current maximal capacity of the cache |
cache_hit_total | counter | cache | Number of cache hits |
cache_miss_total | counter | cache | Number of cache misses |
cache_eviction_total | counter | cache, trigger | Number of cache evictions |
component_compilation_queue_length | gauge | Number of compilation requests enqueued | |
compilation_time_seconds | histogram | Time to compile a WASM component to native code |
Shard Manager
Shard manager exposes the following Prometheus metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
api_success_seconds | histogram | api, api_type | Measures the duration of successfully serving API requests (both HTTP and gRPC) |
api_failure_seconds | histogram | api, api_type, error | Measures the duration of failed API requests (both HTTP and gRPC) |
grpc_api_active_streams | gauge | Number of open incoming gRPC streams | |
external_call_success_seconds | histogram | target, op | Duration of successful outgoing calls |
external_call_response_size_bytes | histogram | target, op | Size of the response of outgoing calls |
external_call_retry_total | counter | target, op | Number of failed outgoing calls that got retried |
external_call_failure_total | counter | target, op | Number of failed outgoing calls not to be retried |
redis_success_seconds | histogram | svc, api, cmd | Duration of successful Redis calls |
redis_failure_total | counter | svc, api, cmd | Number of failed Redis calls |
redis_serialized_size_bytes | histogram | svc, entity | Size of serialized Redis entities |
redis_deserialized_size_bytes | histogram | svc, entity | Size of deserialized Redis entities |
shard_manager_is_leader | gauge | 1 while this replica holds the etcd leadership lease, 0 while it stands by (distributed mode only) | |
shard_manager_is_serving | gauge | 1 while this replica serves calls: a leader whose startup has completed, or a shard manager in local mode; 0 otherwise. A replica that has just won the election holds the leadership but serves only after its startup | |
shard_manager_leader_since_epoch | gauge | Unix seconds at which this replica was elected, 0 while it stands by (distributed mode only) | |
shard_manager_campaign_attempt_failures_total | counter | Attempts to reach etcd on the way to leadership that failed retriably and were retried (distributed mode only) | |
shard_manager_executor_lease_expired_total | counter | Executor leases that lapsed without a renewal and had their shards released. Only the leader counts, so sum it over all replicas. This counter also sees an executor that no longer answers a scrape |
Worker Executor
Worker executors expose the following Prometheus metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
api_success_seconds | histogram | api, api_type | Measures the duration of successfully serving API requests (both HTTP and gRPC) |
api_failure_seconds | histogram | api, api_type, error | Measures the duration of failed API requests (both HTTP and gRPC) |
grpc_api_active_streams | gauge | Number of open incoming gRPC streams | |
external_call_success_seconds | histogram | target, op | Duration of successful outgoing calls |
external_call_response_size_bytes | histogram | target, op | Size of the response of outgoing calls |
external_call_retry_total | counter | target, op | Number of failed outgoing calls that got retried |
external_call_failure_total | counter | target, op | Number of failed outgoing calls not to be retried |
redis_success_seconds | histogram | svc, api, cmd | Duration of successful Redis calls |
redis_failure_total | counter | svc, api, cmd | Number of failed Redis calls |
redis_serialized_size_bytes | histogram | svc, entity | Size of serialized Redis entities |
redis_deserialized_size_bytes | histogram | svc, entity | Size of deserialized Redis entities |
cache_size | gauge | cache | Current maximal capacity of the cache |
cache_hit_total | counter | cache | Number of cache hits |
cache_miss_total | counter | cache | Number of cache misses |
cache_eviction_total | counter | cache, trigger | Number of cache evictions |
compilation_time_seconds | histogram | Time to compile a WASM component to native code | |
event_total | counter | event | Number of events produced by workers |
event_broadcast_total | counter | event | Number of events broadcasted by the executor |
worker_executor_call_total | counter | api | Number of calls to the worker layer |
promises_count_total | counter | Number of promises created | |
promises_scheduled_complete_total | counter | Number of scheduled promise completions | |
assigned_shard_count | gauge | Number of assigned shards | |
stale_shard_delivery_total | counter | delivery | Shard deliveries dropped for naming a revision older than the last one applied; delivery is register, assign, revoke or renewal. One is normal when a push and a renewal cross; a sustained rate is not |
shard_lease_expiry_timestamp_seconds | gauge | Unix time at which the shard lease of this executor lapses unless it is renewed. Absent on a single-shard executor, which has no lease, and before the executor registers. shard_lease_expiry_timestamp_seconds - time() is the remaining lease time, and it keeps counting down when the renewal loop stops | |
shard_lease_renewals_total | counter | outcome | Shard lease renewal attempts; outcome is granted, not_registered, timeout, lease_not_found or failed. On lease_not_found the executor drops its shards and registers again |
shard_admission_refusals_total | counter | reason | Agent admissions refused by the shard assignment; reason is not_registered, lease_lapsed (the executor fenced itself) or not_owned (the agent is on another executor’s shard, which is normal during a rebalance) |
create_worker_seconds | histogram | Time to create a new worker | |
create_worker_failure_total | counter | error | Number of failed worker creations |
invocation_total | counter | mode, outcome | Number of invocations |
invocation_consumption_total | histogram | Amount of fuel consumed by an invocation | |
allocated_memory_bytes | histogram | Amount of memory allocated by a single memory.grow instruction | |
host_function_call_total | counter | interface, name | Number of calls to specific host functions |
resume_worker_seconds | histogram | Time taken to resume a worker | |
replayed_functions_count | histogram | Number of functions replayed per forker resumption | |
oplog_svc_call_total | counter | api | Number of calls to the oplog layer |
Durable Stream Sessions
| Metric | Type | Labels | Description |
|---|---|---|---|
golem_durable_stream_lifecycle_total | counter | event | Registered and terminal durable stream transitions |
golem_durable_stream_open | gauge | Open streams owned by loaded producers | |
golem_durable_stream_producer_operation_total | counter | operation, outcome | Producer commits and replay no-ops |
golem_durable_stream_catch_up_distance_events | histogram | Committed events read before joining a live tail | |
golem_durable_stream_live_join_total | counter | outcome | Accepted and rejected live-tail joins |
golem_durable_stream_active_readers | gauge | Attached live-tail readers | |
golem_durable_stream_backpressure_total | counter | Fan-outs that waited on a full reader queue | |
golem_durable_stream_attempt_total | counter | operation, outcome | Start, resume, and takeover attempts |
golem_durable_stream_attachment_epoch | histogram | Accepted attachment epochs | |
golem_durable_stream_journal_lag_events | histogram | Source events not yet recorded in the value-only consumer journal | |
golem_durable_stream_attachment_operation_total | counter | operation, outcome | Prepare, activate, renew, and finalize attachment operations |
golem_durable_stream_lease_remaining_seconds | histogram | Lease headroom observed during attachment work | |
golem_durable_stream_reconciliation_total | counter | outcome | Reconciler decisions and failures |
golem_durable_stream_cascade_total | counter | outcome | Durable producer-deletion cascade outcomes |
golem_durable_stream_limit_violation_total | counter | limit | Rejections by bounded protocol-v1 limit category |
foreign_stream_fingerprint_mismatch_total | counter | Foreign-stream accesses rejected because the current agent fingerprint differs. This metric has no golem_ prefix. |
For configuration, recovery, and deletion diagnostics, see Operating Durable Stream Sessions.
Agent Filesystems
These metrics are for the XFS volume of the agent filesystems. An executor exports them at 0 from the start when it uses an XFS storage mode. The project metrics are only for ManagedXfs. For the usage of the volume, use the volume metrics of your platform, for example kubelet_volume_stats_available_bytes on Kubernetes.
| Metric | Type | Labels | Description |
|---|---|---|---|
filesystem_pressure_recovery_seconds | histogram | outcome | Time of a recovery of free space after a write failed for space. The recovery unloads idle agents until the free space reaches the target. outcome is recovered, target_not_reached (no idle agent was left to unload), deadline, observation_failed (the free space could not be read), cleanup_failed (the cleanup of an unloaded agent filesystem failed), not_under_pressure (the volume had enough space, so the failure has another cause) or unavailable (the executor is shutting down). |
filesystem_pressure_unload_attempts_total | counter | outcome | Each unload of an idle agent that a recovery of free space tried. outcome is unloaded, ineligible (the agent got new work or is being deleted before the unload ran; the recovery goes on with the next agent), cleanup_failed (the agent stopped, but the cleanup of its filesystem failed), task_failed (the unload task panicked or was cancelled) or deadline (the recovery deadline passed before the unload finished). Many ineligible attempts with target_not_reached recoveries mean that the candidates woke up, not that none were idle. |
filesystem_storage_refusals_total | counter | cause | Agent filesystem calls that failed because no space was left. cause is agent_quota (the agent used up its own quota, which is normal) or physical_capacity (the volume is full, and a recovery did not free enough). |
filesystem_project_cleanup_failures_total | counter | operation | Failed cleanups of managed XFS projects. The project stays reserved, and its space stays used, until a later cleanup succeeds. operation is delete (the delete of an agent filesystem), drop (the fallback cleanup of an agent filesystem that was dropped without a delete) or stale (the cleanup of a project left over at the path of a new agent filesystem). |
filesystem_project_stale_reclaims_total | counter | Managed XFS projects that a new agent filesystem found left over at its path, for example after a crash, and cleaned. A rate that keeps going up shows that cleanups fail. |
Filesystem Snapshots
| Metric | Type | Labels | Description |
|---|---|---|---|
filesystem_snapshot_capture_seconds | histogram | outcome | Time that a capture of an agent filesystem, or a check of its initial files on an executor without filesystem snapshots, stops the file calls |
filesystem_snapshot_upload_seconds | histogram | kind, outcome | Time of an upload of a filesystem snapshot, with its retries |
filesystem_snapshot_uploaded_bytes_total | counter | kind | File sizes of the trees of the uploaded filesystem snapshots |
filesystem_snapshot_uploads_in_progress | gauge | Runs of uploads that hold a slot now | |
filesystem_snapshot_restore_seconds | histogram | outcome | Time of a restore of a filesystem snapshot, with the wait for a restore slot and the retries |
filesystem_snapshot_dropped_confirmations_total | counter | reason | Uploaded filesystem snapshots that no confirmation record names |
filesystem_snapshot_leaked_cleanups_total | counter | operation | Clean-ups of filesystem snapshots that left snapshots in blob storage |
filesystem_snapshot_cleanups_pending | gauge | Agents with clean-up work of filesystem snapshots that waits or runs | |
filesystem_snapshot_copy_seconds | histogram | outcome | Time of a copy of the filesystem snapshots of a fork source |
filesystem_snapshot_skipped_total | counter | kind, reason | Filesystem snapshots that the admission skipped |
filesystem_snapshot_restored_age_seconds | histogram | Time from the save of a filesystem snapshot to its successful restore | |
filesystem_snapshot_delete_seconds | histogram | operation, outcome | Time of a delete of filesystem snapshots, with its prune |
filesystem_snapshot_prune_seconds | histogram | outcome | Time of a prune of a filesystem snapshot repository after a delete |
filesystem_snapshot_prune_skipped_total | counter | reason | Deletes of filesystem snapshots after which no prune ran |
filesystem_snapshot_retention_runs_total | counter | outcome | Retention runs after a filesystem snapshot |
filesystem_snapshot_stored_bytes_total | counter | source | Bytes of repository files that the executor wrote to blob storage |
filesystem_snapshot_removed_bytes_total | counter | source | Bytes that the executor deleted from the filesystem snapshot storage |
filesystem_snapshot_released_bytes_total | counter | Packed bytes of deleted filesystem snapshots that a later prune can remove |
For the label values and what to watch, see Filesystem Snapshots.
Worker Service
Worker service exposes the following Prometheus metrics:
| Metric | Type | Labels | Description |
|---|---|---|---|
poem_request_count | counter | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Counts every request on the HTTP interface |
poem_errors_count | counter | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Counts failed requests on the HTTP interface |
poem_request_duration_ms | histogram | http.request.method, url.full, [http.path_pattern], http.response.status_code, exception.message | Measures the duration of handling requests on the HTTP interface |
api_success_seconds | histogram | api, api_type | Measures the duration of successfully serving API requests (both HTTP and gRPC) |
api_failure_seconds | histogram | api, api_type, error | Measures the duration of failed API requests (both HTTP and gRPC) |
grpc_api_active_streams | gauge | Number of open incoming gRPC streams | |
http_api_active_streams | gauge | kind: connect, invocation_session, durable_stream_sse | Number of open long-lived HTTP streams: agent connect WebSockets, invocation session WebSockets and durable stream reads with live=sse |
external_call_success_seconds | histogram | target, op | Duration of successful outgoing calls |
external_call_response_size_bytes | histogram | target, op | Size of the response of outgoing calls |
external_call_retry_total | counter | target, op | Number of failed outgoing calls that got retried |
external_call_failure_total | counter | target, op | Number of failed outgoing calls not to be retried |
cache_size | gauge | cache | Current maximal capacity of the cache |
cache_hit_total | counter | cache | Number of cache hits |
cache_miss_total | counter | cache | Number of cache misses |
cache_eviction_total | counter | cache, trigger | Number of cache evictions |
http_session_reattach_attempts_total | counter | HTTP session reattach attempts | |
http_session_reattach_seconds | histogram | outcome | Duration of an HTTP session reattach |
http_session_stale_fencing_total | counter | Stale HTTP sessions that a newer attachment fenced | |
http_session_explicit_cancellations_total | counter | HTTP sessions that ended by cancellation | |
http_session_terminal_total | counter | cause | HTTP sessions that ended without a completed exchange, by cause |
http_session_cleanup_seconds | histogram | outcome | Duration of the cleanup of an HTTP session that did not complete |
durable_stream_append_total | counter | outcome | Durable stream append outcomes: accepted, duplicate, fenced, and gap. |
durable_stream_load_rejection_total | counter | reason | Durable stream requests rejected by per-stream reader, per-node reader, catch-up-rate, or append-rate limits. append-rate-limit identifies append admission failures. |
ephemeral_phantom_invocation_rejection_total | counter | reason | Ephemeral invocation requests rejected for an invalid phantom ID |
openapi_cache_total | counter | outcome | OpenAPI cache lookups and stale fencing |
openapi_generation_seconds | histogram | outcome | Duration of an OpenAPI generation |
OTLP Plugin
The OTLP plugin exports durable stream summaries separately from the service’s Prometheus endpoint:
| Metric | Type | Labels | Description |
|---|---|---|---|
golem.durable_stream.event_count | counter | event, outcome | Durable stream summary events. event distinguishes registration, aggregated items, stream end/cancellation, and session result/finish/cancellation/expiry. Terminal summaries may include outcome (success or error). |
Item counts are aggregated into the metric value. The plugin does not create one span or label value per streamed item and does not export item payloads through this summary.
session-cancellation counts cancellation-related journal records (request, consumer intent, and application), not unique logical cancellations. One cancellation can produce several records. A session result and its eventual stream or session terminal are separate events; an early result is not stream completion.
Collector delivery is best effort even though source lifecycle state advances when a batch is accepted.
For application-level observability with traces, logs, and metrics exported via OpenTelemetry, see the Observability page.