Control the retry policy

Using Golem’s retry mechanism

Golem applies retry policies to retryable host operations and agent failures. The selected policy controls delays and retry limits. Permanent errors, permission failures, stack overflows, and traps are not retried.

A trap is permanent. Return an error for a transient condition and use a retry schedule. For application-level errors, the application must handle the returned error and apply that schedule; returning an error does not automatically activate a host retry policy.

Do not predicate retry policies on is-idempotent: the host no longer supplies that property. Remote-write eligibility does not guarantee exactly-once effects; the peer must enforce an idempotency key.

If the maximum number of retries is reached, the agent will be marked as failed and no further invocations will be possible on it.

During recovery, Golem replays the oplog and reuses results already recorded for preceding durable operations. Raising an unhandled exception does not cause those earlier operations to execute again.

Host retry policies govern retryable Golem operations and agent failures. They do not retry arbitrary application-level errors. Handle these errors in your application.

Raising a trap

Use a trap to stop an invocation on an unrecoverable condition. An explicit trap cannot be caught and is never retried.

TypeScript

Use the SDK’s trap function. It never returns:


import { trap } from "@golemcloud/golem-ts-sdk"
 
trap("Invalid persistent state")

Effect TypeScript

The Effect SDK does not currently expose a public trap helper. Use Effect.fail for application-level errors that your program can handle; it is not a trap. Retry.withPolicy propagates these errors rather than retrying them.

Rust

The Rust SDK does not currently expose a public trap helper. Return a Result::Err for application-level errors that the caller can handle; it is not a trap.

Scala

Use HostApi.trap. It never returns:


import golem.HostApi
 
HostApi.trap("Invalid persistent state")

MoonBit

Import golemcloud/golem_sdk/api as @api, then call trap. It never returns:


@api.trap("Invalid persistent state")

Go

The Go SDK does not currently expose a public trap helper. Return an error for application-level errors that the caller can handle; it is not a trap.

Customizing the retry policy

The retry policy which controls the maximum number of retries and the exponential backoff is a global configuration of the Golem servers, but it can be customized for each agent.

TypeScript

The @golemcloud/golem-ts-sdk package exports the withRetryPolicy function to temporarily install a named retry policy for a block of code:


import {withRetryPolicy, type NamedRetryPolicy} from "@golemcloud/golem-ts-sdk"
 
// Durations are expected as nanoseconds
const policy: NamedRetryPolicy = {
  name: "temporary-policy",
  priority: 1000,
  predicate: {
    nodes: [{tag: "pred-true"}],
  },
  policy: {
    nodes: [
      {
        tag: "count-box",
        val: {
          maxRetries: 3,
          inner: 1,
        },
      },
      {
        tag: "clamp-delay",
        val: {
          minDelay: 100_000_000n, // 100 milliseconds
          maxDelay: 2_000_000_000n, // 2 seconds
          inner: 2,
        },
      },
      {
        tag: "exponential",
        val: {
          baseDelay: 100_000_000n,
          factor: 1.5,
        },
      },
    ],
  },
}
 
const result: string = withRetryPolicy(policy, () => {
  // this closure runs with the custom retry policy
  return "hello"
})

The policy passed to withRetryPolicy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


export type RetryPredicate = {
  nodes: PredicateNode[];
};
 
export type RetryPolicy = {
  nodes: PolicyNode[];
};
 
export type NamedRetryPolicy = {
  name: string;
  priority: number;
  predicate: RetryPredicate;
  policy: RetryPolicy;
};

Effect TypeScript

The Effect SDK provides a typed retry-policy DSL and Retry.withPolicy, which installs a policy for one Effect. It restores the previous policy with the same name when its scope closes, or removes the temporary policy if none existed:


import { Duration, Effect } from "effect"
import { Retry } from "@golemcloud/effect-golem"
 
const policy = Retry.NamedPolicy.named(
  "temporary-policy",
  Retry.Policy.exponential(Duration.millis(100), 1.5)
    .clamp(Duration.millis(100), Duration.seconds(2))
    .maxRetries(3),
).priority(1000)
 
const program = Retry.withPolicy(
  policy,
  Effect.succeed("hello"),
)

Use Retry.setPolicy(policy) instead when the policy should remain installed on the durable agent until it is overwritten or removed.

Rust

The golem_rust crate exports the with_retry_policy function to temporarily install a named retry policy for a block of code:


use golem_rust::bindings::golem::api::retry::{
    ClampConfig, CountBoxConfig, ExponentialConfig, PolicyNode, PredicateNode,
};
use golem_rust::retry::{with_retry_policy, NamedRetryPolicy, RetryPolicy, RetryPredicate};
 
let policy = NamedRetryPolicy {
    name: "temporary-policy".to_string(),
    priority: 1000,
    predicate: RetryPredicate {
        nodes: vec![PredicateNode::PredTrue],
    },
    policy: RetryPolicy {
        nodes: vec![
            PolicyNode::CountBox(CountBoxConfig {
                max_retries: 3,
                inner: 1,
            }),
            PolicyNode::ClampDelay(ClampConfig {
                min_delay: 100_000_000u64, // 100 milliseconds
                max_delay: 2_000_000_000u64, // 2 seconds
                inner: 2,
            }),
            PolicyNode::Exponential(ExponentialConfig {
                base_delay: 100_000_000u64,
                factor: 1.5,
            }),
        ],
    },
};
 
with_retry_policy(policy, || {
    "hello".to_string()
})

The policy passed to with_retry_policy is a NamedRetryPolicy, which contains a name, priority, predicate, and policy tree:


pub struct RetryPredicate {
    pub nodes: Vec<PredicateNode>,
}
 
pub struct RetryPolicy {
    pub nodes: Vec<PolicyNode>,
}
 
pub struct NamedRetryPolicy {
    pub name: String,
    pub priority: u32,
    pub predicate: RetryPredicate,
    pub policy: RetryPolicy,
}

Scala

In Scala, golem.Guards exposes withRetryPolicy to temporarily install a named retry policy for a Future-returning block:


import golem.Guards
import golem.host.Retry
 
import scala.concurrent.Future
import scala.concurrent.duration._
 
val policy = Retry.named(
  "temporary-policy",
  Retry.Policy
    .exponential(100.millis, 1.5)
    .clamp(100.millis, 2.seconds)
    .maxRetries(3)
).priority(1000)
 
val result: Future[String] =
  Guards.withRetryPolicy(policy) {
    Future.successful("hello")
  }

Retry.named creates a typed policy that matches all failures by default. The guard restores the previous policy when the returned Future completes.

MoonBit

Assuming the golemcloud/golem_sdk/api package is imported as @api, use the high-level policy builders and with_named_policy:


let policy = @api.NamedPolicy::named(
  "temporary-policy",
  @api.Policy::exponential(@api.Duration::millis(100), 1.5)
    .clamp(@api.Duration::millis(100), @api.Duration::seconds(2))
    .max_retries(3),
).priority(1000)
 
let result : String = @api.with_named_policy(policy, fn() {
  // this callback runs with the custom retry policy
  "hello"
}) catch {
  error => raise error
}

Go

The github.com/golemcloud/golem/sdks/go/golem/retry package provides a typed retry-policy builder and retry.With, which installs a named policy and returns a function that restores the previous policy with the same name, or removes the temporary policy if none existed. Use it with defer to scope the policy to a block of code:


import (
    "time"
 
    "github.com/golemcloud/golem/sdks/go/golem/retry"
)
 
policy := retry.Named(
    "temporary-policy",
    retry.Exponential(100*time.Millisecond, 1.5).
        Clamp(100*time.Millisecond, 2*time.Second).
        MaxRetries(3),
).WithPriority(1000)
 
result := func() string {
    defer retry.With(policy)()
    // this function runs with the custom retry policy
    return "hello"
}()

Use retry.Set(policy) instead when the policy should remain installed on the durable agent until it is overwritten or removed with retry.Remove. retry.With and retry.Set panic on an invalid policy; policy.Validate() reports the problem as an error instead.

A named policy retries host operations. To retry a piece of the agent’s own code, run a policy in the agent: Policy.Schedule() checks it, and Schedule.Retry calls an operation until it returns nil or the policy gives up, with durable sleeps between the attempts:


sched, err := retry.Exponential(100*time.Millisecond, 2).MaxRetries(4).Schedule()
if err != nil {
    return err
}
err = sched.Retry(func() error { return charge(order) })

Named retry policies in golem.yaml

Retry policies can also be defined declaratively in the application manifest (golem.yaml). These named retry policies are created in the environment during deployment, making them available to all agents without requiring code changes.


retryPolicyDefaults:
  local:
    default-retry:
      priority: 10
      predicate: true
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: { secs: 1, nanos: 0 }
              factor: 2.0

Manifest policy types reference

The following policy types can be used to construct retry policy trees:

Type	Fields	Description
`"immediate"`	—	Retry immediately
`"never"`	—	Never retry
`periodic`	`{ secs, nanos }`	Fixed delay between retries
`exponential`	`{ baseDelay, factor }`	Exponentially increasing delay
`fibonacci`	`{ first, second }`	Fibonacci-sequence delays
`countBox`	`{ maxRetries, inner }`	Limit total retry count
`timeBox`	`{ limit, inner }`	Limit total retry time
`clamp`	`{ minDelay, maxDelay, inner }`	Clamp delay range
`addDelay`	`{ delay, inner }`	Add fixed delay to inner policy
`jitter`	`{ factor, inner }`	Add random jitter
`filteredOn`	`{ predicate, inner }`	Apply only when predicate matches
`andThen`	`[policy1, policy2]`	Sequential composition
`union`	`[policy1, policy2]`	Union of two policies
`intersect`	`[policy1, policy2]`	Intersection of two policies

Manifest predicates reference

Predicates control when a retry policy is applied. The following predicate types are available:

Type	Fields	Description
`"true"`	—	Always matches
`"false"`	—	Never matches
`propEq`	`{ property, value }`	Property equals value
`propNeq`	`{ property, value }`	Property not equal
`propGt` / `propGte`	`{ property, value }`	Greater than / greater or equal
`propLt` / `propLte`	`{ property, value }`	Less than / less or equal
`propExists`	string	Property exists
`propIn`	`{ property, values }`	Property in set
`propMatches`	`{ property, pattern }`	Glob match
`propStartsWith`	`{ property, prefix }`	Starts with prefix
`propContains`	`{ property, substring }`	Contains substring
`and`	`[pred1, pred2]`	Logical AND
`or`	`[pred1, pred2]`	Logical OR
`not`	predicate	Logical NOT

Predicate values in the manifest are strings, integers, or booleans. For example:


predicate:
  propEq:
    property: status-code
    value: 503