Configuring Semantic Retry Policies (Go)

Golem retries retryable host operations (RPC, HTTP, RDBMS) and classified transient failures according to named policy rules. Each rule is a retry.NamedPolicy: a retry.Policy (the strategy) plus a retry.Predicate deciding when it applies and a priority. Per operation, the runtime resolves the highest-priority rule whose predicate matches and applies its policy.

Rules come from three tiers, and the Go retry subpackage (github.com/golemcloud/golem/sdks/go/golem/retry) works with all of them:

• Manifest (golem.yaml) or CLI — operator-tuned base rules; the code reads them back.
• Code-owned — retry.Set(...) registers a rule that persists for the agent.
• Per-call — retry.With(...) overrides a rule for one call, restored on return.

Steps

1. Declare base rules in golem.yaml (retryPolicyDefaults), scoped per environment.
2. Build strategies in code from a base policy plus fluent combinators.
3. Apply them: retry.Set for code-owned rules (often in the constructor), retry.With for per-call overrides.
4. Query active rules with retry.ByName / retry.Policies / retry.PolicyNames.
5. Retry your own code with Policy.Schedule() and Schedule.Retry.

1. Define Retry Policies in the Application Manifest

Add definitions under retryPolicyDefaults in golem.yaml, scoped per environment. This is platform-level config (shared with the other language SDKs):

retryPolicyDefaults:
  prod:
    http-transient:
      priority: 10
      predicate:
        and:
          - propEq: { property: "error-type", value: "transient" }
          - propEq: { property: "uri-scheme", value: "https" }
      policy:
        countBox:
          maxRetries: 5
          inner:
            jitter:
              factor: 0.15
              inner:
                clamp:
                  minDelay: "100ms"
                  maxDelay: "5s"
                  inner:
                    exponential:
                      baseDelay: "200ms"
                      factor: 2.0
    catch-all:
      priority: 0
      predicate: true
      policy:
        countBox:
          maxRetries: 3
          inner:
            exponential:
              baseDelay: "100ms"
              factor: 3.0

Policies are evaluated in descending priority order; the first matching predicate’s policy wins. If none match, the built-in default (3 retries, exponential backoff, clamped to [100ms, 1s], 15% jitter) applies.

2. Build and Apply Policies in Code

Use the retry subpackage. A Policy/Predicate/NamedPolicy is an immutable value; combinators return a new value, so malformed graphs are unrepresentable.

import (
	"time"
 
	"github.com/golemcloud/golem/sdks/go/golem/retry"
)
 
pol := retry.Exponential(200*time.Millisecond, 2.0).
	Clamp(100*time.Millisecond, 5*time.Second).
	WithJitter(0.15).
	OnlyWhen(retry.ErrorType.Eq("transient")).
	MaxRetries(5)
 
rule := retry.Named("http-transient", pol).
	WithPriority(10).
	AppliesWhen(retry.URIScheme.Eq("https"))

Base policies

Combinators (methods on Policy)

// Exponential, clamped, jittered, capped:
retry.Exponential(200*time.Millisecond, 2.0).
	Clamp(100*time.Millisecond, 5*time.Second).
	WithJitter(0.15).
	MaxRetries(5)
 
// Periodic with a wall-clock budget:
retry.Periodic(time.Second).Within(60 * time.Second)
 
// Immediate retries, then fall back to exponential:
retry.Immediate().MaxRetries(3).
	AndThen(retry.Exponential(time.Second, 2.0).MaxRetries(5))
 
// Never retry:
retry.Never()

Predicates

Build a predicate from a property (retry.PropName) using a comparison method, then compose with .And / .Or / .Not:

// A property comparison:
retry.ErrorType.Eq("transient")
retry.StatusCode.OneOf(500, 502, 503, 504)
 
// Compose:
retry.ErrorType.Eq("transient").And(retry.URIScheme.Eq("https"))
 
// Always / never:
retry.MatchAlways()
retry.MatchNever()

PropName comparison methods: .Eq, .Neq, .Gt, .Gte, .Lt, .Lte, .Exists(), .OneOf(...), .MatchesGlob(pat), .StartsWith(prefix), .Contains(sub). Comparison values may be a string, any integer type, or a bool.

Available properties

Do not predicate on is-idempotent; the host no longer supplies that property. The explicit golem:api/host.trap function raises a deterministic trap and is never retried, even by a catch-all policy. Host retry policies repair eligible individual calls, not completed effects or whole saga attempts.

A trap is permanent. Return an error for a transient condition and use a retry schedule. Application-level errors need application-level handling; returning an error does not automatically activate a host retry policy.

Exported retry.PropName constants (use retry.Prop("name") for any other):

A policy keyed on a property not present in the current context is silently skipped for that decision. So a StatusCode-keyed policy fires only for HTTP responses, a TrapType-keyed policy only from the trap path — design one policy per context (or use .Or/.And to match several) rather than expecting one policy to cover everything.

Apply: Set (code-owned) and With (per-call)

retry.Set registers/overwrites a rule and persists it for the agent (it takes precedence over a manifest/CLI rule of the same name). Reach for it when the code owns the rule — often in the constructor. Both Set and With panic on an invalid policy (check first with NamedPolicy.Validate()).

var codeOwned = retry.Named("code-owned",
	retry.Exponential(100*time.Millisecond, 2.0).
		Clamp(50*time.Millisecond, 2*time.Second).
		OnlyWhen(retry.StatusCode.Gte(500)).
		MaxRetries(4)).
	WithPriority(20).
	AppliesWhen(retry.TargetAgentType.Eq("InventoryAgent"))
 
var agent = retrying.Agent.Implement(func(retrying.ID) *state {
	retry.Set(codeOwned) // register the code-owned rule when the worker starts
	return &state{}
})

retry.With applies a rule for the current call only and returns a restore function that reinstates the previously registered rule (or removes it) — scope it with defer:

agent.Handle(retrying.Burst, func(_ *golem.Context[state], _ golem.Unit) retrying.Snapshot {
	// Override the code-owned rule just for this call, restored on return.
	defer retry.With(retry.Named("code-owned", retry.Immediate().MaxRetries(10)).
		WithPriority(99).
		AppliesWhen(retry.TargetAgentType.Eq("InventoryAgent")))()
	return doWork()
})

retry.Remove(name) deletes a rule.

Concurrency. Set, Remove, and With apply at the worker level, not per goroutine. With is safe as long as its scope does not await while other goroutines run concurrently; don’t hold an override open across a CallAsync fan-out.

3. Query Policies at Runtime

// Names only (cheap — does not decode bodies):
names := retry.PolicyNames()
 
// Fully decoded rules:
for _, p := range retry.Policies() {
	_ = p.Name()     // string
	_ = p.Priority() // uint32
}
 
// A specific rule by name:
if np, found := retry.ByName("http-transient"); found {
	_ = np.Priority()
}
 
// What policy the runtime would pick for a hypothetical operation; a property
// value is a string, an integer type or a bool:
pol, matched, err := retry.Resolve("GET", "https://api.example.com/x",
	map[string]any{"status-code": 503})
if err != nil {
	return err // Invalid selector; do not treat it as no matching policy.
}
_ = pol
_ = matched

NamedPolicy accessors: .Name(), .Priority(), .Strategy() (opaque Policy), .Applicability() (Predicate), and .Validate() (returns the first lowering error, or nil).

4. Live-Editing Policies via CLI

Retry policies can be managed at runtime without redeployment (takes effect immediately for running agents):

golem retry-policy create http-transient \
  --priority 10 \
  --predicate '{ "and": [{ "propEq": { "property": "error-type", "value": "transient" } }, { "propEq": { "property": "uri-scheme", "value": "https" } }] }' \
  --policy '{ "countBox": { "maxRetries": 5, "inner": { "exponential": { "baseDelay": "200ms", "factor": 2.0 } } } }'
 
golem retry-policy list
golem retry-policy get http-transient
golem retry-policy update http-transient --priority 15
golem retry-policy delete http-transient

5. Retry Your Own Code: Schedule

A host policy retries host operations (an HTTP request, an RPC call). To retry a piece of your own code — an operation that fails with your own error, or a response the host treats as a success — run a policy in the agent with Policy.Schedule():

sched, err := retry.Exponential(100*time.Millisecond, 2).
	OnlyWhen(retry.StatusCode.OneOf(502, 503)).
	MaxRetries(4).
	Schedule()
if err != nil {
	return err // the checks of Set, plus size bounds
}
sched = sched.WithProperties(func(err error) map[retry.PropName]any {
	var se *StatusError
	if errors.As(err, &se) {
		return map[retry.PropName]any{retry.StatusCode: se.Status}
	}
	return nil
})
 
body, err := sched.RetryValue(func() (string, error) { return fetchQuote(id) })
// or, with no result: err = sched.Retry(func() error { return charge(order) })

• Retry runs the operation at least once, then again until it returns nil or the policy gives up, and returns the operation’s last error unchanged.
• It installs no named policy and the host keeps no retry state: the loop is ordinary agent code. Its sleeps (time.Sleep) and the operation’s host calls are durable, so a replay neither sleeps nor calls again.
• WithProperties projects each failure to the properties the OnlyWhen predicates read; it runs after every failure. A predicate that needs a missing property gives up rather than falling back to another branch of the policy.
• A policy from the host works too: np, _ := retry.ByName("http-transient"); sched, err := np.Strategy().Schedule().
• Each Retry starts the policy from the beginning; a Schedule is a value you can keep as package data.

6. Default Retry Policy

When no user-defined policy matches, Golem uses the built-in default: priority 0, predicate true, up to 3 retries, exponential backoff (factor 3.0), delays clamped to [100ms, 1s], 15% jitter.

Key Constraints

• Policy/Predicate/NamedPolicy are immutable values; combinators return new values (build once as package data where possible).
• retry.Set and retry.With panic on an invalid policy — validate first with NamedPolicy.Validate() if the inputs are dynamic.
• Policies are defined per-environment; names must be unique within one; higher priority is evaluated first.
• A property-keyed policy is silently skipped in a context that lacks that property.
• retry.With is scoped to the current call — the previous rule is restored on return; it is worker-global, so mind concurrent awaits.
• A host policy retries host operations only. Retry your own code with Policy.Schedule() and Schedule.Retry / Schedule.RetryValue.

Related Guides