Go SDK Retries

[igor-kupczynski]

How the Go SDK retries today, and what I propose to change. This first part documents the current state only; proposed changes to follow.

Architecture

The v1 SDK (sdks/go/) is a thin wrapper over the legacy v0 client (pkg/client/). It adds no transport logic of its own — feature clients call the legacy REST client or the gRPC sub-clients directly, so all retry behavior lives in pkg/client/.

We have these categories of clients:

• gRPC unary calls — request/response RPCs.
• gRPC streams and listeners — long-lived subscriptions that Recv in a loop.
• REST / HTTP — the generated REST client.

Retry is configured in two places: the gRPC dial + interceptors in pkg/client/client.go, and the REST client, also built in client.go on the default http.Client with a request hook that adds the Authorization: Bearer <token> header and nothing else.

Current state

1. gRPC unary: single shared policy

All unary RPCs flow through one chained interceptor installed at dial time:

• Max 5 attempts.
• Exponential backoff with jitter (5s base, 10% jitter).
• 30s per-retry timeout.
• Retried codes: ResourceExhausted, DeadlineExceeded, FailedPrecondition, Internal, Unavailable.
• Globally disable-able via NoGrpcRetry.

This covers WorkflowService, AdminService v1, Dispatcher, V1Dispatcher, and EventsService. The SDK's Admin() holds both the v0 WorkflowService and v1 AdminService stubs, and both wrap the same retry conn — so every admin RPC the SDK issues is retried. This layer is uniform.

Note: a second, retry-less gRPC dialer exists at pkg/client/v1/grpc-client.go, but it is not an SDK path. It is hatchet-api's internal REST→gRPC proxy client, which forwards inbound REST calls to the engine's gRPC admin service. We consider it out of scope for this project.

2. gRPC streams and listeners: each is different

The retries here can mean two things:

• Automatic gRPC retry (interceptor) — a built-in gRPC retry layer beneath the stream (same 5x policy as section 1), handled by the library with no app code involved. When enabled, a transient gRPC error (either while opening the stream or after events have already been received) is handled without the caller seeing it. The interceptor drops the failed stream, opens a new one, re-sends the original subscribe request, and resumes delivery to the same Recv loop, for up to 5 attempts in total (the original plus up to 4 reopens, with backoff) before the error is finally returned. The retry opens a fresh subscription rather than resuming the broken one. Continuity across the gap is not guaranteed and events around the failure window can be dropped. When disabled (On/Stream pass grpc_retry.Disable()), there is no retry and the first error is returned immediately.
• Reconnect loop (hand-rolled) — application-level code that opens a fresh stream and re-subscribes after a live stream drops, layered on top of (or instead of) the interceptor.

Streams

Streams open a single gRPC stream, consume it in one Recv loop, and return on the first error the interceptor doesn't absorb. There is no hand-rolled reconnect, so recovery beyond that is up to the SDK consumer.

Stream	Automatic retry (5x)	On mid-stream error
On / Stream (SubscribeToWorkflowEvents)	Disabled (grpc_retry.Disable())	Returns on first non-EOF error
StreamByAdditionalMetadata (SubscribeToWorkflowEvents)	Enabled	Retriable codes retried ≤5x (stream re-opened, subscribe replayed); other errors returned

Listeners

Listeners are long-lived objects that own a reconnect loop and re-subscribe across many underlying streams over their lifetime.

Listener	What it delivers	Automatic retry (5x)	Reconnect loop (hand-rolled)	Recovery after giving up
Worker action listener Listen / ListenV2	Work assignments to the worker (the tasks it should run)	Enabled	Yes — up to 5 resubscribe attempts	None — it stops delivering work and raises a fatal error, so the worker stops processing. Recovery means restarting the worker.
WorkflowRunsListener	Run-completion notifications to callers awaiting a run	Disabled	Yes — stops after 10 consecutive failures	The loop stops and anything currently awaiting a run result hangs. Restarts on its own the next time the app awaits a run, re-subscribing the still-pending ones.
DurableEventsListener	Durable-event/signal deliveries to waiting durable tasks	Disabled	Yes — stops after 10 consecutive failures	Same as WorkflowRunsListener, but restarts the next time a durable task waits on an event.
DurableTaskListener (bidi)	Durable-task requests/responses between worker and engine	Disabled	Yes — unlimited, retries every 2s	N/A — never gives up.

3. REST / HTTP: no retries

The REST client is the default http.Client with no retry transport. How a non-2xx is surfaced varies by method.

REST operations with zero retry: CronClient/CronsClient (Create/Delete/List/Get), ScheduleClient/SchedulesClient (Create/Delete/List/Get), RunsClient (REST parts; GetDetails and the stream path are gRPC), WorkflowsClient, WorkersClient, MetricsClient, RateLimitsClient (REST parts), CELClient, FiltersClient, LogsClient, WebhooksClient, and all cloudrest endpoints.

Discrepancies

1. REST / HTTP performs no retries.
2. Most streaming listeners stop their active loop after errors. The worker action listener stops on the first reconnect that fully fails (the worker returns on that first error); WorkflowRunsListener and DurableEventsListener (after 10 consecutive) exit their listen goroutine, stranding in-flight waiters. The latter two are not permanently dead, as a later AddWorkflowRun / AddSignal re-arms the loop and re-subscribes still-registered handlers. There is no automatic, transparent recovery. Only DurableTaskListener reconnects on its own.
3. The same capability retries differently depending on which API you call. Several features are reachable over both REST (no retry) and gRPC (5x), so retry behavior depends on the entry point, not the operation:
   • Schedules / crons — the public Schedules() / Crons() methods (Create / Delete / List / Get) are REST (0 retry), but schedules/crons defined in code on a workflow (registered over gRPC) or created through the legacy gRPC scheduling call are retried (5x).
   • Run data — fetching full run details over gRPC (GetRunDetails) is retried (5x), while the public REST reads (Runs().Get / GetStatus / List) are not (0).
   • Rate limits — defining a rate limit over gRPC (PutRateLimit) is retried (5x), while listing them over REST is not (0).
4. Stream interceptor retry is applied inconsistently. On/Stream pass grpc_retry.Disable(), while StreamByAdditionalMetadata and the worker Listen/ListenV2 streams leave the interceptor enabled — so retriable Recv failures are silently retried (≤5x, re-opening the stream) on some streams but propagate immediately on others within the same layer.
5. Reconnect logic is duplicated and uncentralized. Four hand-rolled reconnect loops use different constants (5 vs 10 thresholds; fixed 1s/2s/5s sleeps) with no shared code, and none reuse the interceptor's exponential backoff.

Comparison with TypeScript / Python / Ruby

A quick cross-SDK pass shows the same patterns and the same or worse inconsistencies as Go:

• Go is the only SDK that retries unary gRPC via a dial-time interceptor, so its unary coverage is uniform and its retried codes are an explicit allowlist. TS (retrier wrapper) and Python (tenacity decorator) retry per call-site, so coverage is patchy and they retry on almost any error. Ruby has no unary retry at all.
• Go is the only SDK with a per-attempt deadline (30s). TS/Python/Ruby unary calls and most REST calls have no timeout and can hang (Ruby REST is the lone exception, 60s default).
• REST is unretried in Go/TS/Ruby; only Python retries REST (reads only, and oddly 404s too).
• Stream reconnect is hand-rolled with different constants in every SDK — so discrepancies chore: add root package.json with workspaces  #2 and feat: improve debugging experience of workflow runs #5 above are cross-SDK, not Go-specific.
• gRPC keepalive is identical across all four SDKs (10s / 60s).

I suggest that after we align the Go SDK we will port the same changes to the other SDKs.

Proposed changes

Key tenets

Two different retry policies (for reads and for writes)

• Retry policy is keyed on operation semantics: whether a call is retried depends on idempotency.
• The SDK auto-retries idempotent reads, plus the one safe write case: a dial/connection failure where the request never reached the server.
• We'd need idempotency keys to safely retry writes in general.
• The policy doesn't depend on transport: the same policy applies whether the call goes over REST or gRPC.

Respect the caller's context

This is idiomatic Go and also the principle of least surprise for callers. The SDK will retry, but only as long as the caller's context allows it.

Centralized retry primitives

A single backoff/jitter primitive replaces the scattered interceptor + four separate reconnect loops, shared by the unary interceptor, the REST RoundTripper, and the stream reconnect loop.

Streams reconnect as long as they are wanted

• Worker-owned infra streams reconnect forever.
• Caller subscriptions reconnect while the caller's context is live.
• We don't change the lossy reconnect in streams (i.e., after a reconnect we get only new events, there's no replay of missed events).

Behavior

• No new SDK APIs.
  • The SDK absorbs the safe "not sent" case internally (gRPC transparent retry; REST dial-retry).
  • By the time an error reaches the caller it's either "may have executed" (DeadlineExceeded, post-send Unavailable) or "won't succeed as-is" (InvalidArgument, AlreadyExists → existing DedupeViolationErr, …).
  • Callers tell these apart with status.Code(err) (gRPC) and errors.As(err, &net.OpError{}) (REST); we just guarantee the underlying error is never flattened.
  • No new error types or signatures.
• Retry only genuinely transient codes: Unavailable, ResourceExhausted, Internal, DeadlineExceeded.
  • Drop FailedPrecondition: wrong system state, not a transient fault.
  • Keep Internal: it usually wraps a transient transport error (HTTP/2 reset, proxy/LB hiccup), and only idempotent reads retry, so the double-execute risk never applies.
  • DeadlineExceeded only while the caller's context still has budget.
• REST retries only 502/503/504.
  • Not 4xx.
  • Bonus: REST honors 429/Retry-After.
• Full jitter replaces today's 10% jitter.
• One reconnect loop for streams.
  • Replaces the four per-listener loops (Listen/ListenV2, WorkflowRunsListener, DurableEventsListener, DurableTaskListener).
  • Drops their ad-hoc constants (5 vs 10 thresholds; 1s/2s/5s sleeps).

Defaults

• Two backoff profiles, both full jitter, factor 2x (sync calls and long-lived streams want opposite things).
  • Unary + REST: base 250ms, per-delay cap 5s, max 4 retries; hard-bounded by the caller's context deadline. ~30s per-attempt backstop only when no deadline is set; lowers today's 5s base.
  • Streams: base 1s, per-delay cap 30s, no attempt cap (reconnect while wanted).
• Config stays internal with sane defaults.
  • Exposed controls: the context deadline (the real per-call bound) and one on/off escape hatch (today's NoGrpcRetry, generalized to "disable SDK retries").
  • No per-call attempt/backoff/code knobs now; add a functional-option config later if demand appears (non-breaking).

Out of scope (deferred)

• Load-shedding during a brownout: retry budget, circuit breaker
  • The gRPC hot path already runs uniform 5x retries with no budget today, so this work doesn't regress it.
  • Deffered for now, but I'd like to revisit it soon after this one (gRPC's native retryThrottling is the near-free first step; a circuit breaker is a separate, more invasive feature we likely don't need).
• Idempotency keys on RunWorkflow/Push — the future unlock that would make write retries safe and let the SDK be the single retry layer for writes too.
• The internal REST→gRPC proxy client at pkg/client/v1/grpc-client.go (not an SDK path).

[igor-kupczynski]

Proposed PR plan

After digging through the Go SDK retry paths and doing a couple of adversarial passes on the scope, I think we should split this into a staged sequence. The first PR should improve the retry foundation without forcing the most contentious behavior change: method-level gRPC retry classification for writes.

Today all user-facing SDK gRPC stubs are built from pkg/client/client.go, where a single unary interceptor retries every unary RPC, including writes. Tightening that by operation semantics is probably the right end state, but it changes failure behavior for existing users and should be reviewed as its own PR.

PR 1: Retry foundation and REST reads

Scope:

• Add shared retry/backoff helpers with context-aware sleep.
• Add a REST retry Doer / HTTP client wrapper for idempotent reads only.
• Retry GET/HEAD on 502, 503, 504, and transport errors.
• Add per-attempt REST timeouts so retryable calls do not hang forever when the caller provides no deadline.
• Retry GET/HEAD 429 by honoring valid capped Retry-After values and falling back to bounded exponential backoff with jitter when the header is missing, invalid, or oversized.
• Never sleep past caller context deadlines.
• Wire the transport into both main REST and cloud REST clients.
• Move gRPC retry construction behind the helper.
• Drop codes.FailedPrecondition from the gRPC retry code list.
• Switch gRPC retry backoff to full jitter while preserving the current base/cap timing, max attempts, and per-retry timeout.
• Add and document a generalized HATCHET_CLIENT_NO_RETRY kill switch; keep HATCHET_CLIENT_NO_GRPC_RETRY backward-compatible as gRPC-only.
• Keep gRPC unary method classification out of this PR.
• Do not lower the gRPC base delay to 250ms while writes still share the unary interceptor.

REST writes and all bodied REST requests should stay out of retry scope in this PR. In Go, an http.Request body is a one-shot io.ReadCloser unless GetBody is populated or the retry layer buffers and reconstructs the body. The generated requests do not set req.GetBody, so retrying POST/PUT/PATCH would need body replay plumbing and idempotency review. This is why idempotent-looking PATCH, POST, or DELETE cases are deferred rather than accidentally omitted.

FailedPrecondition audit result for PR 1:

• The only production server return of codes.FailedPrecondition is internal/services/dispatcher/server.go in DispatcherImpl.Heartbeat, when a worker has a prior ListenV2 listener session but IsActive is false.
• The only user-facing Go SDK path that observes it is the worker background heartbeat loop in pkg/client/dispatcher.go.
• Current transparent retry does not reopen the listener stream. It only blocks the heartbeat goroutine behind the global gRPC retry backoff while listener reconnect remains driven separately by ListenV2.Recv errors.
• Dropping it is therefore a low-risk semantic cleanup: FailedPrecondition is a rejected heartbeat / inactive-listener signal, not a transient transport fault.
• This should not be framed as reducing duplicate-write risk. The duplicate-write risk from retrying non-idempotent gRPC methods remains until the later semantic retry-policy PR.
• Optional follow-up, not a prerequisite: consider Python parity by explicitly treating heartbeat FailedPrecondition as a listener reconnect signal.

PR 1 tests:

• REST retry client tests using httptest: retries GET on 502/503/504, retries transport errors for reads, does not retry POST, preserves the final response body/status, closes or drains discarded bodies, keeps auth headers, and stops when the request context expires.
• REST timeout tests: a hung response is bounded by the per-attempt timeout when no caller deadline is set, and the caller deadline wins when it is shorter.
• REST 429 tests: parses delta-seconds and HTTP-date Retry-After, falls back to jittered exponential backoff for missing/invalid/oversized headers, applies context deadline truncation, and does not retry POST.
• Backoff tests using deterministic randomness or injected clock/random source so full-jitter behavior is testable without sleeps.
• gRPC backoff tests verifying the retry helper uses full jitter while preserving the intended base/cap timing.
• gRPC retry-code tests verifying FailedPrecondition is no longer retryable while the remaining current retry codes still are.
• Config tests verifying HATCHET_CLIENT_NO_RETRY disables both retry layers and HATCHET_CLIENT_NO_GRPC_RETRY remains gRPC-only.

Status update (2026-06-22): PR 1 is implemented in #4240. PR 2 is implemented in #4257 and should be treated as the current source of truth for stream reconnect mechanics. The original uniform "reconnect while wanted" plan was refined during implementation: sync callers need bounded reconnect, singleton listeners need lifecycle contexts, and EOF/NoProgress require per-stream rules. PR 2 still intentionally excludes gRPC unary method classification, REST write retries, and retry status-code changes.

PR 2: Stream reconnect and lifecycle

Build on the PR 1 backoff helper and address the long-lived stream/listener discrepancies. PR #4257 implements this intent, but the implementation clarified several mechanics that should be treated as the PR 2 design rather than incidental details:

• Split reconnect into two contracts:
  • Synchronous setup/send paths (AddWorkflowRun, AddSignal, initial listener setup, and send-on-existing-stream paths) use bounded reconnect attempts (StreamSyncMaxAttempts = 5) and return an error to the caller instead of blocking forever while the engine is down.
  • Background established-listener paths (listen() loops and worker action recv loops) reconnect unboundedly for classified transient stream errors while the listener lifecycle context is alive.
• Keep sync and background reconnect coordination separate. Sync reconnect must not coalesce with background reconnect through the same singleflight group, because callers need bounded return semantics while background loops own long-lived recovery.
• Separate caller context from listener lifecycle context:
  • Caller context bounds initial setup and synchronous operations.
  • WorkflowRunsListener and DurableEventsListener own dedicated lifecycle contexts that are cancelled by Close(), not by whichever caller first created the singleton listener.
• Disable the gRPC retry interceptor on long-lived streams that have application-level reconnect (Listen/ListenV2, StreamByAdditionalMetadata, and existing managed listener streams), so there is one recovery layer.
• Preserve singleflight + generation coalescing on workflow/durable listeners so concurrent reconnects coalesce without stale streams winning.
• Add explicit NoProgress handling. Unbounded background reconnect applies only to classified transient errors (Unavailable, Internal, DeadlineExceeded, ResourceExhausted). Unknown/plain errors and EOF/no-progress classifications stop after maxConsecutiveStreamNoProgress = 10 consecutive failures without a successful Recv; sync paths fail immediately on no-progress.
• Treat EOF per API, not as one universal rule:
  • WorkflowRunsListener / DurableEventsListener: reconnect on EOF only when handlers are still registered (shouldReconnectOnEOF).
  • StreamByAdditionalMetadata: reconnect on EOF while the caller context is live; EOF is stream death for the metadata subscription.
  • On / Stream: keep existing one-shot behavior and return nil on EOF.
• Remove the outer retry loop from Workflow.Result(). Bounded tolerance lives inside AddWorkflowRun; Result() registers once, removes the handler on error, and returns. Remaining gap: Result() still has no caller context, so the blocking result wait cannot be cancelled by the caller.
• Change worker action listener behavior intentionally: transient recv/subscribe failures no longer stop the worker after the previous fixed cap. Workers retry indefinitely on classified transient outages until Actions(ctx) is cancelled; permanent errors still surface on errCh, and V2→V1 fallback remains one-way. This is an operational behavior change because Kubernetes-style restart-on-panic may no longer fire during prolonged engine outages, so PR 2 should include clear structured logs when the worker listener enters reconnect/degraded state and when it recovers. Do not add a new public health/state API for this.
• Keep DurableTaskListener mostly frozen in PR 2. Only switch reconnect sleeps to cancellable retry.Sleep; do not apply the new stream classifier, exponential backoff, or permanent-error stop yet. Durable task reconnect has pending ack/callback survival and EOF semantics that need a dedicated follow-up design.

These are good plan changes, not terrible ideas. The main caution is the worker action listener semantic change: it is intentional and likely better for resiliency, but it should be documented as operator-visible and covered by logging in PR 2 rather than by adding a separate public health signal.

PR 2 tests:

• Listener tests in pkg/client/listener_test.go and pkg/client/durable_listener_test.go: bounded sync reconnect returns when the engine is down, background reconnect continues past the sync cap for classified transient errors, context cancellation stops promptly, closed listeners do not reconnect, replayed handlers survive reconnect, and stale generations do not overwrite newer streams.
• EOF/no-progress tests: metadata stream reconnects on EOF while caller context is live; workflow/durable listeners reconnect on EOF only with registered handlers; On / Stream remain one-shot; no-progress failures stop after the consecutive cap in background loops and fail immediately in sync paths.
• Workflow.Result() tests verifying it calls AddWorkflowRun once, removes the handler on registration error, and relies on bounded reconnect inside AddWorkflowRun rather than an outer retry loop.
• Worker action listener tests around more than five transient subscribe/recv failures, permanent error surfacing, V2-to-V1 fallback, V1 Unimplemented terminal behavior, and context cancellation.
• Worker action listener logging tests or assertions verifying prolonged reconnect/degraded state and recovery are visible through structured logs, without adding a public health/state API.
• Durable task listener tests verifying reconnect sleep is cancellable without otherwise changing durable-task stream semantics.
• Stream retry tests should use the stream sleep test hook to avoid real backoff delays. Tests using the global sleep hook must not run with t.Parallel().

Concrete follow-ups after PR 2:

• PR 2B: Caller-facing waits and one-shot stream semantics. Own PR, not part of feat(sdks/go): stream reconnect and lifecycle for listeners #4257. Add Workflow.ResultWithContext(ctx) so callers can cancel the blocking result wait. In the same PR, settle the On / Stream transient Recv policy because those APIs are also caller-owned, context-bound subscriptions. If On / Stream reconnect semantics become contentious, split them out, but do not leave them as an implicit PR 2 obligation.
• PR 2C: Durable task reconnect hardening. Own design/implementation PR. Do not bundle with feat(sdks/go): stream reconnect and lifecycle for listeners #4257. Durable task reconnect has pending event acks, eviction acks, callback survival, buffered completions, and EOF behavior that conflict with the generic stream classifier. Scope this PR to jitter/backoff, terminal error handling, and pending callback/ack cleanup semantics.
• PR 3: gRPC semantic retry policy. Remains the contentious unary method classification PR. It should not absorb the PR 2B/2C follow-ups unless a change is directly needed for method-level retry classification.

PR 3: gRPC semantic retry policy

This is the contentious behavior change and should stand alone:

• Classify unary gRPC methods by retry safety.
• Stop retrying clearly non-idempotent writes by default, including TriggerWorkflow, BulkTriggerWorkflow, EventsService.Push, BulkPush, ScheduleWorkflow, Register, SendStepActionEvent, and SendGroupKeyActionEvent.
• Keep retries for clear reads such as GetRunDetails and GetVersion.
• Decide explicitly whether low-risk idempotent writes like PutRateLimit, Heartbeat, Unsubscribe, ReleaseSlot, and UpsertWorkerLabels stay retried.
• After write classification lands, consider lowering unary backoff to the proposed 250ms base and 5s cap.

PR 3 tests:

• gRPC method-classifier tests verifying non-idempotent writes are not retried and clear reads still are.
• Mock unary invoker tests: retryable read methods invoke multiple attempts on retryable status codes; non-idempotent write methods invoke once.

Explicit non-goals for this issue sequence

• PR 1 does not classify every gRPC unary method by idempotency or stop retrying non-idempotent gRPC writes by default; PR 3 handles that separately.
• PR 1 drops FailedPrecondition and switches gRPC to full jitter, but does not lower unary retry base delay to 250ms or change method-level retry behavior.
• We do not retry REST writes or other bodied requests in this sequence unless/until we add body replay support and operation-specific idempotency review.
• We do not add public per-call retry knobs or a full functional-option retry config.
• We do not add idempotency keys for RunWorkflow/Push or other write operations.
• We do not port the policy to TypeScript, Python, or Ruby in this issue.
• We do not add retry budgets, circuit breakers, or gRPC native retry throttling here.
• We do not touch the internal REST-to-gRPC proxy client at pkg/client/v1/grpc-client.go.
• PR 2 does not fully unify DurableTaskListener reconnect semantics; durable task keeps fixed-interval reconnect and existing pending ack/callback behavior except for cancellable sleeps.
• PR 2 does not add reconnect behavior to On / Stream; those remain one-shot.
• PR 2 does not add a caller-cancellable Workflow.Result() API.
• PR 2 does not own Workflow.ResultWithContext(ctx) or On / Stream transient reconnect semantics; PR 2B owns that follow-up.
• PR 2 does not own durable task reconnect hardening; PR 2C owns that follow-up.

[igor-kupczynski]

Progress: PR #4240 merged (first scoped slice).

The first scoped PR off this tracking issue, #4240 ("feat(sdks/go): add retry foundation for REST reads"), merged on 2026-06-24.

What it delivered:

• Shared retry/backoff helpers in pkg/client/retry/.
• REST: retries for bodyless GET/HEAD only — 5 attempts, full jitter (0-250ms / 0-500ms / 0-1s / 0-2s). Bodied requests (POST/PUT/PATCH) are excluded because the generated REST clients don't set req.GetBody, so body replay would need per-endpoint idempotency review.
• gRPC: behavior largely unchanged, but now uses full-jitter backoff (0-5s / 0-10s / 0-20s / 0-40s) and drops FailedPrecondition from the retryable codes (audit found it's only produced by DispatcherImpl.Heartbeat for inactive listeners — a rejected-heartbeat signal, not a transient fault).
• Controls: HATCHET_CLIENT_NO_RETRY disables both REST + gRPC SDK retries; HATCHET_CLIENT_NO_GRPC_RETRY stays gRPC-only.
• Review fixes during the PR: validate REST max-attempts >= 1; normalize gRPC backoff to zero-based attempts.

Still pending (deferred follow-ups from this issue):

• gRPC method-level write classification — duplicate-write risk on retried unary writes is unchanged.
• REST write retries (needs body replay + idempotency review).
• Stream/listener reconnect normalization — tracked in draft PR feat(sdks/go): stream reconnect and lifecycle for listeners #4257.