feat(sdks/go): stream reconnect and lifecycle for listeners#4257

Draft

igor-kupczynski wants to merge 8 commits into

mainfrom

stream-reconnect-lifecycle

igor-kupczynski commented Jun 22, 2026 •

edited

Loading

Contributor

Description

Implements PR #2 for issue #4228: stream reconnect and lifecycle for the legacy Go SDK (pkg/client). This PR stacks on #4240 (rubylabs-audit-go-sdk), which added REST read retries and shared backoff primitives; here we extend that foundation to gRPC stream listeners.

Long-lived gRPC subscriptions (workflow runs, durable events, worker actions, metadata streams) now use explicit app-level reconnect with full-jitter backoff instead of relying on the unary gRPC interceptor or ad-hoc hot loops. Sync paths (initial subscribe / AddWorkflowRun) use bounded reconnect; background listen loops reconnect unboundedly while the listener remains open. Permanent errors and consecutive no-progress failures still surface to callers.

Dependencies: #4240 (rubylabs-audit-go-sdk base branch)

Fixes #4228

Type of change

Bug fix (non-breaking change which fixes an issue)
Documentation change (pure documentation change)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
Refactor (non-breaking changes to code which doesn't change any behaviour)
CI (any automation pipeline changes)
Chore (changes which are not directly related to any business logic)
Test changes (add, refactor, improve or change a test)
This change requires a documentation update

What's Changed

Checklist

Changes have been:

Tested (unit, integration, or manually with steps specified)
Linted and formatted
Documented (where applicable)
Added to CHANGELOG (where applicable) -- see Keep a Changelog

Testing

go test ./pkg/client/retry ./pkg/client -count=1
Stream backoff / classifier unit tests (pkg/client/retry/stream_test.go)
Workflow listener: bounded sync subscribe, background reconnect past sync cap, EOF / no-handler guard, permanent error stop, handler replay, stale generation, grpc_retry.Disable() verification
Durable listener: bounded AddSignal, permanent error stop, reconnect during retry backoff
Worker dispatcher: >5 transient Recv failures, V2→V1 fallback, V1 Unimplemented terminal, ctx cancel
StreamByAdditionalMetadata recv reconnect (including EOF)
Durable task listener cancellation during sleep
Workflow.Result() fail-fast when subscribe fails

Related

Stacks on feat(sdks/go): add retry foundation for REST reads #4240 (rubylabs-audit-go-sdk) — retry foundation for REST reads
Part of issue Go SDK Retries #4228 (Go SDK retry / reconnect project)

Remaining risks

Unknown stream errors now stop / no-progress instead of hot-looping — may surface latent bugs sooner
Workflow.Result() has no caller ctx; relies on bounded sync reconnect only
Worker no longer terminates after 5 transient retries (intentional; documented in code)

🤖 AI Disclosure

I acknowledge that an LLM was used in the creation of this Pull Request, in accordance with Hatchet's AI_POLICY.md.
Details: Cursor (Claude) used for implementation, test coverage, review follow-ups, and PR description drafting across the Go SDK Retries #4228 PR chore: add root package.json with workspaces #2 scope.

vercel Bot commented Jun 22, 2026 •

edited

Loading

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
hatchet-docs	Ready	Preview, Comment	Jun 30, 2026 5:08pm

github-actions Bot added the engine label

vercel Bot deployed to Preview

June 22, 2026 13:20

View deployment

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

vercel Bot deployed to Preview

June 22, 2026 13:31

View deployment

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

vercel Bot deployed to Preview

June 22, 2026 17:18

View deployment

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

igor-kupczynski mentioned this pull request

Go SDK Retries #4228

Open

vercel Bot deployed to Preview

June 22, 2026 17:51

View deployment

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

vercel Bot deployed to Preview

June 22, 2026 18:03

View deployment

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=true). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

github-actions Bot commented Jun 22, 2026

Contributor

⚠️ Optional test failure: The load-deadlock job failed on this PR (optimistic-scheduling=false). This check is non-mandatory and does not block merging, but may be worth investigating. View logs

vercel Bot deployed to Preview

June 22, 2026 18:24

View deployment

vercel Bot deployed to Preview

June 24, 2026 09:48

View deployment

Base automatically changed from rubylabs-audit-go-sdk to main

June 24, 2026 10:04

igor-kupczynski added 6 commits

June 30, 2026 10:55


          feat(sdks/go): stream reconnect and lifecycle for listeners

339dd37


          fix(sdks/go): address stream reconnect review findings

fef5631

Stop unbounded background reconnect loops on NoProgress errors, pass caller
context into initial subscribe/listen setup, and make transient-failure
reconnect tests deterministic via a stream sleep test hook.


          fix(sdks/go): EOF reconnect and Result fail-fast per review

fb528fa


          fix(sdks/go): address Bugbot stream reconnect findings

af00a7f


          refactor(sdks/go): simplify stream reconnect implementation

82328f3


          fix(sdks/go): fix stream reconnect test failures

2d96f31

igor-kupczynski force-pushed the stream-reconnect-lifecycle branch from 27d4227 to 2d96f31 Compare

June 30, 2026 08:59

vercel Bot deployed to Preview

June 30, 2026 09:02

View deployment


          refactor(sdks/go): consolidate stream reconnect lifecycle

25ebf46

Share listener reconnect state across stream consumers and split worker action listening into smaller lifecycle helpers.

vercel Bot deployed to Preview

June 30, 2026 10:09

View deployment


          refactor(sdks/go): share stream listen loop

cd4e22c

Move duplicated workflow and durable event listen loops into the shared reconnect helper and cover the behavior once at the policy layer.

github-actions Bot commented Jun 30, 2026

Contributor

Benchmark results

goos: linux
goarch: amd64
pkg: github.com/hatchet-dev/hatchet/pkg/scheduling/v1
cpu: AMD Ryzen 9 7950X3D 16-Core Processor          
              │ /tmp/old.txt │         /tmp/new.txt         │
              │    sec/op    │   sec/op     vs base         │
RateLimiter-8    48.80µ ± 8%   48.96µ ± 7%  ~ (p=1.000 n=6)

              │ /tmp/old.txt │         /tmp/new.txt          │
              │     B/op     │     B/op      vs base         │
RateLimiter-8   137.7Ki ± 0%   137.7Ki ± 0%  ~ (p=0.303 n=6)

              │ /tmp/old.txt │          /tmp/new.txt          │
              │  allocs/op   │  allocs/op   vs base           │
RateLimiter-8    1.022k ± 0%   1.022k ± 0%  ~ (p=1.000 n=6) ¹
¹ all samples are equal

_{Compared against main (6b176e1)}

vercel Bot deployed to Preview

June 30, 2026 17:08

View deployment

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

engine