[Cross-SDK] Expose portable worker connection and lifecycle status

[YunchuWang]

Summary

Portable Durable Task workers generally report startup success before they have
connected to the backend. Today, customers do not have a supported API for
observing the worker's lifecycle or connection state and must infer it from logs.

This feature request asks for a consistent, public way across the .NET, Python,
Java, and JavaScript/TypeScript SDKs to determine whether a worker is starting,
connected, reconnecting, stopping, stopped, or permanently faulted.

Customer problem

Customers running workers in Kubernetes, Azure Container Apps, worker services,
or other self-hosted environments may want to:

• expose an application readiness or health endpoint;
• wait for a worker connection before accepting dependent work;
• alert when a worker has been disconnected for too long;
• coordinate shutdown, failover, or custom lifecycle management; and
• distinguish "the worker loop was launched" from "the backend connection is active."

Parsing log messages is not a stable API, and transport channel state alone does
not prove that the worker's GetWorkItems stream is active.

Desired outcome

Provide at least one supported, typed mechanism for customers to observe worker
lifecycle and backend-connection state, with equivalent semantics across the
portable SDKs.

Existing startup behavior does not need to change. In particular, this request
does not require start() or host startup to block until connected.

Design options to consider

The following are alternative or independently adoptable approaches. This
feature does not require implementing all of them.

1. Pollable status snapshot
   Expose a current status/state property, potentially with timestamps and the
   last sanitized connection error.

2. Status transition notifications
   Expose an idiomatic event, listener, callback, observable, or async iterator
   for customers that do not want to poll.

3. Explicit wait helper
   Add an opt-in WaitUntilConnected/wait_until_connected API with timeout and
   cancellation, while preserving nonblocking startup by default.

4. Protocol-backed readiness
   If local SDK signals are not strong enough, add an explicit backend
   acknowledgment that the work-item stream has been accepted and registered.
   This would be a larger, separately coordinated protocol change.

Possible common states include NotStarted, Connecting, Connected,
Reconnecting, Stopping, Stopped, and terminal Faulted. Exact naming and
which option provides the best cost/value tradeoff are open for design.

Scope

• Align semantics across durabletask-dotnet, durabletask-python,
  durabletask-java, and durabletask-js.
• Avoid requiring customers to parse logs or consume raw gRPC/protobuf types.
• Clearly distinguish transient reconnecting from a worker loop that has
  permanently terminated.
• Document what "Connected" or "Ready" proves and how quickly a half-open
  connection is detected.

Detailed cross-SDK findings and design tradeoffs are included in a follow-up
comment.

Related work

• Worker can silently hang forever when starting durabletask-js#69
• fix: bound worker hello and cancel reconnect backoff durabletask-js#353
• Worker stops permanently after peer resets GetWorkItems stream durabletask-python#175
• Reconnect workers after peer stream cancellation durabletask-python#176
• [Feature] Consider Adding API to DurableTaskClient for Fetching Task Hub Metadata #336

[YunchuWang]

Detailed cross-SDK findings

Research date: 2026-08-13. The links below are pinned to the default-branch
revisions examined.

The official portable SDK matrix currently contains .NET, Python, Java, and
JavaScript/TypeScript:
https://learn.microsoft.com/azure/durable-task/sdks/durable-task-overview

Go is also available experimentally for self-hosted scenarios, so it is noted
separately but should not necessarily be part of the initial conformance scope.

Current startup and reconnect behavior

SDK	What startup currently means	Connection and reconnect behavior	Public worker status API
.NET	The worker is a BackgroundService. Generic Host startup schedules its long-running ExecuteAsync; it does not wait for backend connectivity.	A bounded Hello handshake is followed by the work-item stream. Connection failures use full-jitter exponential backoff and retry indefinitely until host shutdown. Repeated transport failures can recreate an SDK-owned channel, and silent streams are detected.	None
Python	TaskHubGrpcWorker.start() creates a background thread, sets a private running flag, and returns before that thread connects.	Uses a bounded Hello, full-jitter exponential reconnect backoff, channel recreation after repeated transport failures, and indefinite retries until stop().	None; _is_running is private
Java	start() launches a background thread and returns. startAndBlock() blocks for the worker lifetime, not merely until the first connection.	It opens GetWorkItems without a separate Hello handshake and retries failures every five seconds.	None
JavaScript/TypeScript	await worker.start() launches a fire-and-forget loop and returns before Hello or stream readiness.	On current main, an immediate cold-start Hello failure terminates the loop while _isRunning remains true; after a stream was established, stream loss uses indefinite exponential reconnect. Open PR #353 adds bounded Hello attempts, cancellable retry, and indefinite cold-start retry while intentionally preserving nonblocking start().	None; _isRunning is private

Sources:

• .NET worker base:
  https://github.com/microsoft/durabletask-dotnet/blob/ab4be1251f5977d019f3120d96d7dd889d4d7f12/src/Worker/Core/Hosting/DurableTaskWorker.cs
• .NET gRPC worker:
  https://github.com/microsoft/durabletask-dotnet/blob/ab4be1251f5977d019f3120d96d7dd889d4d7f12/src/Worker/Grpc/GrpcDurableTaskWorker.cs
• Python start():
  https://github.com/microsoft/durabletask-python/blob/95e07470d637037ca4a7ad6c93684a17da35a887/durabletask/worker.py#L751-L768
• Java lifecycle methods:
  https://github.com/microsoft/durabletask-java/blob/f6c9681fe07c99315fedcc1c9db1fb96cbb173d3/client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcWorker.java#L124-L168
• Java retry loop:
  https://github.com/microsoft/durabletask-java/blob/f6c9681fe07c99315fedcc1c9db1fb96cbb173d3/client/src/main/java/com/microsoft/durabletask/DurableTaskGrpcWorker.java#L543-L558
• JavaScript start():
  https://github.com/microsoft/durabletask-js/blob/0d132c59431bba8535857818ab26653f7874fc1d/packages/durabletask-js/src/worker/task-hub-grpc-worker.ts#L386-L403
• JavaScript first-attempt failure path:
  https://github.com/microsoft/durabletask-js/blob/0d132c59431bba8535857818ab26653f7874fc1d/packages/durabletask-js/src/worker/task-hub-grpc-worker.ts#L491-L505
• JavaScript retry hardening:
  fix: bound worker hello and cancel reconnect backoff durabletask-js#353

Experimental Go behavior

The experimental Go client is the one exception to the dominant startup model.
StartWorkItemListener performs its first Hello and GetWorkItems setup
synchronously. It returns the first connection error to the caller. Once that
initial setup succeeds, later reconnects happen in a background goroutine with
indefinite backoff.

Source:
https://github.com/microsoft/durabletask-go/blob/9c9e2d6d4cc3609c28bc2cc660ab5311f0217593/client/worker_grpc.go

Existing signals are internal or log-based

All four SDKs emit connection or retry log messages, and JavaScript supports a
structured logger. These can be observed, but they are not a typed lifecycle
contract and their event/message details are not an appropriate customer state
API.

The shared protocol contains:

• a backend Ping RPC:
  https://github.com/microsoft/durabletask-protobuf/blob/67a2f1d3b9ea8a210eb86addb636a5d8a6484dbc/protos/backend_service.proto#L100-L104
• an inbound HealthPing work-item message:
  https://github.com/microsoft/durabletask-protobuf/blob/67a2f1d3b9ea8a210eb86addb636a5d8a6484dbc/protos/orchestrator_service.proto#L877-L894

Neither is exposed as public worker status. Ping proves backend reachability,
not that this worker's work-item stream is registered. HealthPing is consumed
internally as stream traffic.

The distinction matters in JavaScript today: the "worker connected" log is
written after Hello succeeds and a lazy gRPC stream object is created, before
response metadata, a health ping, or a work item is received:
https://github.com/microsoft/durabletask-js/blob/0d132c59431bba8535857818ab26653f7874fc1d/packages/durabletask-js/src/worker/task-hub-grpc-worker.ts#L405-L420

Design tradeoffs

These are options to evaluate, not cumulative requirements.

Option A: pollable status snapshot

Example data could include state, state-change time, last-connected time, last
successful backend contact, advisory retry attempt, and a sanitized SDK-owned
error code/message.

Pros:

• Small public surface and relatively low implementation cost.
• Directly supports synchronous readiness/health endpoints.
• Does not require a backend or protocol change.

Cons:

• Customers must poll and choose their own interval.
• Short transitions can be missed.
• Every customer needing startup coordination must implement timeout and race
  handling.

Option B: transition subscription

Each language could use its idiomatic mechanism: .NET event/observable/async
enumerable, Python callback or iterator, Java listener, and JavaScript event or
async iterator.

Pros:

• Good for alerting and explicit lifecycle management.
• Avoids wasteful polling and preserves transition context.

Cons:

• More concurrency and disposal semantics must be specified.
• Slow or throwing subscribers must never block or terminate the worker loop.
• A replay-latest rule is needed so subscribing after start() does not miss
  the current state.

Option C: explicit wait-until-connected helper

Keep startup nonblocking and provide an opt-in wait with required timeout and
cancellation.

Pros:

• Directly solves startup coordination.
• Avoids changing established host/start behavior.
• Centralizes read/subscribe race handling inside the SDK.

Cons:

• Requires a precise definition of "Connected."
• Does not by itself support ongoing alerting unless paired with polling or
  notifications.

Option D: protocol-backed readiness acknowledgment

The backend explicitly acknowledges that GetWorkItems was accepted and the
worker was registered, or the protocol guarantees an immediate equivalent
signal.

Pros:

• Provides the strongest and most consistent readiness definition.
• Avoids treating Hello or lazy stream creation as worker readiness.

Cons:

• Requires protobuf and backend changes plus capability/version negotiation.
• Older backends need a documented fallback.
• Requires coordinated rollout across more repositories.

State-model considerations

If a state enum is selected, one possible model is:

• NotStarted: start has never been called.
• Connecting: start was called, but this run has never connected.
• Connected: the strongest agreed work-stream acceptance signal was observed.
• Reconnecting: this run was connected previously and retries are active.
• Stopping: shutdown is in progress.
• Stopped: shutdown completed and no attempts remain.
• Faulted: the worker loop terminated due to an unrecoverable error.

Faulted should not represent ordinary connection failures when the worker is
still retrying. Alerting for a prolonged outage would instead use
Connecting/Reconnecting plus elapsed time and last successful contact.

Important semantic questions:

• Does Connected mean Hello succeeded, response metadata arrived, or the
  first health/work message arrived?
• How stale can Connected become during a half-open connection?
• Are transitions published only when the state changes, or on every retry?
• Does a subscriber immediately receive the current snapshot?
• How are subscriber exceptions, ordering, unsubscription, and backpressure
  handled?
• Does waiting fail promptly when the worker becomes Stopped or Faulted?

Related evidence

• JavaScript issue Refactor TaskActivity and TaskOrchestrator base classes #69 explicitly says customers expect awaited startup to
  reflect readiness:
  Worker can silently hang forever when starting durabletask-js#69
• JavaScript PR DurableTaskClient throwing RpcException "An attempt was made to access a socket in a way forbidden by its access permissions" #353 intentionally preserves nonblocking startup and does not
  add readiness semantics:
  fix: bound worker hello and cancel reconnect backoff durabletask-js#353
• Python issue Refactor TaskEntity to TaskEntity<TState> #175 documents a process that remained healthy after its worker
  silently stopped receiving work. PR Patch GH automation #176 fixed reconnect behavior:
  Worker stops permanently after peer resets GetWorkItems stream durabletask-python#175
  Reconnect workers after peer stream cancellation durabletask-python#176
• .NET issue [Feature] Consider Adding API to DurableTaskClient for Fetching Task Hub Metadata #336 requests task-hub/backend health information for Kubernetes,
  but it is client/backend metadata rather than worker-stream state:
  [Feature] Consider Adding API to DurableTaskClient for Fetching Task Hub Metadata #336