Repository navigation
fix: fail gateway requests when target silo dies - #10651
Closed
ReubenBond wants to merge 1 commit into
Closed
ReubenBond wants to merge 1 commit into
ReubenBond wants to merge 1 commit into
Conversation
Track gateway-forwarded client requests by destination silo and reject them promptly when membership declares that silo dead. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR improves Orleans runtime messaging behavior for external clients connected via a gateway by ensuring that requests forwarded to a silo which subsequently dies are failed promptly (via transient SiloUnavailableException) instead of waiting for the full client-side response timeout. This fits into the runtime’s liveness/messaging layer by extending the existing “break outstanding messages to dead silo” behavior to include gateway-forwarded client requests.
Changes:
- Track gateway-forwarded, non-one-way client requests (keyed by
CorrelationId+TargetSilo) and synthesize transient rejection responses when the destination silo is declared dead. - Plumb dead-silo notifications to the gateway (
SiloConnectionMaintainer) and add a fast-path rejection inMessageCenterfor gateway-forwarded requests targeting known-dead silos. - Add regression + unit tests covering breaking behavior, terminal vs status responses, destination removal, and TTL-based cleanup.
Show a summary per file
| File | Description |
|---|---|
| test/Orleans.Runtime.Tests/MembershipTests/SilosStopTests.cs | Adds a functional regression test verifying prompt failure of a gateway-forwarded client request when the target silo is killed. |
| test/Orleans.Runtime.Tests/ClientConnectionTests/GatewayInFlightRequestTrackerTests.cs | Adds focused unit tests for the new gateway in-flight request tracking behavior (terminal/status responses, dead-silo removal, TTL expiry). |
| src/Orleans.Runtime/Networking/SiloConnectionMaintainer.cs | Extends dead-silo handling to also break gateway-tracked client requests when a remote silo is declared dead. |
| src/Orleans.Runtime/Messaging/MessageCenter.cs | Hooks gateway request tracking into send path and adds an early rejection path for gateway client requests targeting dead silos. |
| src/Orleans.Runtime/Messaging/Gateway.cs | Implements per-client in-flight request tracking, periodic expiry cleanup, and dead-silo break logic which synthesizes transient rejections. |
Review details
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
- Files reviewed: 5/5 changed files
- Comments generated: 1
- Review effort level: Lite
Comment on lines
+269
to
+272
| foreach (var client in clients.Values) | ||
| { | ||
| client.BreakOutstandingMessagesToSilo(deadSilo); | ||
| } |
Member
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
External client requests forwarded through a gateway currently wait for their normal response timeout when the destination silo dies before responding. This change lets the gateway complete those calls promptly with the established transient
SiloUnavailableExceptionrejection.The gateway now tracks forwarded requests in each connected client's state after placement has selected the final
TargetSilo. Terminal responses and rejections remove entries through the same client-state delivery path, while status responses retain them. Membership death removes matching requests and delivers synthetic rejections through the owning client state, preserving one-way and client reply-routing behavior.Tracking is bounded by the request TTL or configured response timeout and is also cleared on completion, disconnect, client drop, and destination failure. A deterministic cluster regression pins a long-running request behind a gateway to a secondary silo, confirms execution has begun, kills that silo, and verifies prompt completion. Focused tests also cover terminal versus status responses, one-way exclusion, destination removal, and TTL cleanup.
Fixes #10165
Microsoft Reviewers: Open in CodeFlow