fix(messaging): reject gateway requests to dead silos

[ReubenBond]

Fixes #10165.
Fixes #11335.

External clients own their callbacks outside the silo runtime. Gateway-forwarded requests therefore waited for the client response timeout when their destination silo died, even though membership could promptly fail calls originating inside the cluster.

Track external-client requests by destination and attempt at the gateway's concrete remote transport enqueue. Per-client synchronization makes registration and enqueue atomic with membership-driven removal. When membership declares the destination dead, remove its outstanding requests and return transient rejections backed by SiloUnavailableException. Gateway-addressed system targets and one-way messages retain their existing paths.

Preserve ingress-gateway ownership across forwarding, transport retries, and client reconnects. Transport retries retain a local callback to the original client state and recheck the attempt, destination, forwarding generation, and original retention deadline under the enqueue lock. Removed or superseded attempts are suppressed with a debug diagnostic. Forwarding updates follow the source-matched hop chain, active retry attempts suppress stale responses, and terminal responses complete the ingress gateway's tracker. Mixed-version responses with a missing forward count complete when their sender is not a known earlier owner. Responses can reach a replacement gateway using bounded routing repair. If replacement lookup fails or selects a dead silo, retained client state queues the original response for reconnect. Disconnect releases destination ownership and retains bounded attempt markers for delayed sends after reconnect; completion, expiry, drop, and shutdown retire the remaining state.

Keep request snapshots lightweight, copy mutable cache-invalidation headers, and preserve explicit TTL semantics. Requests without a TTL use the silo response timeout for tracker retention. Responses arriving after tracker expiry remain eligible for delivery: the client callback owns the call deadline and duplicate recognition. This preserves longer client deadlines without adding tombstones. Forwarding-limit exhaustion follows the existing forwarding-failure rejection path. The delivery-semantics documentation explains prompt failure and the execution uncertainty applications reconcile when retrying.

Regression coverage exercises ownership generations, forwarding and migration, membership/send ordering, timeout and disconnect cleanup, and external-client rejection delivery. Fake-time cases cover expiry followed by terminal responses and older responses arriving during an active retry. Successful-completion coverage uses a tracking lifecycle event rather than a fixed execution window, directory fault injection verifies response retention during reconnect, and transport-retry coverage exercises the actual silo-connection retry path across active, retired, and released ownership states.

Microsoft Reviewers: Open in CodeFlow

[Copilot]

Pull request overview

This PR addresses a messaging gap in the Orleans gateway: gateway-forwarded external client requests were not tracked by destination silo, so when membership declared a target silo dead those requests could linger until normal client timeout instead of failing promptly with SiloUnavailableException. The change integrates per-client in-flight request tracking into the gateway and adds a regression test which exercises an external client through its gateway.

Changes:

• Track gateway-forwarded (non-local) client requests after addressing, expire them via existing maintenance, and clear them on disconnect/shutdown.
• Listen for SiloStatus.Dead notifications and reject tracked requests to the dead silo via the existing client response path.
• Add a functional liveness test which ensures a gateway-forwarded request breaks promptly when the destination silo is killed.

Show a summary per file

File	Description
test/Orleans.Runtime.Tests/MembershipTests/SilosStopTests.cs	Adds a regression which forces a gateway-forwarded in-flight request, then kills the destination silo and asserts prompt SiloUnavailableException.
src/Orleans.Runtime/Messaging/MessageCenter.cs	Hooks gateway request tracking into the outbound send path.
src/Orleans.Runtime/Messaging/Gateway.cs	Adds per-client in-flight request tracking, periodic expiry, dead-silo rejection, and lifecycle cleanup for forwarded requests.

Review details

Tip

Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Suppressed comments (1)

src/Orleans.Runtime/Messaging/Gateway.cs:536

• RejectRequestsToSilo includes the full request Message in the SiloUnavailableException text ("... for message: {request}"). Message.ToString() appends BodyObject, so this can leak request payload details back to external clients and also inflate rejection strings. Prefer a sanitized message which does not embed the full request (e.g., include only CorrelationId).

                    var exception = new SiloUnavailableException(
                        $"The target silo {deadSilo} became unavailable for message: {request}.");
                    _gateway.messageCenter.RejectMessage(

• Files reviewed: 3/3 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

test/Orleans.Runtime.Tests/MembershipTests/SilosStopTests.cs:65

• Test name is inconsistent with the existing pattern in this file ("...RequestsBreak" vs "...RequestBreaks"). Consider aligning the naming to reduce confusion when scanning similar tests.

        public async Task SiloUngracefulShutdown_GatewayForwardedRequestBreaks()

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

test/Orleans.Runtime.Tests/MembershipTests/SilosStopTests.cs:134

• LongRunningTaskObserver stores the call id in a mutable field and completes a non-generic TaskCompletionSource. If OnCallStarted were invoked more than once (e.g., due to retries or reentrancy), _callId could be overwritten after _started is completed, making the final Assert nondeterministic. Capture the call id as the TaskCompletionSource result instead.

            private readonly TaskCompletionSource _started = new(TaskCreationOptions.RunContinuationsAsynchronously);
            private Guid _callId;

            public void OnCallStarted(Guid callId)
            {

src/Orleans.Runtime/Messaging/Gateway.cs:463

• ClientState.TrackRequest reads the current connection via the Connection property, which returns the backing field without a volatile read. Since _connection is updated via Interlocked.Exchange, a non-volatile read here can observe a stale value and either skip tracking while connected (breaking the feature) or track after disconnect (leaking entries until TTL cleanup). Use Volatile.Read for both reads in this method.

            public void TrackRequest(Message message)
            {
                var connection = Connection;
                if (connection is null)
                {

• Files reviewed: 3/3 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

src/Orleans.Runtime/Messaging/Gateway.cs:478

• CreateRequestSnapshot assigns CacheInvalidationHeader by reference, which can alias the original message's mutable List. That list can be appended to later during forwarding/cache invalidation, so the snapshot can observe concurrent mutations and potentially race during serialization of the synthesized rejection. Copy the list when snapshotting to avoid sharing mutable state between messages.

                TargetSilo = message.TargetSilo,
                TargetGrain = message.TargetGrain,
                SendingSilo = message.SendingSilo,
                SendingGrain = message.SendingGrain,
                CacheInvalidationHeader = message.CacheInvalidationHeader,

src/Orleans.Runtime/Messaging/Gateway.cs:492

• ClearPendingRequests() is called on disconnect and gateway shutdown, but not when a client is dropped/removed (ClientState.Drop()). Since dropped clients are removed from Gateway.clients, the maintenance loop will no longer call DropExpiredRequests() for that ClientState, so any tracked _pendingRequests entries can be retained for the lifetime of the ClientState. Consider clearing _pendingRequests when dropping the client as well.

            public void ClearPendingRequests()
            {
                lock (_pendingRequests)
                {
                    _pendingRequests.Clear();

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Runtime/Messaging/Gateway.cs:236

• TrackRequest will also record SystemTarget requests if they have a non-local TargetSilo and a client SendingGrain. However, Gateway.TryToReroute explicitly allows SystemTarget routing via gateway addresses (not membership SiloAddress values), so these entries will never match SiloStatusChangeNotification (which reports membership silo addresses) and therefore cannot be rejected promptly when the silo dies. Consider skipping SystemTarget messages here to avoid tracking overhead and misleading entries.

            if (message.Direction != Message.Directions.Request
                || message.TargetSilo is not { } targetSilo
                || targetSilo.Matches(siloAddress)
                || !ClientGrainId.TryParse(message.SendingGrain, out var clientId)
                || !clients.TryGetValue(clientId, out var client))

• Files reviewed: 4/4 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 6/6 changed files
• Comments generated: 1
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Orleans.Runtime/Messaging/Gateway.cs:78

• requestMaintenancePeriod can become 1ms when messagingOptions.ResponseTimeout <= 0 (because Min returns a non-positive value and Max clamps to 1ms). Since non-positive response timeouts are explicitly supported (and tracking may still occur via per-message TTL), this can create an unnecessarily tight maintenance loop and avoidable CPU usage.

            var requestMaintenancePeriod = Max(
                TimeSpan.FromMilliseconds(1),
                Min(messagingOptions.ResponseTimeout, TimeSpan.FromSeconds(1)));

src/Orleans.Runtime/Messaging/Gateway.cs:132

• PerformRequestMaintenance logs exceptions using LogErrorGatewayMaintenanceError, which emits the message "Error performing gateway maintenance". That makes it hard to distinguish request-tracking maintenance failures from the existing gateway maintenance loop when diagnosing issues.

                catch (Exception exception)
                {
                    LogErrorGatewayMaintenanceError(logger, exception);
                }
            }

• Files reviewed: 6/6 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical message-disposal leaks remain in Gateway and MessageCenter paths.

Review effort: Lite
Findings: 5

Open (5)

• Forwarding status updates are not disposed · New
• Original message leaks when dead destination removes snapshot · New
• Failed forwarded request is not disposed · New
• Terminal forwarding updates are not disposed · New
• Failed forwarded client request is not disposed · New

Resolved since last review (2)

• Route failed terminal responses instead of dropping them
• Preserve tracking retention and hop history across forwarded attempts