Repository navigation
feat(messaging): process inbox messages - #11284
ReubenBond wants to merge 31 commits into
Conversation
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved lifecycle, selection-mutation, scheduling, recovery, retention, validation, and outbox-contract issues remain.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 2
Open (3)
What changed in this PR
Adds a non-packable Durable Messaging inbox runtime integrated with Journaling and Durable Jobs, including recoverable ownership, transactional handler processing, retries, deduplication, and dead-lettering.
Changes:
- Adds durable messaging contracts, routing, diagnostics, and configuration.
- Implements journal-backed acceptance, recovery, pump coordination, and handler processing.
- Adds Journaling integration and comprehensive contract and functional tests.
| File | Reviewed change |
|---|---|
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbeTests.cs |
Snapshot probe tests |
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs |
Snapshot waiting helper |
test/Orleans.DurableMessaging.Tests/Support/ReceiverTestServices.cs |
Test service composition |
test/Orleans.DurableMessaging.Tests/Support/JournaledTestOutbox.cs |
Journal-backed test outbox |
test/Orleans.DurableMessaging.Tests/Support/HandlerProbe.cs |
Handler synchronization helper |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingMetricProbe.cs |
Metrics test probe |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingClusterFixture.cs |
Messaging test cluster |
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingBehaviorTestBase.cs |
Shared behavior test base |
test/Orleans.DurableMessaging.Tests/Support/ControlledJournalStorageProvider.cs |
Journal failure injection |
test/Orleans.DurableMessaging.Tests/Support/ControlledDurableJobManager.cs |
Controlled job scheduling |
test/Orleans.DurableMessaging.Tests/Orleans.DurableMessaging.Tests.csproj |
Test project configuration |
test/Orleans.DurableMessaging.Tests/Functional/MessagingRoutingAndDeadLetterTests.cs |
Routing and dead-letter tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxHandlerTransactionTests.cs |
Handler transaction tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxCapacityBehaviorTests.cs |
Capacity behavior tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxCallbackOwnershipTests.cs |
Callback ownership tests |
test/Orleans.DurableMessaging.Tests/Functional/InboxAcceptanceBehaviorTests.cs |
Acceptance behavior tests |
test/Orleans.DurableMessaging.Tests/Functional/DurableMessagingClusterCollection.cs |
Test collection definition |
test/Orleans.DurableMessaging.Tests/Functional/DedupeExpiryBehaviorTests.cs |
Deduplication expiry tests |
test/Orleans.DurableMessaging.Tests/Contracts/HandlerRoutingContractTests.cs |
Handler routing contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DurableMessagingPumpResultsTests.cs |
Pump result contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DurableEnvelopeContractTests.cs |
Envelope contracts |
test/Orleans.DurableMessaging.Tests/Contracts/DeliveryAndOptionsContractTests.cs |
Delivery and options contracts |
test/Orleans.DurableMessaging.Tests/Contracts/ActivationValidationTests.cs |
Activation validation contracts |
src/Orleans.Journaling/IJournaledStateObserver.cs |
Journal lifecycle observer API |
src/Orleans.Journaling/IJournaledStateManager.cs |
Observer registration API |
src/Orleans.Journaling/IJournaledGrainParticipant.cs |
Participant contract |
src/Orleans.Journaling/DurableGrain.cs |
Participant initialization |
src/Orleans.DurableMessaging/RoutePrefixHandler.cs |
Prefix route handler |
src/Orleans.DurableMessaging/RouteKeyHandler.cs |
Exact route handler |
src/Orleans.DurableMessaging/README.md |
Runtime documentation |
src/Orleans.DurableMessaging/Orleans.DurableMessaging.csproj |
Messaging project configuration |
src/Orleans.DurableMessaging/InboxHandlerSelectionContext.cs |
Handler selection context |
src/Orleans.DurableMessaging/InboxHandlerContext.cs |
Handler context implementation |
src/Orleans.DurableMessaging/IInboxHandlerContext.cs |
Handler context API |
src/Orleans.DurableMessaging/IInboxHandler.cs |
Handler contract |
src/Orleans.DurableMessaging/IDurableOutbox.cs |
Outbox contract |
src/Orleans.DurableMessaging/IDurableMessagingDiagnostics.cs |
Diagnostics API |
src/Orleans.DurableMessaging/IDurableInboxExtension.cs |
Inbox extension API |
src/Orleans.DurableMessaging/IDurableInbox.cs |
Inbox API |
src/Orleans.DurableMessaging/DurableMessagingTime.cs |
Time arithmetic helpers |
src/Orleans.DurableMessaging/DurableMessagingStateNames.cs |
Durable state names |
src/Orleans.DurableMessaging/DurableMessagingStateManagerCapabilities.cs |
State manager capability checks |
src/Orleans.DurableMessaging/DurableMessagingPumpResults.cs |
Pump execution results |
src/Orleans.DurableMessaging/DurableMessagingPumpCoordinator.cs |
Pump coordination |
src/Orleans.DurableMessaging/DurableMessagingJobOwnership.cs |
Job ownership metadata |
src/Orleans.DurableMessaging/DurableMessagingInstruments.cs |
Messaging metrics |
src/Orleans.DurableMessaging/DurableMessagingGrainParticipant.cs |
Messaging participant |
src/Orleans.DurableMessaging/DurableMessagingActivationValidator.cs |
Activation validation |
src/Orleans.DurableMessaging/DurableMessageState.cs |
Retry and dead-letter state |
src/Orleans.DurableMessaging/DurableInbox.cs |
Inbox storage and routing |
src/Orleans.DurableMessaging/DurableEnvelopeData.cs |
Deferred payload access |
src/Orleans.DurableMessaging/DurableEnvelope.cs |
Durable envelope model |
src/Orleans.DurableMessaging/DurableDeadLetterRetention.cs |
Dead-letter retention |
src/Orleans.DurableMessaging/DeliveryStatus.cs |
Delivery status values |
src/Orleans.DurableMessaging/DeliveryResult.cs |
Delivery result contract |
src/Orleans.DurableMessaging/CorrelationHandler.cs |
Correlation routing |
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs |
Inbox configuration |
src/api/Orleans.Journaling/Orleans.Journaling.cs |
Generated Journaling API |
Orleans.slnx |
Solution project registration |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Code coverage
Report-only conclusion: improved. The current-main baseline is commit Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities. The comparison remains report-only while normal line and branch variance is calibrated. Coverage details |
There was a problem hiding this comment.
Copilot review overview
🔵 Needs a closer look
Six unresolved moderate findings and one nit remain, covering state rollback, option validation, shutdown and recovery handling, test-outbox idempotency, and an unused injected field.
Review effort: Lite
Findings: 2
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
One critical build issue and five moderate correctness, lifecycle, and test-isolation issues remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 3
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Six unresolved findings remain, including build-blocking and critical correctness issues.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 5
Open (6)
Composed keys with multi-segment child values lose segments · New Import CancellationToken in the handler context contract · New Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary Honor idempotent duplicate sends in the test outbox
4d18916 to
b05c432
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved ownership validation, compilation, and routing/performance issues require changes before approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 6
Open (6)
Reject delivery when recovered ownership is invalid · New Import CancellationToken in the handler context contract Composed keys with multi-segment child values lose segments Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary
Resolved since last review (1)
b05c432 to
c7b6bd9
Compare
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Unresolved ownership-race, retry/dead-letter accounting, deduplication, and shutdown-fault handling issues block approval.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 1
Open (2)
Resolved since last review (7)
Reject delivery when recovered ownership is invalid Import CancellationToken in the handler context contract Composed keys with multi-segment child values lose segments Remove the unused grain factory dependency Do not dispose the shutdown token while timers can still enter Handler selection does not enforce its read-only boundary Honor idempotent duplicate sends in the test outbox
c7b6bd9 to
67a4a2e
Compare
Typed handlers receive every decoded body, so the delegate owns resources decoded while cancellation arrives.
Adds an optional delivery time to the envelope contract. The receiving inbox processes the message once the Durable Jobs clock reaches DeliverAt. DeliverAt is part of command equivalence, is omitted from JSON when unset, and MessageType<T>.Create and the typed OutboxExtensions.Send overloads accept it as an optional argument.
…iverAt MaxCapacity bounds pending inbox messages without DeliverAt, and the inbox admits scheduled messages regardless of it. Pending outbox intent equality includes DeliverAt.
…tracts IOutboxMessageSource provides the envelopes of messages staged with IOutbox.SendFromSource, so the payload lives only in application-owned journaled state. IInboxMessageSink accepts delivered messages directly into application-owned journaled state. IOutbox.Register and IInbox.Register add sources and sinks.
| if (_handlers.IsEmpty || _messages.Count == 0) | ||
| { | ||
| return false; | ||
| } |
|
|
||
| private static int CountOccurrences(ReadOnlySpan<byte> data, ReadOnlySpan<byte> value) | ||
| { | ||
| var count = 0; |
| <PackageReference Include="Microsoft.Extensions.ObjectPool" /> | ||
| <ProjectReference Include="$(SourceRoot)src\Orleans.Core.Abstractions\Orleans.Core.Abstractions.csproj" /> | ||
| <ProjectReference Include="$(SourceRoot)src\Orleans.Serialization\Orleans.Serialization.csproj" /> | ||
| <ProjectReference Include="$(SourceRoot)src\Orleans.Journaling\Orleans.Journaling.csproj" /> |
…withdraw them IOutbox.Send(HierarchicalKey) replaces SendFromSource, and IOutboxMessageSource.TryGetMessage replaces CanProvide and GetEnvelope. The source supplies the whole OutboxMessage; returning false withdraws a pending message.
| if (entry.Message.Envelope.DeliverAt is { } deliverAt && deliverAt > jobNow) delay = deliverAt - jobNow; | ||
| if (entry.State.NextAttemptAt is { } retryAt && retryAt > now && retryAt - now > delay) delay = retryAt - now; | ||
| return delay; |
…alKey MessageId is a thin readonly struct around HierarchicalKey with its own codec, copier, and JSON converter. Envelope.MessageId, IOutbox.Send, IOutboxMessageSource, MessageType, and OutboxExtensions use MessageId for message identity; HierarchicalKey is unchanged.
After a send, an outbox source that throws or returns an invalid message dead-letters that entry alone and receives OnRetired(DeadLettered) in the same journal write. An exception during Send propagates without staging anything.
Adds IJournaledStateHook registration on IJournaledStateManager, IStateMachine.PrepareWriteAsync, and pre/post-commit exceptions so features such as the durable inbox can establish persistence prerequisites before a write and observe its outcome.
Adds the durable inbox, its journaled state, and the per-activation messaging pump which processes accepted messages on durable-job wake-ups. AddMessaging registers the inbox, inbox dead-letter diagnostics, and the grain-type configurator which installs the inbox extension during activation.
Messaging relies only on IStateMachine.PrepareWriteAsync and JournaledStatePreCommitException. Removes IJournaledStateHook, JournaledStateOperation, AddHookAsync/RemoveHookAsync, and JournaledStatePostCommitException, restoring their consumers to main.
A failed completion write deactivates the grain and disposes its activation services, so resolving the pump inside the exception filter threw and let the cancellation escape.
Deactivation_StopsPumpAndWaitsForActiveHandler armed a handler barrier that observed the shutdown token, so the handler exited on shutdown and deactivation could complete before the test asserted it was still pending. The barrier now ignores cancellation for this test, so deactivation waits until the test releases the handler.
Fail the activation when a pump pass or wake-up repair fails unexpectedly, let DeliverAsync callers stop waiting on cancellation while the owned write continues, validate recovered dead-letter keys, skip grain types without an implementation class, and make DeadLetter<T> serializable for diagnostics returned over RPC.
A pump job can run and complete before its scheduling call returns. Track a wake-up epoch which advances whenever a job run clears the scheduled wake-up, and record a schedule as a guaranteed wake-up only when no run completed while it was in flight.
The inbox accepts, deduplicates, and journals a message with Envelope.DeliverAt, then processes it once it is durable and the Durable Jobs clock reaches DeliverAt. The pump reschedules its durable wake-up for the earliest due message, and scheduled messages never count toward MaxCapacity.
…alence JournaledTestOutbox now coalesces a repeated send only when EnvelopeEquivalence.AreEquivalent holds, so DeliverAt participates in the comparison like the production outbox.
AddMessaging replaces any earlier IInbox registration, because inbox delivery targets the concrete Inbox and handlers must register on that instance.
… sinks After validation and deduplication, the inbox offers each message without DeliverAt to the registered IInboxMessageSink instances. A sink which accepts it stages the message in its own journaled state, and the inbox journals only the processed record in the same write, so the payload is journaled once. A sink which throws dead-letters the message. Declined and scheduled messages follow the handler path.
…t outbox JournaledTestOutbox implements IOutbox.Send(HierarchicalKey), and the sink test grain drops its unused StagePayload helper.
…alKey The inbox, its journaled state, inbox dead-letter diagnostics, and the inbox tests identify messages with MessageId.



Layer 2 of 4 in the durable messaging stack:
This PR is based on #11282. Review only the commits above it.
Problem
#11282 defines the messaging contracts, but nothing durably accepts or processes messages. To guarantee at-least-once processing with deduplication, a grain's inbox must:
Orleans Journaling had no way for a state to set up persistence prerequisites before a write. The inbox needs one to make sure a durable wake-up exists before pending work becomes durable.
Solution
Journaling. This PR adds only two Journaling APIs:
IStateMachine.PrepareWriteAsynclets a state establish its persistence prerequisites before its pending changes are captured. The manager prepares every registered state before each append or snapshot. After any asynchronous preparation completes, it repeats preparation until one full pass completes synchronously. That final pass covers changes staged on interleaved turns.JournaledStatePreCommitExceptionreports a preparation failure. Pending changes are kept so the caller can retry the write explicitly.The generated API surface is updated to match.
Messaging:
Inboxis an event-sourced journal state (orleans.messaging.inbox) that works with the binary and JSON journal formats.DeduplicationWindow.MaxCapacitybackpressure and returnsHandlerNotFoundwhen the grain has no handlers.DeliverAsyncreturnsAcceptedonly after the acceptance is durable.CompleteorFail.MaxProcessingAttempts, the message moves to dead letters.MessagingPumpruns one loop per activation on the grain scheduler. One durable job per grain wakes it while work remains. Wake-ups are scheduled on the durable jobsTimeProvider. The inbox'sPrepareWriteAsyncmakes sure the wake-up exists before newly staged work is captured.DeadLetterRetentionPeriodandMaxRetainedDeadLetters.IMessagingDiagnosticsexposes inbox dead letters and stages their removal.AddMessagingregisters the inbox, the pump, diagnostics, metrics, and a grain-type configurator. For grains that derive fromDurableGrainor implementIMessagingGrain, the configurator rejects stateless workers and installs the inbox extension during activation.Rationale
IStateMachine.OnWriteCompleted.IOutboxthat captures handler output, so this layer can be checked on its own.