Skip to content

feat(messaging): process inbox messages - #11284

Open
ReubenBond wants to merge 31 commits into
dotnet:mainfrom
ReubenBond:rb-potential-happiness
Open

ReubenBond wants to merge 31 commits into
dotnet:mainfrom
ReubenBond:rb-potential-happiness

Conversation

@ReubenBond

@ReubenBond ReubenBond commented Sep 16, 2026 •

Copy link
Copy Markdown
Member

Layer 2 of 4 in the durable messaging stack:

  1. feat(messaging): add durable messaging contracts #11282: contracts
  2. feat(messaging): process inbox messages #11284: inbox processing (this PR)
  3. feat(messaging): deliver outbox messages #11285: outbox delivery
  4. feat(messaging): add durable grain inbox and outbox messaging #10693: integration, docs, samples, AOT smoke, benchmarks, playground

This PR is based on #11282. Review only the commits above it.

Problem

#11282 defines the messaging contracts, but nothing durably accepts or processes messages. To guarantee at-least-once processing with deduplication, a grain's inbox must:

  • persist accepted messages with the grain's other journaled state,
  • process them only after that write completes, and
  • commit handler outcomes atomically with the handler's business changes.

Orleans Journaling had no way for a state to set up persistence prerequisites before a write. The inbox needs one to make sure a durable wake-up exists before pending work becomes durable.

Solution

Journaling. This PR adds only two Journaling APIs:

  • IStateMachine.PrepareWriteAsync lets a state establish its persistence prerequisites before its pending changes are captured. The manager prepares every registered state before each append or snapshot. After any asynchronous preparation completes, it repeats preparation until one full pass completes synchronously. That final pass covers changes staged on interleaved turns.
  • JournaledStatePreCommitException reports a preparation failure. Pending changes are kept so the caller can retry the write explicitly.

The generated API surface is updated to match.

Messaging:

  • Inbox is an event-sourced journal state (orleans.messaging.inbox) that works with the binary and JSON journal formats.
    • It deduplicates by message ID across pending messages and the DeduplicationWindow.
    • It applies MaxCapacity backpressure and returns HandlerNotFound when the grain has no handlers.
    • DeliverAsync returns Accepted only after the acceptance is durable.
  • Handlers finish each message with Complete or Fail.
    • A handler exception retries the message with exponential backoff. After MaxProcessingAttempts, the message moves to dead letters.
    • Returning without an outcome, or staging two different outcomes, deactivates the grain.
    • Each batch's outcomes and the handler's business changes are persisted in one journal write.
  • MessagingPump runs one loop per activation on the grain scheduler. One durable job per grain wakes it while work remains. Wake-ups are scheduled on the durable jobs TimeProvider. The inbox's PrepareWriteAsync makes sure the wake-up exists before newly staged work is captured.
  • Dead letters are retained according to DeadLetterRetentionPeriod and MaxRetainedDeadLetters. IMessagingDiagnostics exposes inbox dead letters and stages their removal.
  • AddMessaging registers the inbox, the pump, diagnostics, metrics, and a grain-type configurator. For grains that derive from DurableGrain or implement IMessagingGrain, the configurator rejects stateless workers and installs the inbox extension during activation.
  • Metrics: received and processed counters, a processing-duration histogram, and an inbox-depth up-down counter.

Rationale

  • Gating processing on journal durability, and committing outcomes together with business state, gives at-least-once processing with exactly-once effects on the grain's own state. No separate transaction mechanism is needed.
  • Every pump pass derives its work from current journaled state. Early, late, or duplicate wake-ups are therefore harmless, and one durable job per grain is enough.
  • Write preparation is the one Journaling extension messaging needs. Tests observe write acknowledgement through IStateMachine.OnWriteCompleted.
  • The outbox runtime is left to feat(messaging): deliver outbox messages #11285. Functional tests run against the production inbox with a journaled test IOutbox that captures handler output, so this layer can be checked on its own.

Copilot AI lite review requested due to automatic review settings September 16, 2026 22:04

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved lifecycle, selection-mutation, scheduling, recovery, retention, validation, and outbox-contract issues remain.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)
What changed in this PR

Adds a non-packable Durable Messaging inbox runtime integrated with Journaling and Durable Jobs, including recoverable ownership, transactional handler processing, retries, deduplication, and dead-lettering.

Changes:

  • Adds durable messaging contracts, routing, diagnostics, and configuration.
  • Implements journal-backed acceptance, recovery, pump coordination, and handler processing.
  • Adds Journaling integration and comprehensive contract and functional tests.
File Reviewed change
test/​Orleans.DurableMessaging.Tests/​Support/​SnapshotProbeTests.cs Snapshot probe tests
test/​Orleans.DurableMessaging.Tests/​Support/​SnapshotProbe.cs Snapshot waiting helper
test/​Orleans.DurableMessaging.Tests/​Support/​ReceiverTestServices.cs Test service composition
test/​Orleans.DurableMessaging.Tests/​Support/​JournaledTestOutbox.cs Journal-backed test outbox
test/​Orleans.DurableMessaging.Tests/​Support/​HandlerProbe.cs Handler synchronization helper
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingMetricProbe.cs Metrics test probe
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingClusterFixture.cs Messaging test cluster
test/​Orleans.DurableMessaging.Tests/​Support/​DurableMessagingBehaviorTestBase.cs Shared behavior test base
test/​Orleans.DurableMessaging.Tests/​Support/​ControlledJournalStorageProvider.cs Journal failure injection
test/​Orleans.DurableMessaging.Tests/​Support/​ControlledDurableJobManager.cs Controlled job scheduling
test/​Orleans.DurableMessaging.Tests/​Orleans.DurableMessaging.Tests.csproj Test project configuration
test/​Orleans.DurableMessaging.Tests/​Functional/​MessagingRoutingAndDeadLetterTests.cs Routing and dead-letter tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxHandlerTransactionTests.cs Handler transaction tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxCapacityBehaviorTests.cs Capacity behavior tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxCallbackOwnershipTests.cs Callback ownership tests
test/​Orleans.DurableMessaging.Tests/​Functional/​InboxAcceptanceBehaviorTests.cs Acceptance behavior tests
test/​Orleans.DurableMessaging.Tests/​Functional/​DurableMessagingClusterCollection.cs Test collection definition
test/​Orleans.DurableMessaging.Tests/​Functional/​DedupeExpiryBehaviorTests.cs Deduplication expiry tests
test/​Orleans.DurableMessaging.Tests/​Contracts/​HandlerRoutingContractTests.cs Handler routing contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DurableMessagingPumpResultsTests.cs Pump result contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DurableEnvelopeContractTests.cs Envelope contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​DeliveryAndOptionsContractTests.cs Delivery and options contracts
test/​Orleans.DurableMessaging.Tests/​Contracts/​ActivationValidationTests.cs Activation validation contracts
src/​Orleans.Journaling/​IJournaledStateObserver.cs Journal lifecycle observer API
src/​Orleans.Journaling/​IJournaledStateManager.cs Observer registration API
src/​Orleans.Journaling/​IJournaledGrainParticipant.cs Participant contract
src/​Orleans.Journaling/​DurableGrain.cs Participant initialization
src/​Orleans.DurableMessaging/​RoutePrefixHandler.cs Prefix route handler
src/​Orleans.DurableMessaging/​RouteKeyHandler.cs Exact route handler
src/​Orleans.DurableMessaging/​README.md Runtime documentation
src/​Orleans.DurableMessaging/​Orleans.DurableMessaging.csproj Messaging project configuration
src/​Orleans.DurableMessaging/​InboxHandlerSelectionContext.cs Handler selection context
src/​Orleans.DurableMessaging/​InboxHandlerContext.cs Handler context implementation
src/​Orleans.DurableMessaging/​IInboxHandlerContext.cs Handler context API
src/​Orleans.DurableMessaging/​IInboxHandler.cs Handler contract
src/​Orleans.DurableMessaging/​IDurableOutbox.cs Outbox contract
src/​Orleans.DurableMessaging/​IDurableMessagingDiagnostics.cs Diagnostics API
src/​Orleans.DurableMessaging/​IDurableInboxExtension.cs Inbox extension API
src/​Orleans.DurableMessaging/​IDurableInbox.cs Inbox API
src/​Orleans.DurableMessaging/​DurableMessagingTime.cs Time arithmetic helpers
src/​Orleans.DurableMessaging/​DurableMessagingStateNames.cs Durable state names
src/​Orleans.DurableMessaging/​DurableMessagingStateManagerCapabilities.cs State manager capability checks
src/​Orleans.DurableMessaging/​DurableMessagingPumpResults.cs Pump execution results
src/​Orleans.DurableMessaging/​DurableMessagingPumpCoordinator.cs Pump coordination
src/​Orleans.DurableMessaging/​DurableMessagingJobOwnership.cs Job ownership metadata
src/​Orleans.DurableMessaging/​DurableMessagingInstruments.cs Messaging metrics
src/​Orleans.DurableMessaging/​DurableMessagingGrainParticipant.cs Messaging participant
src/​Orleans.DurableMessaging/​DurableMessagingActivationValidator.cs Activation validation
src/​Orleans.DurableMessaging/​DurableMessageState.cs Retry and dead-letter state
src/​Orleans.DurableMessaging/​DurableInbox.cs Inbox storage and routing
src/​Orleans.DurableMessaging/​DurableEnvelopeData.cs Deferred payload access
src/​Orleans.DurableMessaging/​DurableEnvelope.cs Durable envelope model
src/​Orleans.DurableMessaging/​DurableDeadLetterRetention.cs Dead-letter retention
src/​Orleans.DurableMessaging/​DeliveryStatus.cs Delivery status values
src/​Orleans.DurableMessaging/​DeliveryResult.cs Delivery result contract
src/​Orleans.DurableMessaging/​CorrelationHandler.cs Correlation routing
src/​Orleans.DurableMessaging/​Configuration/​DurableInboxOptions.cs Inbox configuration
src/​api/​Orleans.Journaling/​Orleans.Journaling.cs Generated Journaling API
Orleans.slnx Solution project registration

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Comment thread test/Orleans.DurableMessaging.Tests/Support/JournaledTestOutbox.cs Outdated
Copilot AI review requested due to automatic review settings September 16, 2026 22:19
@github-actions

github-actions Bot commented Sep 16, 2026 •

Copy link
Copy Markdown
Contributor

Code coverage

Metric Pull request Current main Variance
Lines 83.61% (119,889 / 143,395) 83.44% (118,572 / 142,101) +0.1655 pp
Branches 73.31% (35,779 / 48,803) 73.01% (35,269 / 48,308) +0.3045 pp

Report-only conclusion: improved.

The current-main baseline is commit 8a48ca2157 and uses the same reviewed coverage matrix.

Coverage combines every CI test matrix job, including providers, CodeGen, .NET 8/10, Linux, Windows, and macOS, using canonical physical source and branch identities.

The comparison remains report-only while normal line and branch variance is calibrated.

Coverage details

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🔵 Needs a closer look

Six unresolved moderate findings and one nit remain, covering state rollback, option validation, shutdown and recovery handling, test-outbox idempotency, and an unused injected field.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)

Copilot AI review requested due to automatic review settings September 16, 2026 22:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One critical build issue and five moderate correctness, lifecycle, and test-isolation issues remain unresolved.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 3 High severity · 1 Medium severity

Open (4)

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Six unresolved findings remain, including build-blocking and critical correctness issues.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 5 High severity · 1 Medium severity

Open (6)

Comment thread src/Orleans.DurableMessaging/HierarchicalKey.cs Outdated
Comment thread src/Orleans.DurableMessaging/IInboxHandlerContext.cs Outdated
Copilot AI review requested due to automatic review settings September 16, 2026 23:05
@ReubenBond
ReubenBond force-pushed the rb-potential-happiness branch from 4d18916 to b05c432 Compare September 16, 2026 23:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved ownership validation, compilation, and routing/performance issues require changes before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 6 High severity

Open (6)
Resolved since last review (1)

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Unresolved ownership-race, retry/dead-letter accounting, deduplication, and shutdown-fault handling issues block approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 High severity · 1 Medium severity

Open (2)
Resolved since last review (7)

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Copilot AI review requested due to automatic review settings September 17, 2026 01:26
@ReubenBond
ReubenBond force-pushed the rb-potential-happiness branch from c7b6bd9 to 67a4a2e Compare September 17, 2026 01:27

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One critical and two moderate findings remain unresolved in durable inbox acceptance and scheduling.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 2 High severity · 1 Medium severity

Open (3)

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Copilot AI review requested due to automatic review settings September 17, 2026 01:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

One or more issues must be addressed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1 Medium severity

Open (1)
Resolved since last review (3)

Comment thread src/Orleans.DurableMessaging/DurableInboxExtension.cs Outdated
Typed handlers receive every decoded body, so the delegate owns resources decoded while cancellation arrives.
Adds an optional delivery time to the envelope contract. The receiving inbox processes the message once the Durable Jobs clock reaches DeliverAt. DeliverAt is part of command equivalence, is omitted from JSON when unset, and MessageType<T>.Create and the typed OutboxExtensions.Send overloads accept it as an optional argument.
…iverAt

MaxCapacity bounds pending inbox messages without DeliverAt, and the inbox admits scheduled messages regardless of it. Pending outbox intent equality includes DeliverAt.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The broad changes include unresolved contract/documentation and test-helper issues and warrant human review.

0 open findings

🧠 Review effort: Lite

…tracts

IOutboxMessageSource provides the envelopes of messages staged with IOutbox.SendFromSource, so the payload lives only in application-owned journaled state. IInboxMessageSink accepts delivered messages directly into application-owned journaled state. IOutbox.Register and IInbox.Register add sources and sinks.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The no-handler polling issue is critical, and two test-helper defects remain unresolved.

3 open findings

🧠 Review effort: Lite

Comment on lines +208 to +211
if (_handlers.IsEmpty || _messages.Count == 0)
{
return false;
}

private static int CountOccurrences(ReadOnlySpan<byte> data, ReadOnlySpan<byte> value)
{
var count = 0;
Comment on lines +21 to +24
<PackageReference Include="Microsoft.Extensions.ObjectPool" />
<ProjectReference Include="$(SourceRoot)src\Orleans.Core.Abstractions\Orleans.Core.Abstractions.csproj" />
<ProjectReference Include="$(SourceRoot)src\Orleans.Serialization\Orleans.Serialization.csproj" />
<ProjectReference Include="$(SourceRoot)src\Orleans.Journaling\Orleans.Journaling.csproj" />
…withdraw them

IOutbox.Send(HierarchicalKey) replaces SendFromSource, and IOutboxMessageSource.TryGetMessage replaces CanProvide and GetEnvelope. The source supplies the whole OutboxMessage; returning false withdraws a pending message.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved retry scheduling, startup validation, persistence-metric ordering, dependency, and test-helper issues remain.

4 open findings

🧠 Review effort: Lite

Comment on lines +398 to +400
if (entry.Message.Envelope.DeliverAt is { } deliverAt && deliverAt > jobNow) delay = deliverAt - jobNow;
if (entry.State.NextAttemptAt is { } retryAt && retryAt > now && retryAt - now > delay) delay = retryAt - now;
return delay;
…alKey

MessageId is a thin readonly struct around HierarchicalKey with its own codec, copier, and JSON converter. Envelope.MessageId, IOutbox.Send, IOutboxMessageSource, MessageType, and OutboxExtensions use MessageId for message identity; HierarchicalKey is unchanged.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After a send, an outbox source that throws or returns an invalid message dead-letters that entry alone and receives OnRetired(DeadLettered) in the same journal write. An exception during Send propagates without staging anything.
Adds IJournaledStateHook registration on IJournaledStateManager, IStateMachine.PrepareWriteAsync, and pre/post-commit exceptions so features such as the durable inbox can establish persistence prerequisites before a write and observe its outcome.
Adds the durable inbox, its journaled state, and the per-activation messaging pump which processes accepted messages on durable-job wake-ups. AddMessaging registers the inbox, inbox dead-letter diagnostics, and the grain-type configurator which installs the inbox extension during activation.
Messaging relies only on IStateMachine.PrepareWriteAsync and JournaledStatePreCommitException. Removes IJournaledStateHook, JournaledStateOperation, AddHookAsync/RemoveHookAsync, and JournaledStatePostCommitException, restoring their consumers to main.
A failed completion write deactivates the grain and disposes its activation services, so resolving the pump inside the exception filter threw and let the cancellation escape.
Deactivation_StopsPumpAndWaitsForActiveHandler armed a handler barrier that observed the shutdown token, so the handler exited on shutdown and deactivation could complete before the test asserted it was still pending. The barrier now ignores cancellation for this test, so deactivation waits until the test releases the handler.
Fail the activation when a pump pass or wake-up repair fails unexpectedly, let DeliverAsync callers stop waiting on cancellation while the owned write continues, validate recovered dead-letter keys, skip grain types without an implementation class, and make DeadLetter<T> serializable for diagnostics returned over RPC.
A pump job can run and complete before its scheduling call returns. Track a wake-up epoch which advances whenever a job run clears the scheduled wake-up, and record a schedule as a guaranteed wake-up only when no run completed while it was in flight.
The inbox accepts, deduplicates, and journals a message with Envelope.DeliverAt, then processes it once it is durable and the Durable Jobs clock reaches DeliverAt. The pump reschedules its durable wake-up for the earliest due message, and scheduled messages never count toward MaxCapacity.
…alence

JournaledTestOutbox now coalesces a repeated send only when EnvelopeEquivalence.AreEquivalent holds, so DeliverAt participates in the comparison like the production outbox.
AddMessaging replaces any earlier IInbox registration, because inbox delivery targets the concrete Inbox and handlers must register on that instance.
… sinks

After validation and deduplication, the inbox offers each message without DeliverAt to the registered IInboxMessageSink instances. A sink which accepts it stages the message in its own journaled state, and the inbox journals only the processed record in the same write, so the payload is journaled once. A sink which throws dead-letters the message. Declined and scheduled messages follow the handler path.
…t outbox

JournaledTestOutbox implements IOutbox.Send(HierarchicalKey), and the sink test grain drops its unused StagePayload helper.
…alKey

The inbox, its journaled state, inbox dead-letter diagnostics, and the inbox tests identify messages with MessageId.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Four moderate issues remain unresolved, including validation timing, metric isolation, empty-payload handling, and delivery-time preservation.

4 open findings

🧠 Review effort: Lite

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants