feat(durable-messaging): compose application identity and typed subjects

[ReubenBond]

Adds the application composition layer for grain-scoped durable inboxes and outboxes: typed subjects and send/reply helpers, hosting registration, handler terminal outcomes, executable recipes, a stock-reservation sample, and an actual-journal-acknowledgement sequential benchmark.

DurableMessageType<T> binds an exact ordinal subject to ordinary Serializer<T>. Typed sends encode each body into a GC-owned byte[] before staging. Published command bytes remain immutable; direct APIs share their array references, while ordinary RPC copying and deserialization isolate arrays. Batch sends enumerate once and stage each message independently, so stable identities support retries after partial staging. Replies derive the deterministic result child and use the application's explicit destination.

Handlers register concise method groups in OnActivateAsync, after recovery and before requests or pump turns begin. Typed routing freezes its subject bindings and supports explicit state with static delegates. Handlers finish asynchronous preparation before one synchronous final block containing outgoing intents, business mutation, Complete() or Fail(reason), and actual return. The runtime owns persistence and acknowledgement. The stock sample models acceptance, insufficient stock and invalid quantity as typed business results; rejections complete with unchanged stock. Fail retains diagnosed permanent processing failures as dead letters.

The dead-letter completion test observes both acknowledged write retirement and metric publication before checking the processed count. Compiled payload snippets follow repository formatting. Fixes #11475 and #11462.

Layer 4 of 4, above #11285 at aebeff7047913b03f26651250f91bcea0247d9c1, which includes receiver #11284 at 8f14ee9ed6c64fa6499cedd078f8176aaf7765c3 and contracts #11282 at ef6a16b0ead84f52a53896cece8e90ff168fe6c5. Review the incremental composition layer. The chain incorporates main 42682376d80e1176a952ba13aa2de25a307b576e.

The included unmerged prerequisites are journal hooks #11465 at 095bdb4eeb47bc61c7214f68a9df99501b0a8d64 and timers #11464 at 79d7fba537f24fc6e8cce38c9328dd24170d9017. Every journal owner provides the hook list. Queued and running persistence preserves its registrations; the final capture prerequisite runs inline before capture, and distinct pre/post-commit failures preserve the pending-versus-acknowledged outcome. Hook non-reentrancy is an implementation precondition. Timers use provider-native timing, one Idle/Busy/Disposed phase, and context-free tick messages. The approved mandatory-hook interface break has exactly two package-scoped property suppressions, one per target framework.

Arc serialization #11463, dependency caching #11467 and bounded page pooling #11470 remain independent PRs. Snapshot policy #11471 and general Arc correctness #11474 are merged into main. The user-closed RPC ownership #11468, dictionary ownership #11469 and optional format-evidence #11466 work is excluded. Messaging preserves main's ReadOnlySequence journal storage contract, V1 framing, V0/V1 readers and configured-format recovery fallback. All open PRs target main.

[Copilot]

Pull request overview

Adds a new durable, grain-scoped inbox/outbox messaging primitive (Microsoft.Orleans.DurableMessaging) built on Orleans Journaling + Durable Jobs, alongside supporting journaling participant/observer capabilities and Durable Jobs execution semantics needed for reliability, routing, and recovery.

Changes:

• Introduces Microsoft.Orleans.DurableMessaging (envelopes, routing handlers, inbox/outbox contracts, DI/hosting integration, diagnostics, and instrumentation).
• Extends Journaling with composable grain participants and write/recovery observers, plus rollback fencing and codec fixes for snapshot reference-scope replay.
• Extends Durable Jobs with feature handler registry, explicit RetryAt disposition/reset-rescheduling support, and improved execution deduplication.

Show a summary per file

File	Description
test/Orleans.Journaling.Tests/OrleansBinaryCommandCodecTests.cs	Adds regression tests ensuring pre-change snapshot payloads replay with independent reference scopes.
test/Orleans.Journaling.Tests/JournaledGrainParticipantTests.cs	Adds integration coverage for composed journaling participants initialization and failure propagation.
test/Orleans.Journaling.Tests/DurableListDirectWriteTests.cs	Updates test double to implement new revert/rollback surface.
test/Orleans.Journaling.Tests/DurableCollectionDirectWriteTests.cs	Updates test double to implement new revert/rollback surface.
test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs	Adds snapshot-based test synchronization utility.
test/Orleans.DurableMessaging.Tests/Support/HandlerProbe.cs	Adds barrier utility to block/release handlers deterministically in tests.
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingTestGrains.cs	Adds durable-messaging test grain, message/effect models, and handler behavior for scenarios.
test/Orleans.DurableMessaging.Tests/Support/DurableMessagingClusterFixture.cs	Adds reusable in-process cluster fixture with Durable Jobs + Journaling + Durable Messaging wiring.
test/Orleans.DurableMessaging.Tests/Support/ControlledJournalStorageProvider.cs	Adds controllable journal storage provider for injecting write blocking/failures in tests.
test/Orleans.DurableMessaging.Tests/Orleans.DurableMessaging.Tests.csproj	Introduces new Durable Messaging test project.
test/Orleans.DurableMessaging.Tests/Hosting/PublicDurableMessagingRegistrationTests.cs	Validates DI registrations, options behavior, and absence of friend-access requirements.
test/Orleans.DurableMessaging.Tests/Functional/MultiSiloDurableMessagingFailoverTests.cs	Adds multi-silo failover recovery test for stable inbox job ownership and exactly-once effects.
test/Orleans.DurableMessaging.Tests/Functional/InboxCapacityBehaviorTests.cs	Adds backpressure + recovery behavior tests under capacity constraints.
test/Orleans.DurableMessaging.Tests/Functional/DedupeExpiryBehaviorTests.cs	Adds dedupe expiry/compaction behavior test allowing later reprocessing.
test/Orleans.DurableMessaging.Tests/Contracts/HandlerRoutingContractTests.cs	Adds routing contract tests for exact, prefix, correlation, and typed handler behaviors.
test/Orleans.DurableMessaging.Tests/Contracts/DurableEnvelopeContractTests.cs	Adds envelope/builder/serializer contract tests including reply-to and context.
test/Orleans.DurableMessaging.Tests/Contracts/DeliveryAndOptionsContractTests.cs	Adds contracts for delivery result/status and options validation defaults/boundaries.
test/Orleans.DurableJobs.Tests/DurableJobs/JobShardTests.cs	Adds shard-level test distinguishing reschedule reset vs failure retry attempt semantics.
test/Orleans.DurableJobs.Tests/DurableJobs/JobShardManagerTestsRunner.cs	Adds reassignment test ensuring reschedule reset semantics persist through shard reassignment.
test/Orleans.DurableJobs.Tests/DurableJobs/DurableJobFeatureHandlerTests.cs	Adds coverage for registry behavior and durable job run-result compatibility semantics.
test/Orleans.Core.Tests/Orleans.Core.Tests.csproj	Links HierarchicalKey tests into core test project.
test/Orleans.Core.Tests/DurableJobs/ShardExecutorTests.cs	Adds executor tests for RetryAt, unknown disposition handling, and legacy shard behavior.
test/Orleans.Core.Tests/DurableJobs/DurableJobsExtensionsTests.cs	Adds DI scope/registry behavior tests and explicit replacement/decoration rejection.
test/Orleans.Core.Tests/DurableJobs/DurableJobReceiverExtensionTests.cs	Updates execution identity to (JobId, RunId), adds retention/dedup + feature handler precedence tests.
test/Benchmarks/Journaling/DurableListJournalBenchmarks.cs	Updates benchmark test double to implement new revert/rollback surface.
src/Orleans.Journaling/JournaledStateManager.cs	Adds rollback support flag, observer notifications, recovery fencing, and revert-pending-changes support.
src/Orleans.Journaling/IJournaledStateObserver.cs	Introduces observer interface for write/recovery boundaries.
src/Orleans.Journaling/IJournaledStateManager.cs	Adds SupportsRollback, observer registration, and RevertPendingChangesAsync surface.
src/Orleans.Journaling/IJournaledGrainParticipant.cs	Introduces participant initialization hook for composed journaling features.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableSetCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableQueueCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableListCommandCodec.cs	Fixes snapshot replay to reset reference scope per element.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryDurableDictionaryCommandCodec.cs	Fixes snapshot replay to reset reference scope per key/value entry.
src/Orleans.Journaling/Formats/OrleansBinary/OrleansBinaryCommandCodecHelpers.cs	Adds helper to read values using independent serializer sessions (reference-scope reset).
src/Orleans.Journaling/DurableGrain.cs	Initializes composed participants before recovery and documents WriteStateAsync.
src/Orleans.DurableMessaging/RoutePrefixHandler.cs	Adds prefix-based routing handler base class and documentation.
src/Orleans.DurableMessaging/RouteKeyHandler.cs	Adds exact-route routing handler base class and documentation.
src/Orleans.DurableMessaging/README.md	Adds package readme with configuration and semantics overview.
src/Orleans.DurableMessaging/Orleans.DurableMessaging.csproj	Adds new packable Durable Messaging project.
src/Orleans.DurableMessaging/InboxHandlerContext.cs	Adds handler context implementation for sending outbox messages and building envelopes.
src/Orleans.DurableMessaging/IInboxHandlerContext.cs	Adds public handler context contract.
src/Orleans.DurableMessaging/IInboxHandler.cs	Adds handler interfaces including typed handler adapter and documentation.
src/Orleans.DurableMessaging/IDurableOutbox.cs	Adds outbox public contract.
src/Orleans.DurableMessaging/IDurableMessagingDiagnostics.cs	Adds diagnostics contract + internal implementation for dead letters.
src/Orleans.DurableMessaging/IDurableInboxExtension.cs	Adds grain extension contract for durable inbox delivery.
src/Orleans.DurableMessaging/IDurableInbox.cs	Adds inbox public contract including handler registration and lookup.
src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs	Adds ISiloBuilder/IServiceCollection registration, option validation, and rollback requirement checks.
src/Orleans.DurableMessaging/DurableMessagingPumpResults.cs	Adds pump execution result tracking and one-shot timer handle helper.
src/Orleans.DurableMessaging/DurableMessagingInstruments.cs	Adds meters/counters/histograms for inbox/outbox behavior and depth tracking.
src/Orleans.DurableMessaging/DurableMessagingGrainParticipant.cs	Ensures messaging services materialize via journaling participant initialization.
src/Orleans.DurableMessaging/DurableMessageState.cs	Adds durable state models for inbox/outbox attempts and dead letters.
src/Orleans.DurableMessaging/DurableInbox.cs	Adds inbox implementation with handler registration/selection and storage-backed message access.
src/Orleans.DurableMessaging/DurableEnvelopeData.cs	Adds deferred (slice-based) envelope body/context storage and accessors.
src/Orleans.DurableMessaging/DurableEnvelope.cs	Adds durable envelope type (routing, correlation, reply-to, metadata, and payload).
src/Orleans.DurableMessaging/DeliveryStatus.cs	Adds delivery status enum contract.
src/Orleans.DurableMessaging/DeliveryResult.cs	Adds delivery result struct contract and factories.
src/Orleans.DurableMessaging/CorrelationHandler.cs	Adds correlation-hierarchy-based routing handler base class.
src/Orleans.DurableMessaging/Configuration/DurableInboxOptions.cs	Adds durable messaging options and validation contract.
src/Orleans.DurableJobs/ShardExecutor.cs	Adds RetryAt handling with reset-rescheduling path and explicit legacy-shard failure.
src/Orleans.DurableJobs/JournaledJobShard.cs	Implements reset-rescheduling in journaled shard via IResettableJobShard.
src/Orleans.DurableJobs/JobShard.cs	Adds reset-rescheduling support to base shard and introduces IResettableJobShard.
src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs	Updates receiver extension for feature-handler lookup, new execution identity, and retention semantics.
src/Orleans.DurableJobs/IDurableJobHandlerRegistry.cs	Adds activation-scoped feature handler registry + lookup implementation.
src/Orleans.DurableJobs/Hosting/DurableJobsOptions.cs	Adds completed-attempt retention option and validation.
src/Orleans.DurableJobs/Hosting/DurableJobsExtensions.cs	Registers registry in DI and explicitly rejects replacement/decoration.
src/Orleans.DurableJobs/DurableJobRunResult.cs	Adds RetryAt disposition and related fields/compatibility semantics.
src/api/Orleans.Journaling/Orleans.Journaling.cs	Updates generated API surface for journaling observer/participant/rollback additions.
src/api/Orleans.DurableJobs/Orleans.DurableJobs.cs	Updates generated API surface for feature handlers/registry and RetryAt semantics.
src/api/Orleans.Core.Abstractions/Orleans.Core.Abstractions.cs	Updates generated API surface to include HierarchicalKey type and codec.
Orleans.slnx	Adds Durable Messaging projects (src + tests) to the solution.
docs/site/src/data/unpublished-api-packages.json	Marks Durable Messaging package as unpublished.
docs/site/src/data/external-link-allowlist.json	Allow-lists NuGet link for unpublished Durable Messaging package.
docs/site/src/content/docs/toc.yml	Adds Durable Messaging doc page to Journaling section.
docs/site/src/content/docs/resources/nuget-packages.md	Adds Durable Messaging to NuGet package catalog and updates guidance blurb.
docs/site/src/content/docs/grains/durable-messaging.md	Adds end-user documentation for durable messaging semantics and requirements.

Review details

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

• Files reviewed: 83/83 changed files
• Comments generated: 4
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (4)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.DurableMessaging/DurableInbox.cs:43

• Exact-route handler storage uses the default string comparer, which is culture-sensitive. Route keys are protocol identifiers and should use ordinal comparisons (matching the rest of this PR’s routing semantics). Use StringComparer.Ordinal for _exactRouteHandlers to avoid surprising behavior under non-invariant cultures (e.g., Turkish-I issues).

        _inbox = inbox;
        _processed = processed;
        _handlers = new List<IInboxHandler>();
        _exactRouteHandlers = new Dictionary<string, IInboxHandler>();
        _capacity = capacity;

src/Orleans.Journaling/IJournaledStateObserver.cs:16

• The remarks contradict the interface contract: they say the manager does not notify observers before a write, but this interface defines OnWriteStarted and JournaledStateManager invokes it before capturing state. This is confusing for implementers; update the remarks to reflect the actual callback sequence.
  src/Orleans.DurableMessaging/RouteKeyHandler.cs:21
• The remarks suggest implementing IInboxHandler directly for prefix-based routing, but this PR also introduces RoutePrefixHandler for that exact scenario. Updating the docs helps steer users to the supported helper type and keeps the guidance consistent.
  src/Orleans.DurableMessaging/InboxHandlerContext.cs:125
• The XML docs reference IStateMachineManager.WriteStateAsync(), which doesn't exist in this package and appears to be a leftover name. This should point to IJournaledStateManager.WriteStateAsync() to avoid misleading API consumers.

• Files reviewed: 83/83 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (2)

Previously missed (2) — in code that hasn't changed since the last review.

src/Orleans.DurableJobs/IDurableJobReceiverExtension.cs:185

• LongPollGetJobStatusAsync creates a CancellationTokenSource but never cancels it. That means the Task.Delay will always run to completion even when the job finishes quickly, causing avoidable timer allocations/work per call.

                using var cts = new CancellationTokenSource();
                var longPollDuration = TimeSpan.FromTicks(Math.Min(_shared.MessagingOptions.ResponseTimeout.Divide(2).Ticks, _shared.Options.JobStatusPollInterval.Ticks));
                await Task.WhenAny(Task.Delay(longPollDuration, cts.Token), state.Task);

                if (!state.Task.IsCompleted)
                {
                    return DurableJobRunResult.PollAfter(_shared.Options.JobStatusPollInterval);
                }

src/Orleans.DurableMessaging/Hosting/DurableMessagingExtensions.cs:58

• The DurableInboxOptions validation predicate swallows all exceptions and turns them into a generic validation failure. That can hide unexpected exceptions (eg, NullReferenceException) and makes misconfiguration harder to diagnose. Consider only converting known validation exceptions into a failed predicate and letting other exceptions bubble.

        optionsBuilder.Validate(
            options =>
            {
                try
                {
                    options.Validate();
                    return true;
                }
                catch
                {
                    return false;
                }
            },
            "DurableInboxOptions validation failed.");

• Files reviewed: 83/83 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33

• WaitAsync adds a waiter to the per-grain list and then returns waiter.Completion.Task.WaitAsync(...), but if that wait times out/throws, the waiter remains in _waiters indefinitely. This can leak memory and can also keep evaluating stale predicates on subsequent Publish calls. Consider awaiting with a cleanup finally that removes the waiter (and optionally removes the empty list from _waiters).

• Files reviewed: 85/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (4)

Previously missed (3) — in code that hasn't changed since the last review.

src/Orleans.Journaling/JournaledStateManager.cs:309

• WorkLoop captures a stable observer snapshot into the local observers array, but OnWriteStarted() is invoked by iterating _observers directly. If an observer is registered during OnWritePreparingAsync (or from within another observer callback), it can change _observers mid-iteration, causing inconsistent callback sequencing and potential InvalidOperationException due to collection mutation during enumeration. Use the captured observers snapshot for OnWriteStarted to keep the write boundary consistent and mutation-safe.
  src/Orleans.Journaling/JournaledStateManager.cs:493
• OnWriteCompleted() is invoked by iterating _observers directly, even though a stable observers snapshot was captured earlier for this write. If observers are registered during the write pipeline, enumerating the live HashSet can throw (collection modified) or cause some observers to see OnWriteCompleted without the corresponding OnWritePreparingAsync/OnWriteStarted. Use the per-write snapshot for completion callbacks.

This issue also appears on line 516 of the same file.
src/Orleans.Journaling/JournaledStateManager.cs:826

• Recovery completion notifies observers by enumerating _observers directly inside a lock. If any observer callback registers another observer (re-entrantly) this can mutate the HashSet during enumeration and throw, potentially breaking recovery. Take a snapshot of _observers before iterating so callbacks are mutation-safe and sequencing is stable.

src/Orleans.Journaling/JournaledStateManager.cs:516

• In the no-op write path (!hasCommittedBuffer), OnWriteCompleted() is called by iterating _observers directly. This has the same collection-mutation risk as the committed write path and can also notify observers which were registered after OnWritePreparingAsync ran for this write. Prefer using the stable observers snapshot captured at the start of the work item.

• Files reviewed: 85/85 changed files
• Comments generated: 0 new
• Review effort level: Lite

[ReubenBond]

Addressed the latest observer-snapshot review findings in \3d4829b8d. Each write now uses one stable observer snapshot for preparation, start, and completion (including no-op writes), and recovery snapshots observers before callbacks so re-entrant registration cannot mutate an active enumeration. Focused regression coverage passes on net8.0 and net10.0.

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

test/Orleans.DurableMessaging.Tests/Support/SnapshotProbe.cs:33

• WaitAsync adds a waiter to the per-grain list but never removes it if the 30s timeout elapses (WaitAsync throws). That can leak waiters across a test run and slow down Publish() due to ever-growing lists.

• Files reviewed: 85/85 changed files
• Comments generated: 1
• Review effort level: Lite

[ReubenBond]

Addressed the remaining suppressed review findings in two focused commits:\n\n- \�222b4b77\ removes timed-out SnapshotProbe waiters so stale predicates are not retained or evaluated; regression coverage passes on net8.0 and net10.0.\n- \�d214b07b\ narrows durable inbox options validation to the documented \ArgumentOutOfRangeException, allowing unexpected faults to surface; the options-contract test passes on net8.0 and net10.0.

[Copilot]

Review details

Suppressed comments (1)

src/Orleans.Journaling/IJournaledStateManager.cs:58

• IJournaledStateManager is a public interface and RevertPendingChangesAsync is added as a required member (no default implementation). This is a breaking change for any external implementations of IJournaledStateManager. Consider providing a default interface implementation (similar to RegisterObserver) which throws NotSupportedException, and use SupportsRollback to indicate capability.

• Files reviewed: 86/86 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

Review details

• Files reviewed: 86/86 changed files
• Comments generated: 2
• Review effort level: Lite

[Copilot]

Review details

Suppressed comments (1)

Previously missed (1) — in code that hasn't changed since the last review.

src/Orleans.DurableMessaging/DurableOutbox.cs:168

• The XML docs reference IStateMachineManager.WriteStateAsync(), but durable messaging commits are driven by journaling. This should reference IJournaledStateManager.WriteStateAsync() to match the rest of the package docs and avoid misleading API consumers.

    /// The message is persisted atomically with grain state when IStateMachineManager.WriteStateAsync()
    /// is called. The background pump will deliver the message to the target grain ONLY AFTER
    /// the message has been durably persisted.

• Files reviewed: 86/86 changed files
• Comments generated: 0 new
• Review effort level: Lite

[Copilot]

🟡 Changes recommended

Three critical findings remain unresolved.

3 open findings

• Preserve terminal dead-letter status for missing handlers · New
• Add serialization support for DurableDeadLetter diagnostics · New
• Remove unnecessary async from handler to avoid CS1998 · New

2 resolved since last review

• Preserve configured journal format fallback in recovery
• Reset pooled writer after ConsumeSlice to release page pins

🧠 Review effort: Lite

[Copilot]

🟡 Changes recommended

One or more issues must be addressed before approval.

4 open findings

• Prevent stale handler context from rejecting unrelated attempts · New
• Remove unnecessary async from handler to avoid CS1998
• Add serialization support for DurableDeadLetter diagnostics
• Preserve terminal dead-letter status for missing handlers

🧠 Review effort: Lite