Fix blocking S3 calls, unawaited cache notifications and startup provider - #758
Merged
Conversation
SaveThumb blocked three times on S3 I/O - CheckBucketExists().Wait(), PutObjectAsync().Wait() and MakeObjectPublicAsync().Wait() - from whatever thread was serving the request. Thumbnail generation happens on catalogue and product pages, so under load this ties up thread pool threads for the duration of a round trip to S3. The base PictureService.SaveThumb already returns Task and every caller awaits it, so the override becomes async without touching the contract. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RemoveAsync and RemoveByPrefix called _mediator.Publish without awaiting it and returned Task.CompletedTask. A handler that failed did so on the finalizer thread as an unobserved task exception, and a handler that was slow raced the caller, so an invalidation could still be in flight when the next read hit the cache. These were the only two unawaited Publish calls in the solution. Note that the Redis path calls the base with publisher: false, so this only affected the in-memory cache. Clear() cancelled and disposed the reset token before swapping it, so two concurrent calls could dispose a token another thread had already handed to GetMemoryCacheEntryOptions. Swapping first with Interlocked.Exchange removes the window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StartupBase.ConfigureServices called services.BuildServiceProvider() purely to resolve IWebHostEnvironment (ASP0000). That builds a second, undisposed container with its own singleton graph. Every caller is a Program.cs that already has builder.Environment, so the environment is now a parameter. The startup types were also instantiated twice - once to filter BeforeConfigure and once for the rest. They are now created once and the list is filtered. ConfigureRequestPipeline still creates its own instances; sharing them would need static state, and IStartupApplication implementations are stateless. Grand.Module.Migration resolved the scoped IMigrationProcess straight from the root provider. The hosts set ValidateScopes = false so it did not throw, but it rooted the repository graph for the process lifetime; it now runs in its own scope. The feature flag check keeps blocking because Configure is synchronous, but GetAwaiter().GetResult() rethrows the original exception rather than wrapping it in an AggregateException. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The previous commit claimed that swapping the token with Interlocked.Exchange before cancelling removed the race. It does not, and the claim was wrong. Disposing is the actual problem. A writer reads the token source, and MemoryCache then registers an eviction callback on it while storing the entry. Disposing the source makes that registration throw ObjectDisposedException, and swapping the field first only shortens the window - a thread that read the field before the swap still holds the source being disposed. Reproduced with a stress test: Clear running against concurrent SetAsync throws ObjectDisposedException within about 120ms, both on the original code and on the Exchange version. Removing the Dispose call makes it pass. The source is left to the garbage collector. Cancel releases the registrations, and it has no timer and no WaitHandle, so nothing needs releasing eagerly. Interlocked.Exchange is gone too - with Dispose removed it only guarded against two concurrent Clear calls orphaning a token, which is not what this change is about, and this keeps the diff against develop to the single removed line. The stress test only asserts on an exception the race actually raised, so it cannot fail spuriously - it can only miss. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Type: bugfix
Issue
No issues were open for these; they came out of an audit of async and startup code. Three
independent defects, one commit each (plus a follow-up correcting one of them — see below).
1.
AmazonPictureService.SaveThumbblocked three times on S3 I/O.CheckBucketExists().Wait(),PutObjectAsync(...).Wait()andMakeObjectPublicAsync(...).Wait(),all from the thread serving the request. Thumbnail generation runs on catalogue and product pages,
so on an S3-backed installation these tie up thread pool threads for the length of a round trip.
This is the case
.ai/constraints.mdprohibits — blocking in service code, not in a startup path.2.
MemoryCacheBasemishandled cache invalidation in two ways.RemoveAsyncandRemoveByPrefixcalled_mediator.Publishwithout awaiting it and returnedTask.CompletedTask. A failing handler surfaced as an unobserved task exception on the finalizerthread; a slow one raced the caller, so an invalidation could still be in flight when the next read
hit the cache. These were the only two unawaited
Publishcalls in the solution.Clear()disposed_resetCacheToken. A writer reads the token source, andMemoryCachethenregisters an eviction callback on it while storing the entry — that registration throws
ObjectDisposedExceptionon a disposed source.Reproduced with a stress test:
Clearrunning against concurrentSetAsyncthrowsObjectDisposedExceptionwithin roughly 120ms.3.
StartupBase.ConfigureServicesbuilt a throwaway service provider.services.BuildServiceProvider()(ASP0000) purely to resolveIWebHostEnvironment— a second,never-disposed container with its own singleton graph. The same method also instantiated every
IStartupApplicationtwice, once per configuration pass. Separately,Grand.Module.Migration.Startup.StartupApplicationresolved the scopedIMigrationProcessfrom theroot provider.
Solution
SaveThumbis nowasync Taskand awaits the three S3 calls. The basePictureService.SaveThumbalready returnsTaskand every caller awaits it, so nothing elsechanges.
RemoveAsyncandRemoveByPrefixawait the notification.Clear()cancels the token but no longer disposes it. The source is left to the garbagecollector:
Cancelreleases the registrations, and it has no timer and noWaitHandle, sonothing needs releasing eagerly. Against
developthis is a single removed line.ConfigureServicestakesIWebHostEnvironment; all four hosts already havebuilder.Environment.Startup types are instantiated once and the list filtered per pass.
Configureis synchronous, butGetAwaiter().GetResult()rethrows the original exception insteadof wrapping it in an
AggregateException.Grand.Infrastructure.Tests.A correction worth reading before reviewing commit by commit. The first attempt at the
Clear()race swapped the token with
Interlocked.Exchangebefore cancelling and disposing, and the commitmessage claimed that closed the window. It does not: a thread that read the field before the swap
still holds the source being disposed, so the stress test throws on that version exactly as it does
on
develop. The follow-up commit removes theDisposecall, which is the actual fix, and dropsthe
Exchangeas well — withDisposegone it only guarded against two concurrentClearcallsorphaning a token, which is a different and much milder problem, out of scope here.
Deliberately left alone, with reasons:
MemoryCacheBase.Get<T>usessemaphore.Wait(). That blocks on a semaphore, not on aTask, andthe method is a synchronous
ICacheBasemember — removing it means changing the interface.MigrationProductRatingusesGetAwaiter().GetResult()andIStartupApplication.Configureblocks. Both are startup paths with synchronous contracts, which
.ai/constraints.mdexplicitlydoes not treat as precedent.
AzurePictureService.SaveThumbcalls synchronous Azure SDK methods. That is a different problemfrom blocking on a
Taskand would widen this change.Breaking changes
One.
StartupBase.ConfigureServicesgains a third parameter:All four in-repo hosts are updated. A custom host with its own
Program.csmust passbuilder.Environment. It is a compile error, not a silent change. An overload was considered, butkeeping the old one means keeping the
BuildServiceProvidercall it exists to remove.ICacheBaseis unchanged —RemoveAsyncandRemoveByPrefixalready returnedTask, andRedisMessageCacheManageroverrides both with matching signatures.Testing
dotnet build ./GrandNode.sln— succeeds.dotnet test ./src/Tests/Grand.Infrastructure.Tests/Grand.Infrastructure.Tests.csproj— 87/87 pass.dotnet test ./src/Tests/Grand.Modules.Tests/Grand.Modules.Tests.csproj— 18/18 pass.dotnet test ./src/Tests/Grand.Business.Storage.Tests/Grand.Business.Storage.Tests.csproj— 51 pass, 2 pre-existing skips.of
MemoryCacheBase.RemoveAsyncand re-run step 2.RemoveAsync_AwaitsTheNotificationfailswhile
RemoveByPrefix_AwaitsTheNotificationstill passes. Revert._resetCacheToken.Dispose();backinto
Clear()and re-run step 2.Clear_WhileEntriesAreBeingWritten_DoesNotThrowfails withObjectDisposedException: The CancellationTokenSource has been disposed.Revert.The test only asserts on an exception the race actually raised, so it cannot fail spuriously — it
can only miss.
Program.csand the shared composition root.startup; the migration process now executes inside its own DI scope.
picture and load the product page so a thumbnail is generated. It should be created and served
as before.
Steps 7 to 9 have not been run here — they need running hosts, an upgradable database and an S3
bucket.
🤖 Generated with Claude Code