Skip to content

Fix blocking S3 calls, unawaited cache notifications and startup provider - #758

Merged
KrzysztofPajak merged 4 commits into
developfrom
fix/blocking-calls
Aug 8, 2026
Merged

Fix blocking S3 calls, unawaited cache notifications and startup provider#758
KrzysztofPajak merged 4 commits into
developfrom
fix/blocking-calls

Conversation

@KrzysztofPajak

@KrzysztofPajak KrzysztofPajak commented Aug 8, 2026

Copy link
Copy Markdown
Member

Type: bugfix

Issue

No issues were open for these; they came out of an audit of async and startup code. Three
independent defects, one commit each (plus a follow-up correcting one of them — see below).

1. AmazonPictureService.SaveThumb blocked three times on S3 I/O.
CheckBucketExists().Wait(), PutObjectAsync(...).Wait() and MakeObjectPublicAsync(...).Wait(),
all from the thread serving the request. Thumbnail generation runs on catalogue and product pages,
so on an S3-backed installation these tie up thread pool threads for the length of a round trip.
This is the case .ai/constraints.md prohibits — blocking in service code, not in a startup path.

2. MemoryCacheBase mishandled cache invalidation in two ways.

RemoveAsync and RemoveByPrefix called _mediator.Publish without awaiting it and returned
Task.CompletedTask. A failing handler surfaced as an unobserved task exception on the finalizer
thread; a slow one raced the caller, so an invalidation could still be in flight when the next read
hit the cache. These were the only two unawaited Publish calls in the solution.

Clear() disposed _resetCacheToken. A writer reads the token source, and MemoryCache then
registers an eviction callback on it while storing the entry — that registration throws
ObjectDisposedException on a disposed source.

Reproduced with a stress test: Clear running against concurrent SetAsync throws
ObjectDisposedException within roughly 120ms.

3. StartupBase.ConfigureServices built a throwaway service provider.
services.BuildServiceProvider() (ASP0000) purely to resolve IWebHostEnvironment — a second,
never-disposed container with its own singleton graph. The same method also instantiated every
IStartupApplication twice, once per configuration pass. Separately,
Grand.Module.Migration.Startup.StartupApplication resolved the scoped IMigrationProcess from the
root provider.

Solution

  • SaveThumb is now async Task and awaits the three S3 calls. The base
    PictureService.SaveThumb already returns Task and every caller awaits it, so nothing else
    changes.
  • RemoveAsync and RemoveByPrefix await the notification.
  • Clear() cancels the token but no longer disposes it. The source is left to the garbage
    collector: Cancel releases the registrations, and it has no timer and no WaitHandle, so
    nothing needs releasing eagerly. Against develop this is a single removed line.
  • ConfigureServices takes IWebHostEnvironment; all four hosts already have builder.Environment.
    Startup types are instantiated once and the list filtered per pass.
  • The migration startup runs in its own scope. Its feature flag check still blocks, because
    Configure is synchronous, but GetAwaiter().GetResult() rethrows the original exception instead
    of wrapping it in an AggregateException.
  • Three regression tests in Grand.Infrastructure.Tests.

A correction worth reading before reviewing commit by commit. The first attempt at the Clear()
race swapped the token with Interlocked.Exchange before cancelling and disposing, and the commit
message claimed that closed the window. It does not: a thread that read the field before the swap
still holds the source being disposed, so the stress test throws on that version exactly as it does
on develop. The follow-up commit removes the Dispose call, which is the actual fix, and drops
the Exchange as well — with Dispose gone it only guarded against two concurrent Clear calls
orphaning a token, which is a different and much milder problem, out of scope here.

Deliberately left alone, with reasons:

  • MemoryCacheBase.Get<T> uses semaphore.Wait(). That blocks on a semaphore, not on a Task, and
    the method is a synchronous ICacheBase member — removing it means changing the interface.
  • MigrationProductRating uses GetAwaiter().GetResult() and IStartupApplication.Configure
    blocks. Both are startup paths with synchronous contracts, which .ai/constraints.md explicitly
    does not treat as precedent.
  • AzurePictureService.SaveThumb calls synchronous Azure SDK methods. That is a different problem
    from blocking on a Task and would widen this change.

Breaking changes

One. StartupBase.ConfigureServices gains a third parameter:

ConfigureServices(IServiceCollection services, IConfiguration configuration)
ConfigureServices(IServiceCollection services, IConfiguration configuration, IWebHostEnvironment hostingEnvironment)

All four in-repo hosts are updated. A custom host with its own Program.cs must pass
builder.Environment. It is a compile error, not a silent change. An overload was considered, but
keeping the old one means keeping the BuildServiceProvider call it exists to remove.

ICacheBase is unchanged — RemoveAsync and RemoveByPrefix already returned Task, and
RedisMessageCacheManager overrides both with matching signatures.

Testing

  1. dotnet build ./GrandNode.sln — succeeds.
  2. dotnet test ./src/Tests/Grand.Infrastructure.Tests/Grand.Infrastructure.Tests.csproj — 87/87 pass.
  3. dotnet test ./src/Tests/Grand.Modules.Tests/Grand.Modules.Tests.csproj — 18/18 pass.
  4. dotnet test ./src/Tests/Grand.Business.Storage.Tests/Grand.Business.Storage.Tests.csproj — 51 pass, 2 pre-existing skips.
  5. To confirm the notification tests guard the defect, temporarily restore the fire-and-forget body
    of MemoryCacheBase.RemoveAsync and re-run step 2. RemoveAsync_AwaitsTheNotification fails
    while RemoveByPrefix_AwaitsTheNotification still passes. Revert.
  6. To confirm the race test guards the defect, temporarily add _resetCacheToken.Dispose(); back
    into Clear() and re-run step 2. Clear_WhileEntriesAreBeingWritten_DoesNotThrow fails with
    ObjectDisposedException: The CancellationTokenSource has been disposed. Revert.
    The test only asserts on an exception the race actually raised, so it cannot fail spuriously — it
    can only miss.
  7. Start each of the four hosts and confirm they boot and serve a page — this change touches every
    Program.cs and the shared composition root.
  8. On an installation upgrading from an older database version, confirm migrations still run at
    startup; the migration process now executes inside its own DI scope.
  9. S3 path, if you have a bucket configured: set the Amazon storage provider, upload a product
    picture and load the product page so a thumbnail is generated. It should be created and served
    as before.

Steps 7 to 9 have not been run here — they need running hosts, an upgradable database and an S3
bucket.

🤖 Generated with Claude Code

KrzysztofPajak and others added 3 commits August 8, 2026 17:49
SaveThumb blocked three times on S3 I/O - CheckBucketExists().Wait(),
PutObjectAsync().Wait() and MakeObjectPublicAsync().Wait() - from whatever
thread was serving the request. Thumbnail generation happens on catalogue and
product pages, so under load this ties up thread pool threads for the duration
of a round trip to S3.

The base PictureService.SaveThumb already returns Task and every caller awaits
it, so the override becomes async without touching the contract.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
RemoveAsync and RemoveByPrefix called _mediator.Publish without awaiting it and
returned Task.CompletedTask. A handler that failed did so on the finalizer
thread as an unobserved task exception, and a handler that was slow raced the
caller, so an invalidation could still be in flight when the next read hit the
cache.

These were the only two unawaited Publish calls in the solution. Note that the
Redis path calls the base with publisher: false, so this only affected the
in-memory cache.

Clear() cancelled and disposed the reset token before swapping it, so two
concurrent calls could dispose a token another thread had already handed to
GetMemoryCacheEntryOptions. Swapping first with Interlocked.Exchange removes
the window.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
StartupBase.ConfigureServices called services.BuildServiceProvider() purely to
resolve IWebHostEnvironment (ASP0000). That builds a second, undisposed
container with its own singleton graph. Every caller is a Program.cs that
already has builder.Environment, so the environment is now a parameter.

The startup types were also instantiated twice - once to filter BeforeConfigure
and once for the rest. They are now created once and the list is filtered.
ConfigureRequestPipeline still creates its own instances; sharing them would
need static state, and IStartupApplication implementations are stateless.

Grand.Module.Migration resolved the scoped IMigrationProcess straight from the
root provider. The hosts set ValidateScopes = false so it did not throw, but it
rooted the repository graph for the process lifetime; it now runs in its own
scope. The feature flag check keeps blocking because Configure is synchronous,
but GetAwaiter().GetResult() rethrows the original exception rather than
wrapping it in an AggregateException.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings August 8, 2026 15:50

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Comment thread src/Core/Grand.Infrastructure/Caching/MemoryCacheBase.cs Fixed
Comment thread src/Core/Grand.Infrastructure/Caching/MemoryCacheBase.cs Fixed
The previous commit claimed that swapping the token with Interlocked.Exchange
before cancelling removed the race. It does not, and the claim was wrong.

Disposing is the actual problem. A writer reads the token source, and
MemoryCache then registers an eviction callback on it while storing the entry.
Disposing the source makes that registration throw ObjectDisposedException, and
swapping the field first only shortens the window - a thread that read the field
before the swap still holds the source being disposed.

Reproduced with a stress test: Clear running against concurrent SetAsync throws
ObjectDisposedException within about 120ms, both on the original code and on the
Exchange version. Removing the Dispose call makes it pass.

The source is left to the garbage collector. Cancel releases the registrations,
and it has no timer and no WaitHandle, so nothing needs releasing eagerly.
Interlocked.Exchange is gone too - with Dispose removed it only guarded against
two concurrent Clear calls orphaning a token, which is not what this change is
about, and this keeps the diff against develop to the single removed line.

The stress test only asserts on an exception the race actually raised, so it
cannot fail spuriously - it can only miss.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment thread src/Tests/Grand.Infrastructure.Tests/Caching/MemoryCacheBaseTests.cs Dismissed
@KrzysztofPajak
KrzysztofPajak merged commit 5076a0e into develop Aug 8, 2026
4 of 5 checks passed
@KrzysztofPajak
KrzysztofPajak deleted the fix/blocking-calls branch August 8, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants