Skip to content

fix: wasm remote code fetch race writing negative cache entry - #46674

Open
eformat wants to merge 1 commit into
envoyproxy:mainfrom
eformat:wasm-thread-load-fix
Open

fix: wasm remote code fetch race writing negative cache entry#46674
eformat wants to merge 1 commit into
envoyproxy:mainfrom
eformat:wasm-thread-load-fix

Conversation

@eformat

@eformat eformat commented Aug 12, 2026

Copy link
Copy Markdown

● Problem

When multiple Envoy worker threads simultaneously call createWasm() for a remote wasm plugin, Thread A starts the
HTTP fetch and marks the cache entry in_progress = true. Threads B–N find the entry in_progress, call cb(nullptr)
to signal failure, then fall through (missing return) to the code.empty() block. There, they write a negative cache
entry (TTL=10s) and start competing fetch attempts. Thread A's successful download is now racing against a
poisoned negative cache. For the next 10 seconds, any new call to createWasm() hits RemoteLoadCacheNegativeHit and
immediately fails, holding all wasm-filtered requests for up to 10 seconds before fail_open releases them.

  if (it->second.in_progress) {
      cb(nullptr);        // ← signals FAILURE to the plugin
  }                       // ← NO return! falls through
  code = it->second.code; // empty (Thread 1 still downloading)
  if (code.empty()) {
      fetch = true;       // ← Thread 2 ALSO starts downloading now
      it->second.in_progress = true;
  }

● Fix

Add return false immediately after cb(nullptr) in the in_progress branch. Threads B–N still signal failure to the
plugin (allowing fail_open to pass the request through), but they no longer fall through to write the negative
cache entry or start duplicate fetches. Thread A's download completes uncontested, populates the cache as a
successful hit, and all subsequent requests resolve instantly.

● Impact

  • Eliminates the RemoteLoadCacheNegativeHit cascade that causes 10-second dead zones for wasm-filtered requests
  • Fixes the CPU spike from competing parallel fetches (all threads racing to download the same binary)
  • Related: wasm: uri fetch triggers a crash #25052

@eformat
eformat requested a review from kyessenov as a code owner August 12, 2026 22:28
@eformat
eformat had a problem deploying to external-contributors August 12, 2026 22:28 — with GitHub Actions Error
@repokitteh-read-only

Copy link
Copy Markdown

Hi @eformat, welcome and thank you for your contribution.

We will try to review your Pull Request as quickly as possible.

In the meantime, please take a look at the contribution guidelines if you have not done so already.

🐱

Caused by: #46674 was opened by eformat.

see: more, trace.

Signed-off-by: Mike Hepburn <eformat@gmail.com>
@eformat
eformat force-pushed the wasm-thread-load-fix branch from 61bf740 to 74dc928 Compare August 12, 2026 22:29
@eformat
eformat requested a deployment to external-contributors August 12, 2026 22:29 — with GitHub Actions Waiting
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant