[ai-gateway-provider] Concurrent requests can receive each other's responses

[MaratFM]

Summary

When an AiGatewayChatLanguageModel is created once and reused for concurrent requests, a request can receive a response generated for another request. This can expose one user's model output to another user in applications that share a model instance.

Internally, processModelRequest() replaces each wrapped model's config.fetch with a request-local collector. Concurrent calls mutate the same shared config.fetch, causing one collector to capture multiple requests while another captures none. The resulting gateway responses can then be missing or returned to the wrong caller.

Environment

• ai-gateway-provider@3.2.0
• ai@6.0.208
• Node.js 24.12.0

Reproduction

mkdir ai-gateway-concurrency-repro
cd ai-gateway-concurrency-repro
npm init -y
npm install ai@6.0.208 ai-gateway-provider@3.2.0

Save as repro.mjs:

import { generateText } from 'ai';
import { createAiGateway } from 'ai-gateway-provider';
import { createDeepSeek } from 'ai-gateway-provider/providers/deepseek';

const gateway = createAiGateway({
  accountId: 'test',
  gateway: 'test',
  apiKey: 'test',
});

const deepseek = createDeepSeek();
const model = gateway(deepseek('deepseek-chat'));
const gatewayBodies = [];

globalThis.fetch = async (_url, init) => {
  const body = JSON.parse(init.body);
  const prompts = body.map(request => request.query?.messages?.at(-1)?.content);
  gatewayBodies.push(prompts);

  const prompt = prompts[0];
  return new Response(
    JSON.stringify({
      id: 'id',
      object: 'chat.completion',
      created: 0,
      model: 'deepseek-chat',
      choices: [
        {
          index: 0,
          message: { role: 'assistant', content: `ANSWER:${prompt}` },
          finish_reason: 'stop',
        },
      ],
      usage: {
        prompt_tokens: 1,
        completion_tokens: 1,
        total_tokens: 2,
      },
    }),
    {
      headers: {
        'content-type': 'application/json',
        'cf-aig-step': '0',
      },
    },
  );
};

const ask = prompt => generateText({ model, prompt, maxRetries: 0 }).then(result => result.text);

const [a, b] = await Promise.all([ask('A'), ask('B')]);

console.log({ a, b, gatewayBodies });

Run:

node repro.mjs

Output:

{
  a: 'ANSWER:undefined',
  b: 'ANSWER:A',
  gatewayBodies: [ [], [ 'A', 'B' ] ]
}

The reproduction stubs only the final gateway fetch; no Cloudflare or provider credentials are required.

Expected behavior

Concurrent calls using the same model remain isolated:

{ a: 'ANSWER:A', b: 'ANSWER:B', gatewayBodies: [ [ 'A' ], [ 'B' ] ] }

Actual behavior

One gateway request has an empty body and the other contains both provider requests. The responses are consequently missing or assigned to the wrong caller.

Workaround

Create the provider model and gateway wrapper separately for each call instead of sharing the resulting model instance.

[mattzcarey]

@agent-think can you repro this pls

[agent-think]

Requested by @mattzcarey

Verdict: ✅ Reproduced

I reproduced this with the reported ai-gateway-provider@3.2.0 and ai@6.0.208 (Node v24.16.0), without Cloudflare or provider credentials. The exact Node repro from the issue and the deployed Worker both consistently crossed the concurrent calls; the live endpoint reproduced 5/5 times.

Repro URL (expires after 60 mins): https://repro-ai-gateway-concurrency-620.rhetorical-carob.workers.dev

Open it, press Trigger bug, and watch the log show actual versus expected values.

Repro branch: https://github.com/cloudflare/ai/tree/repro/issue-620 — pull this branch to run the repro yourself. npm run repro:node runs the credential-free Node case; npm run deploy builds and deploys the clickable Vite UI.

Minimal repro

package.json       # pins ai-gateway-provider 3.2.0 + ai 6.0.208
repro.mjs          # exact global-fetch Node reproduction from the issue
src/server.ts      # shared model + Promise.all; fake final Gateway binding
src/client.tsx     # Trigger bug button and visible expected/actual log
wrangler.jsonc     # Worker, assets, and Durable Object configuration

The core operation is:

const deepseek = createDeepSeek();
const model = gateway(deepseek("deepseek-chat")); // created once and shared
const ask = prompt =>
  generateText({ model, prompt, maxRetries: 0 }).then(result => result.text);
const [a, b] = await Promise.all([ask("A"), ask("B")]);

Observed vs expected

The exact Node repro produced:

{
  "a": "ANSWER:undefined",
  "b": "ANSWER:A",
  "gatewayBodies": [[], ["A", "B"]]
}

The deployed endpoint returned HTTP 200 with the same result:

{"a":"ANSWER:undefined","b":"ANSWER:A","errors":[null,null],"gatewayBodies":[[],["A","B"]],"reproduced":true}

Expected isolation is:

{
  "a": "ANSWER:A",
  "b": "ANSWER:B",
  "gatewayBodies": [["A"], ["B"]]
}

Root-cause hypothesis

In the source for the tested 3.2.0 tag, processModelRequest() installs each call's request collector by mutating the wrapped model's shared model.config.fetch. Because provider execution yields, a concurrent call can replace that function before either provider fetch occurs, so one collector receives both requests and the other receives none. The response replay path then mutates the same shared config again, allowing the returned gateway response to be crossed as well. Request-scoped model/config instances (or another mechanism that avoids mutating the shared model) should remove both races.

🤖 generated by the repro-agent

[vibhu-dixit]

can i pick up this issue