Skip to content

fix(recorder): record empty JSON Anthropic, Gemini and Ollama turns as empty content - #623

Open
abhiramshaji-cyber wants to merge 2 commits into
CopilotKit:mainfrom
abhiramshaji-cyber:fix/recorder-empty-turns
Open

abhiramshaji-cyber wants to merge 2 commits into
CopilotKit:mainfrom
abhiramshaji-cyber:fix/recorder-empty-turns

Conversation

@abhiramshaji-cyber

Copy link
Copy Markdown
Contributor

When a recorded Anthropic, Gemini or Ollama reply came back empty without streaming, replay answered that request with an error, and now it replays the same empty reply the streamed recording gives.

What this code is for

proxyAndRecord forwards an unmatched request to the real provider, saves the reply as a fixture, and relays it. For a JSON (not streamed) reply it calls buildFixtureResponse, which checks the body against each provider's shape in turn and returns the fixture. If no branch claims the body, the last block saves { error: { type: "proxy_error" }, status }.

Before this PR

A test author records a Claude call with stream: false, and the model ends the turn with nothing to say: { "type": "message", "content": [], "stop_reason": "end_turn" }.

  1. proxyAndRecord gets a 200 and calls buildFixtureResponse.
  2. The Anthropic branch is gated on obj.content.length > 0, so an empty content skips it. A turn with one empty text block enters it but only returns for thinking, redacted or cited blocks, so it falls out too.
  3. No later branch matches, so the unrecognized fallback saves {"error":{"type":"proxy_error"},"status":200}.
  4. On replay the same request gets an error instead of an empty reply.

The same happens for a Gemini candidate with no parts (finishReason: "MAX_TOKENS", or "SAFETY" with no content), which the parts check skips, for a Gemini prompt blocked before any candidate (promptFeedback only), and for an Ollama /api/chat reply with message.content: "" and done_reason: "length", which the Ollama branch never returns for. Streamed, all of these already record { "content": "" }.

After this PR

Same author, same call.

  1. Same as above through step 1.
  2. The Anthropic branch now also takes content: [] when type is "message", and its last line returns { ...usageSpread, content: "" } for any turn it did not return for earlier.
  3. The fixture is { "usage": { "input_tokens": 5, "output_tokens": 0 }, "content": "" }, the same object the streamed recording writes.
  4. On replay the request gets an empty reply with the real usage.

The diagram

flowchart LR

classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827
classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px
classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px

subgraph BEFORE["BEFORE: empty 200 saved as proxy_error"]
direction TB
  B1["Claude replies content: [] with status 200"] --> B2["proxyAndRecord"]
  B2 --> B3["buildFixtureResponse"]
  B3 --> B4{"content.length > 0 ?"}
  B4 -->|"no, falls through"| B5["error fixture, replay fails"]
  B4 -->|"yes, has text"| B6["content fixture"]
end

subgraph AFTER["AFTER: empty 200 saved as content empty"]
direction TB
  A1["Claude replies content: [] with status 200"] --> A2["proxyAndRecord"]
  A2 --> A3["buildFixtureResponse"]
  A3 --> A4{"THE CHANGE: length > 0 or type is message"}
  A4 -->|"empty turn"| A5["content empty plus usage, replay works"]
  A4 -->|"has text"| A6["content fixture, unchanged"]
end

B1 ~~~ A1

class B1,B2,B3,B6 box
class A1,A2,A3,A6 box
class B4,B5 bad
class A4,A5 good

linkStyle 3 stroke:#cc0000,stroke-width:5px
linkStyle 8 stroke:#008800,stroke-width:5px

style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827
style AFTER fill:#f3fff3,stroke:#008800,color:#111827
Loading

The change

All in buildFixtureResponse:

  • Anthropic: the gate is obj.content.length > 0 || obj.type === "message", and the old if (thinkingBlocks.length > 0 || redactedBlocks.length > 0 || "blocks" in citedSpread) around the last return is gone, so a recognized message always ends in content: "" with its usage, reasoning, redacted thinking and citations kept. redactedBlocks was only used by that if, so it went too.
  • Gemini: the branch is entered for Array.isArray(obj.candidates) || isPlainObject(obj.promptFeedback), and a candidate without content.parts reads as parts = [], so it reaches the existing "Recognized Gemini shape but empty content" return. Most of that hunk is the one level of indent the inner if used to add; it reads much smaller with whitespace hidden.
  • Ollama: if (msg.content === "" && typeof obj.done === "boolean") return { ...usageSpread, content: "" }, after the text and tool call returns. done is the same check the /api/generate branch uses.

src/__tests__/recorder-empty-turns.test.ts records each of the six empty shapes twice through the real proxyAndRecord, once as JSON and once streamed, and expects the two fixtures to be equal. All six fail on main with the proxy_error fixture and pass here. I also added a line under [Unreleased] in CHANGELOG.md.

What changes for a user

  • Empty JSON replies from Claude, Gemini and Ollama now replay as an empty reply, the same as their streamed recordings.
  • Usage is kept where the recorder records it (Anthropic); Gemini and Ollama still record no usage on either path, as before.
  • Replies with text, tool calls, thinking, citations or audio record exactly as before.
  • A SAFETY stop records as plain empty content, matching the streamed path, which also keeps no finishReason.

buildFixtureResponse with its one caller proxyAndRecord and its callees
buildFixtureResponse with its one caller proxyAndRecord and its callees, Functions view in Architect

One question, since it changes the fix: for a Gemini SAFETY stop or a blocked prompt, is plain empty content what you want, or would you rather keep those as a distinct fixture (for example with the finishReason)? Happy to change it either way.

… turns as empty content

A successful reply with nothing in it, such as an Anthropic turn with
content [] or one empty text block, a Gemini candidate stopped at
MAX_TOKENS or SAFETY with no parts, a Gemini prompt blocked before any
candidate, or an Ollama chat reply cut at the length limit, fell through
every branch of buildFixtureResponse and was saved as a proxy_error
fixture. Replay then answered with an error for a request that worked.
The streamed version of the same turn already records content "".

Anthropic now owns any message it recognizes and records content "",
Gemini treats a candidate without parts as an empty parts list, and
Ollama records an empty chat message when the reply carries done.
… twin

Each case records the same empty turn twice through the real
proxyAndRecord, once as JSON and once streamed, and expects the two
fixtures to be identical.
@pkg-pr-new

pkg-pr-new Bot commented Nov 17, 2026

Copy link
Copy Markdown

Open in StackBlitz

npm i https://pkg.pr.new/@copilotkit/aimock@623

commit: df5e22b

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant