Repository navigation
fix(recorder): record empty JSON Anthropic, Gemini and Ollama turns as empty content - #623
Open
abhiramshaji-cyber wants to merge 2 commits into
Open
abhiramshaji-cyber wants to merge 2 commits into
abhiramshaji-cyber wants to merge 2 commits into
Conversation
… turns as empty content A successful reply with nothing in it, such as an Anthropic turn with content [] or one empty text block, a Gemini candidate stopped at MAX_TOKENS or SAFETY with no parts, a Gemini prompt blocked before any candidate, or an Ollama chat reply cut at the length limit, fell through every branch of buildFixtureResponse and was saved as a proxy_error fixture. Replay then answered with an error for a request that worked. The streamed version of the same turn already records content "". Anthropic now owns any message it recognizes and records content "", Gemini treats a candidate without parts as an empty parts list, and Ollama records an empty chat message when the reply carries done.
… twin Each case records the same empty turn twice through the real proxyAndRecord, once as JSON and once streamed, and expects the two fixtures to be identical.
commit: |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When a recorded Anthropic, Gemini or Ollama reply came back empty without streaming, replay answered that request with an error, and now it replays the same empty reply the streamed recording gives.
What this code is for
proxyAndRecordforwards an unmatched request to the real provider, saves the reply as a fixture, and relays it. For a JSON (not streamed) reply it callsbuildFixtureResponse, which checks the body against each provider's shape in turn and returns the fixture. If no branch claims the body, the last block saves{ error: { type: "proxy_error" }, status }.Before this PR
A test author records a Claude call with
stream: false, and the model ends the turn with nothing to say:{ "type": "message", "content": [], "stop_reason": "end_turn" }.proxyAndRecordgets a 200 and callsbuildFixtureResponse.obj.content.length > 0, so an emptycontentskips it. A turn with one empty text block enters it but only returns for thinking, redacted or cited blocks, so it falls out too.{"error":{"type":"proxy_error"},"status":200}.The same happens for a Gemini candidate with no
parts(finishReason: "MAX_TOKENS", or"SAFETY"with nocontent), which the parts check skips, for a Gemini prompt blocked before any candidate (promptFeedbackonly), and for an Ollama/api/chatreply withmessage.content: ""anddone_reason: "length", which the Ollama branch never returns for. Streamed, all of these already record{ "content": "" }.After this PR
Same author, same call.
content: []whentypeis"message", and its last line returns{ ...usageSpread, content: "" }for any turn it did not return for earlier.{ "usage": { "input_tokens": 5, "output_tokens": 0 }, "content": "" }, the same object the streamed recording writes.The diagram
flowchart LR classDef box fill:#eef1ff,stroke:#4c5bd4,color:#111827 classDef bad fill:#ffd6d6,stroke:#cc0000,color:#111827,stroke-width:3px classDef good fill:#c8f0c8,stroke:#008800,color:#111827,stroke-width:3px subgraph BEFORE["BEFORE: empty 200 saved as proxy_error"] direction TB B1["Claude replies content: [] with status 200"] --> B2["proxyAndRecord"] B2 --> B3["buildFixtureResponse"] B3 --> B4{"content.length > 0 ?"} B4 -->|"no, falls through"| B5["error fixture, replay fails"] B4 -->|"yes, has text"| B6["content fixture"] end subgraph AFTER["AFTER: empty 200 saved as content empty"] direction TB A1["Claude replies content: [] with status 200"] --> A2["proxyAndRecord"] A2 --> A3["buildFixtureResponse"] A3 --> A4{"THE CHANGE: length > 0 or type is message"} A4 -->|"empty turn"| A5["content empty plus usage, replay works"] A4 -->|"has text"| A6["content fixture, unchanged"] end B1 ~~~ A1 class B1,B2,B3,B6 box class A1,A2,A3,A6 box class B4,B5 bad class A4,A5 good linkStyle 3 stroke:#cc0000,stroke-width:5px linkStyle 8 stroke:#008800,stroke-width:5px style BEFORE fill:#fff5f5,stroke:#cc0000,color:#111827 style AFTER fill:#f3fff3,stroke:#008800,color:#111827The change
All in
buildFixtureResponse:obj.content.length > 0 || obj.type === "message", and the oldif (thinkingBlocks.length > 0 || redactedBlocks.length > 0 || "blocks" in citedSpread)around the last return is gone, so a recognized message always ends incontent: ""with its usage, reasoning, redacted thinking and citations kept.redactedBlockswas only used by thatif, so it went too.Array.isArray(obj.candidates) || isPlainObject(obj.promptFeedback), and a candidate withoutcontent.partsreads asparts = [], so it reaches the existing "Recognized Gemini shape but empty content" return. Most of that hunk is the one level of indent the innerifused to add; it reads much smaller with whitespace hidden.if (msg.content === "" && typeof obj.done === "boolean") return { ...usageSpread, content: "" }, after the text and tool call returns.doneis the same check the/api/generatebranch uses.src/__tests__/recorder-empty-turns.test.tsrecords each of the six empty shapes twice through the realproxyAndRecord, once as JSON and once streamed, and expects the two fixtures to be equal. All six fail on main with theproxy_errorfixture and pass here. I also added a line under[Unreleased]inCHANGELOG.md.What changes for a user
finishReason.buildFixtureResponsewith its one callerproxyAndRecordand its callees, Functions view in ArchitectOne question, since it changes the fix: for a Gemini SAFETY stop or a blocked prompt, is plain empty content what you want, or would you rather keep those as a distinct fixture (for example with the
finishReason)? Happy to change it either way.