Skip to content

fix(parser): fail closed on multiple concatenated tool envelopes - #1788

Open
brkastner wants to merge 1 commit into
agent0ai:mainfrom
brkastner:kas/fix-multi-tool-envelope
Open

fix(parser): fail closed on multiple concatenated tool envelopes#1788
brkastner wants to merge 1 commit into
agent0ai:mainfrom
brkastner:kas/fix-multi-tool-envelope

Conversation

@brkastner

Copy link
Copy Markdown

Problem

When a model emits two concatenated top-level JSON tool envelopes in one turn, Agent Zero renders the raw tool protocol to the user as if it were a chat message. Nothing executes, no repair warning fires, and the unusable-response circuit breaker never counts the turn.

Observed with the Codex OAuth provider in Responses mode (gpt-5.6-sol), where the assistant produced:

{"thoughts":[...],"tool_name":"skills_tool","tool_args":{...}}{"thoughts":[...],"tool_name":"response","tool_args":{"text":"..."}}

The two objects are adjacent with no separator. The user sees both JSON blobs verbatim in the chat window.

Root cause

extract_json_root_strings() correctly enumerates every top-level JSON root. The consumers do not.

extract_tool_request() requires root == content.strip(). With a second envelope present that equality fails, so it returns None. is_misformatted_tool_request() has no notion of multiple roots, so it also returns False.

That combination is a silent fail-open. In agent.py the Responses path treats "no tool request and not misformatted" as ordinary prose and wraps it as response(text=...), so raw tool protocol reaches the user. process_tools() never reaches the fw.msg_misformat.md branch, so StopUnusableResponseLoop never increments _unusable_response_failures. A model that repeats the behaviour is not stopped.

Fix

Classify multiple top-level roots as misformatted when at least one root parses as a tool request.

No root is selected and none is executed. Picking the first envelope would run an action the model did not request in isolation, so this fails closed instead. Routing through the existing misformat path means the current repair warning and circuit breaker handle the turn with no new prompt and no new state.

The change is additive: one new function plus a five-line guard in is_misformatted_tool_request(). extract_tool_request(), extract_json_root_string(), json_parse_dirty(), extract_json_root_strings(), and _is_tool_request() are untouched.

Behaviour change

Measured against the pre-existing public API. RENDERS_RAW is the fail-open condition in agent.py that leaks tool protocol into the chat.

input before after
single envelope executes tool unchanged
tool + response concatenated RENDERS_RAW misformat, repair
tool + tool concatenated RENDERS_RAW misformat, repair
literal }{ inside a string value executes tool unchanged
prose containing ordinary JSON unchanged unchanged
fenced tool JSON misformat unchanged
truncated JSON unchanged unchanged
multiple roots, none a tool request not flagged unchanged

Only the two broken cases move.

Tests

Adds tests/test_multiple_tool_roots.py, 15 tests: the concatenation cases, the circuit-breaker path through StopUnusableResponseLoop at the configured limit, Responses mode not wrapping concatenated protocol as text, and over-trigger guards for literal braces inside strings, nested braces, prose, truncated input, non-string input, and multiple non-tool roots.

Full suite, same container image, stock vs patched:

stock patched
passed 1048 1063
failed 60 60
skipped 1 1

Identical failure set, plus the 15 new tests. The 60 failures and 12 collection errors reproduce on unmodified main and are unrelated to this change.

Note on the provider side

output_text() in helpers/litellm_transport.py joins text across all message items and content blocks with "".join(pieces). If a provider returns two items that each contain a complete envelope, that join produces exactly this shape with no delimiter. I have not changed it, since the correct separator depends on provider semantics and the parser should fail closed regardless. Flagging it as a likely upstream contributor.

Codex OAuth in Responses mode emitted two concatenated top-level JSON
roots. extract_tool_request() returned None (root != content) and
is_misformatted_tool_request() returned False, so the Responses branch in
agent.py wrapped the raw tool protocol as response(text=...) and rendered
it to the user verbatim.

Classify multiple top-level roots as misformatted when at least one root
parses as a tool request. Nothing is selected or executed, so the
existing repair warning and unusable-response circuit breaker handle the
turn. No behaviour change for single envelopes, literal }{ inside
strings, prose, fenced or truncated JSON.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant