[v2.8] Chat stuck in Processing with Kimi: empty streamed completions counted as unusable responses

[TurkerTunali]

GitHub Issue Draft (agent-zero repo)

Title

[v2.8] Chat stuck in "Processing" with Kimi chat model: empty streamed completions + leaked tool-markup tokens counted as unusable responses

Summary

Since upgrading to v2.8 (git tag v2.8, HEAD 5ff106a2), conversations using a Kimi chat model (Moonshot Kimi-K3 / kimi-k2.6 through an OpenAI-compatible custom base URL, provider type other) regularly appear stuck at "Processing…" in the WebUI. The final answer IS rendered, but afterwards the agent runs several extra retry iterations (each shown as another "TOOL CALL … Processing" bubble) plus UTL warnings (A0: Message misformat, no valid tool request found.), until the unusable-response loop guard terminates the turn.

Same model, same preset, same proxy worked cleanly before the v2.8 update → looks like a v2.8 regression on the response-intake side, not a model capability issue.

Evidence (captured from a live v2.8 chat, usr/chats/<id>/chat.json)

Warnings in ONE chat session:

log entry 35: A0: Message misformat, no valid tool request found.
log entry 66: A0: Message misformat, no valid tool request found.
log entry 75: A0: Message misformat, no valid tool request found.
log entry 77: You have sent the same message again... (retry feedback loop)

Triggering AI messages extracted verbatim (see attached evidence.md):

1. Three messages with EMPTY content — length 0, md5 d41d8cd98f00b204e9800998ecf8427e (md5 of empty string). No JSON tool request can be parsed from an empty string, so each one counts as an unusable response.
2. One message whose content was the model's internal special-token tool markup (native tool-call template control tokens emitted as plain text) instead of the required JSON tool request object. The extractor cannot parse this format.

Root-cause analysis (v2.8 code pointers)

• Chat flow always streams: models.py ~L569 sets stream = True whenever reasoning/response/tokens callbacks exist — and they always exist in the agent loop, so every v2.8 chat completion uses the SSE path.
• Stream text comes only from delta.content: helpers/litellm_transport.py ~L509-524 — reasoning_content is captured separately as reasoning_delta and never merged into the tool-request text. Kimi k2/k3 models routinely emit reasoning_content deltas, and the provider can emit a terminal chunk with finish_reason set but empty content → accumulated text is '' → fw.msg_misformat warning.
• Loop guard operates as designed: extensions/python/_functions/agent/Agent/hist_add_warning/end/_90_stop_unusable_response_loop.py counts consecutive misformat/repeat warnings and stops the turn at max_consecutive_unusable_responses. The visible post-answer "Processing" phase = these guard-driven retries.
• Raising the limit 2 → 5 (Settings → Agent) did NOT fix it — the stuck phase only got longer. This confirms the trigger is upstream (empty/unparseable completions), not the guard itself.

Root-cause statement

v2.8's streaming intake treats two Kimi-specific stream artifacts as unusable output:

• A) Empty-terminal-chunk completions — the stream is valid SSE, but accumulated content is an empty string.
• B) Leaked internal tool-markup tokens — the model's native tool-call template tokens are delivered as plain content, which the JSON parser rejects.
  Pre-v2.8 behavior apparently never surfaced these, which explains why the identical model setup regressed on upgrade.

Environment

• Agent Zero v2.8, official Docker image (run_ui.py), git describe --tags → v2.8
• Chat model: provider other, custom OpenAI-compatible base URL, upstream targets moonshotai/Kimi-K3 (primary) and cf/@cf/moonshotai/kimi-k2.6 (failover)
• max_consecutive_unusable_responses = 5 (raised from default 2 — no fix, only longer retry phase)
• Reproduces consistently (multiple chats, every Kimi chat affected)

Reproduction

1. Configure chat model = a Kimi k2/k3-family model through a custom OpenAI-compatible base URL (standard SSE).
2. Run a normal tool-heavy, multi-step task.
3. Observe: the final answer renders, then N extra "Processing…" iterations with misformat warnings, then the loop guard hard-stops the turn.

Cross-check (rules out a simple format-following failure)

A standalone test script (attached tool_call_format_test.py) calling the SAME endpoint/models directly shows the models DO return the required prompt-JSON ({"tool_name": "response", ...}) whenever content is populated. Failures occur only with the two artifact classes above.

Suggested fixes (any one would likely resolve it)

1. Empty-completion tolerance: if a streamed call ends with finish_reason set but empty accumulated content, treat as transient — retry the completion once silently (or do one non-stream fallback attempt) BEFORE counting an unusable response.
2. Reasoning-merge fallback: when content ended empty but reasoning_content accumulated, try parsing the reasoning text as a last-resort source of the tool request in ChatCompletionsTransport. This helps reasoning-style model families generally (Kimi k2/k3, DeepSeek R1-class, GLM-thinking).
3. Detect leaked tool-markup: recognize native tool-call control-token sequences in extract_tools.is_misformatted_tool_request and either convert them or log a distinct diagnostic warning (currently generic misformat burns retries).
4. UI/UX: collapse loop-guard retries into one expandable warning instead of rendering each as a fresh "TOOL CALL … Processing" bubble — users currently think the agent froze even when the answer already arrived.

Workarounds tried (did NOT solve)

• max_consecutive_unusable_responses 2 → 5: only prolongs the visible Processing phase.
• Models pass direct tool-call format tests, so the issue is not general instruction-following.

Attachments

• evidence.md — verbatim triggering AI messages (3x empty + 1x leaked-markup fragment, md5-hashed)
• tool_call_format_test.py — standalone direct test (models return valid prompt-JSON)
• Full chat.json available privately; happy to test patches — reproduces reliably on my setup.