Eval bug: Qwen3-VL MoE tool calls during reasoning are often not parsed properly

[aviallon]

Name and Version

Commit 86a3f0f + #17750

Operating systems

Linux

GGML backends

CUDA

Hardware

NVIDIA L40S

Models

unsloth/Qwen3-VL-30B-A3B-Thinking-1M-GGUF:IQ3_XXS

Problem description & steps to reproduce

Here is an example of payload that can trigger the bug. You may have to restart the request several times to make the issue occur.
request_body(14).json

First Bad Commit

No response

Relevant log output

N/A

[aviallon]

In fact, it looks like we don't parse tool calls emitted during reasoning at all, but this can happen, esp. when we have tool_choice: 'required'

[aviallon]

Tentative fix:

Patch working-around the issue (LLM generated with minor edits)

commit 2c7d654bad93d8602f5fac21fb757742e5b964bb
Author: Antoine Viallon <antoine@lesviallon.fr>
Date:   Thu Dec 11 15:06:12 2025 +0100

    common: fix tool call parsing in reasoning

diff --git a/common/chat-parser.cpp b/common/chat-parser.cpp
index 7a7cfdf0f..976c63f94 100644
--- a/common/chat-parser.cpp
+++ b/common/chat-parser.cpp
@@ -50,6 +50,78 @@ static std::string wrap_code_as_arguments(common_chat_msg_parser & builder, cons
     return arguments;
 }
 
+// Lightweight brace-depth scan to recover JSON tool calls without emitting
+// fragmented partials. Counts braces outside of strings and returns each
+// balanced object containing a "name" field.
+static std::optional<std::vector<json>> scan_tool_calls_by_brace(const std::string & text, bool & partial_found) {
+    partial_found = false;
+    std::vector<json> results;
+
+    auto find_closing = [](const std::string & s, size_t start, bool & partial) -> size_t {
+        GGML_ASSERT(s[start] == '{');
+        int depth = 0;
+        bool in_string = false;
+        bool escape = false;
+        for (size_t i = start; i < s.size(); ++i) {
+            char c = s[i];
+            if (escape) {
+                escape = false;
+                continue;
+            }
+            if (c == '\\' && in_string) {
+                escape = true;
+                continue;
+            }
+            if (c == '"') {
+                in_string = !in_string;
+                continue;
+            }
+            if (in_string) {
+                continue;
+            }
+            if (c == '{') {
+                ++depth;
+            } else if (c == '}') {
+                --depth;
+                if (depth == 0) {
+                    return i;
+                }
+            }
+        }
+        partial = true;
+        return std::string::npos;
+    };
+
+    size_t pos = 0;
+    while (true) {
+        pos = text.find('{', pos);
+        if (pos == std::string::npos) {
+            break;
+        }
+        bool partial = false;
+        auto end = find_closing(text, pos, partial);
+        if (partial) {
+            partial_found = true;
+            break;
+        }
+        std::string candidate = text.substr(pos, end - pos + 1);
+        pos = end + 1;
+        try {
+            auto j = json::parse(candidate);
+            if (j.is_object() && j.contains("name")) {
+                results.push_back(j);
+            }
+        } catch (const std::exception &) {
+            continue;
+        }
+    }
+
+    if (results.empty()) {
+        return std::nullopt;
+    }
+    return results;
+}
+
 /**
  * Takes a prefix regex that must have 1 group to capture the function name, a closing suffix, and expects json parameters in between.
  * Aggregates the prefix, suffix and in-between text into the content.
@@ -325,6 +397,35 @@ bool common_chat_msg_parser::try_parse_reasoning(const std::string & start_think
         if (stripped_reasoning.empty()) {
             return;
         }
+
+        // Opportunistically extract balanced tool calls that models may emit inside <think> blocks.
+        if (syntax_.parse_tool_calls && !is_partial()) {
+            bool brace_partial = false;
+            if (auto scanned = scan_tool_calls_by_brace(stripped_reasoning, brace_partial)) {
+                if (!brace_partial) {
+                    for (const auto & j : *scanned) {
+                        if (!j.is_object()) {
+                            continue;
+                        }
+                        std::string name = j.contains("name") ? j.at("name").template get<std::string>() : "";
+                        if (name.empty()) {
+                            continue;
+                        }
+                        std::string id;
+                        if (j.contains("id") && j.at("id").is_string()) {
+                            id = j.at("id").template get<std::string>();
+                        }
+                        std::string arguments;
+                        if (j.contains("arguments")) {
+                            const auto & args = j.at("arguments");
+                            arguments = args.is_string() ? args.template get<std::string>() : args.dump();
+                        }
+                        add_tool_call(name, id, arguments);
+                    }
+                }
+            }
+        }
+
         if (syntax_.reasoning_in_content) {
             add_content(syntax_.reasoning_format == COMMON_REASONING_FORMAT_DEEPSEEK ? "<think>" : start_think);
             add_content(stripped_reasoning);

Prompt used for generation: Some models emit tool calls during reasoning. Handle tool_call parsing into try_parse_reasoning, using a simple brace-depth parser.

[aldehir]

I think we're better off implementing a parser with the new PEG parser. We can also standardize a way to implement grammar based reasoning budget. I'll look into it this weekend.

[aviallon]

I believe you are right.
I'd also love to be able to override the grammar (via /v1/chat/completions and others) in all cases, even when tool call are enabled.
I don't really understand why we restrict that.

[aldehir]

I'd also love to be able to override the grammar (via /v1/chat/completions and others) in all cases, even when tool call are enabled.
I don't really understand why we restrict that.

I'm guessing it's because you can't really guarantee that the user will produce a grammar that is parseable. I can see it being a problem with models like gpt-oss where it emits harmony tokens and the grammar must include them or it'll generate gibberish. Do you want to constrain just the content or also the tool calling as well?

[aviallon]

You obviously can't guarantee things'll work if the user overrides the grammar, but that is what the user signed up for, didn't he?

I am interested in constraining the entire model output here.
I could use the apply_template and the regular completions API, but I'd rather not to.