Eval bug: Devstral 2 over-constrained during tool call

[aldehir]

Name and Version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
version: 7356 (34ce48d)
built with MSVC 19.44.35221.0 for Windows AMD64

Operating systems

Windows

GGML backends

CUDA

Hardware

NVIDIA RTX 3090

Models

Devstral-2-24B-2512 and likely the other recent Mistral models

Problem description & steps to reproduce

Devstral 2 does not produce valid JSON, by design. For strings it only escapes double quotes \" and nothing else. Example:

"content": "regex = r'\'[^\"]\'"

Note the escaped \' to produce valid code and the escaped \" to not match the closing double quote of the "JSON" string.

The json-schema-to-grammar implementation will constrain it to escape backslashes, so \' => \\' and in turn degrades model output quality. Occasionally leading to an infinite generation of \\\\\\\\\\....

First Bad Commit

No response

Relevant log output

# Added logs to common/sampling.cpp
[35454] Grammar accepted token 17931: \"
[35454] Grammar accepted token 17743: |\
[35454] Grammar rejected token 1039, resampling: '

# Properly escapes \" but not ', should be \\' to be valid JSON

[bonswouar]

Not sure if it's related but there is now: https://github.com/jeffersonwarrior/devstral-proxy

[aldehir]

@bonswouar not quite. llama.cpp already maps its OpenAI Compatible API to the input required by the Mistral chat template. It also doesn't use Mistral's python library, and I assume vLLM does.

This is about the model not properly escaping strings within JSON, which could just be the model. If it starts infinitely generating garbage because it was forced to produce valid JSON, however, then that is something worth looking into.

That said, #17945 seems to have alleviated most of the issues. Before it would refuse to escape Windows file paths, e.g. E:\Development => E:\\Development. I no longer see that problem after several runs.

However, I still see the issue where it tries to generate \' to escape a ' within a string, but needs to produce \\' so it is properly escaped for JSON. The grammar will force it to generate \" (because \' is not a valid escape sequence) which results in incorrect code. I believe I can address this in the parsing/grammar by make it a bit more lenient and relying on the agent loop to correct itself.