Eval bug: Devstral-Small-2-24B-Instruct-2512 doesn't pass the example tool call tests

[blakkd]

Setup:

• Release b7361,
• Linux
• Nvidia RTX 3090
• CUDA backend

When running Mistral proposed test #2 - Call tools one at a time subsequently, the output is never what is expected.

Disclaimer: I can't assure my GPU is functionning correctly and am affraid it's currently dying... This because I recently got weird behavior sometimes. Unfortunately my GPU doesn't support ECC (the error correction mechanism newer cards have) so I can't monitor the error counter, and all I could do is to run FurMark in artifacts detection mode, and had zero artifacts during about 10 min of testing. But maybe the issue is intermittent!

I'm still posting this issue but I guess it needs to be reproduced to confirm...

---

Command:

      /home/user/llama.cpp/build/bin/llama-server
      --model /mnt/storage/GGUFs/Devstral-Small-2-24B-Instruct-2512/Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf
      --mmproj /mnt/storage/GGUFs/Devstral-Small-2-24B-Instruct-2512/mmproj-BF16.gguf
      --ctx-size 30000
      --no-context-shift
      --n-gpu-layers 41
      --temp 0.15
      --top-p 0.95
      --top-k 20
      --min-p 0
      --repeat-penalty 1
      --jinja
      --chat-template-file /mnt/storage/GGUFs/Devstral-Small-2-24B-Instruct-2512/template_mistral.jinja #untouched original Mistral template & prompt
      --host 0.0.0.0
      --port ${PORT}

---

Payload

{
  "model": "Devstral-Small-2-24B-Instruct-2512_50k_vision",
  "messages": [
    {
      "role": "system",
      "content": "You are Devstral-Small-2-24B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris.\nYou power an AI assistant called Le Chat.\nYour knowledge base was last updated on 2023-10-01.\nThe current date is 2025-12-12.\n\nWhen you're not sure about some information or when the user's request requires up-to-date or specific data, you must use the available tools to fetch the information. Do not hesitate to use tools whenever they can provide a more accurate or complete response. If no relevant tools are available, then clearly state that you don't have the information and avoid making up anything.\nIf the user's question is not clear, ambiguous, or does not provide enough context for you to accurately answer the question, you do not try to answer it right away and you rather ask the user to clarify their request (e.g. \"What are some good restaurants around me?\" => \"Where are you?\" or \"When is the next flight to Tokyo\" => \"Where do you travel from?\").\nYou are always very attentive to dates, in particular you try to resolve dates (e.g. \"yesterday\" is 2025-12-11) and when asked about information at specific dates, you discard information that is at another date.\nYou follow these instructions in all languages, and always respond to the user in the language they use or request.\nNext sections describe the capabilities that you have.\n\n# WEB BROWSING INSTRUCTIONS\n\nYou cannot perform any web search or access internet to open URLs, links etc. If it seems like the user is expecting you to do so, you clarify the situation and ask the user to copy paste the text directly in the chat.\n\n# MULTI-MODAL INSTRUCTIONS\n\nYou have the ability to read images, but you cannot generate images. You also cannot transcribe audio files or videos.\nYou cannot read nor transcribe audio files or videos.\n\n# TOOL CALLING INSTRUCTIONS\n\nYou may have access to tools that you can use to fetch information or perform actions. You must use these tools in the following situations:\n\n1. When the request requires up-to-date information.\n2. When the request requires specific data that you do not have in your knowledge base.\n3. When the request involves actions that you cannot perform without tools.\n\nAlways prioritize using tools to provide the most accurate and helpful response. If tools are not available, inform the user that you cannot perform the requested action at the moment."
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "Compute the results steps by steps for the equations that involve only numbers displayed in the image. You have to call tools to perform the operations and can do one operation at a time per equation."
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "https://math-coaching.com/img/fiche/46/expressions-mathematiques.jpg"
          }
        }
      ]
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "add_number",
        "description": "Add two numbers.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "type": "string",
              "description": "The first number."
            },
            "b": {
              "type": "string",
              "description": "The second number."
            }
          },
          "required": [
            "a",
            "b"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "multiply_number",
        "description": "Multiply two numbers.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "type": "string",
              "description": "The first number."
            },
            "b": {
              "type": "string",
              "description": "The second number."
            }
          },
          "required": [
            "a",
            "b"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "substract_number",
        "description": "Substract two numbers.",
        "parameters": {
          "type": "object",
          "properties": {
            "a": {
              "type": "string",
              "description": "The first number."
            },
            "b": {
              "type": "string",
              "description": "The second number."
            }
          },
          "required": [
            "a",
            "b"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "write_a_story",
        "description": "Write a story about science fiction and people with badass laser sabers.",
        "parameters": {}
      }
    },
    {
      "type": "function",
      "function": {
        "name": "terminal",
        "description": "Perform operations from the terminal.",
        "parameters": {
          "type": "object",
          "properties": {
            "command": {
              "type": "string",
              "description": "The command you wish to launch, e.g `ls`, `rm`, ..."
            },
            "args": {
              "type": "string",
              "description": "The arguments to pass to the command."
            }
          },
          "required": [
            "command"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "python",
        "description": "Call a Python interpreter with some Python code that will be ran.",
        "parameters": {
          "type": "object",
          "properties": {
            "code": {
              "type": "string",
              "description": "The Python code to run"
            },
            "result_variable": {
              "type": "string",
              "description": "Variable containing the result you'd like to retrieve from the execution."
            }
          },
          "required": [
            "code",
            "result_variable"
          ]
        }
      }
    }
  ],
  "tool_choice": "auto"

---

Respecting the original proposed test prompt (English text + French image)

3 scenarios:

• The model stops calling the tools in the middle of the successive calls, but without error (case #1, #2 and #3)
• I get a malformed response (case #3 and #4)
• The model calls tools but is stuck in an infinite loop (I don't have the logs for this, sorry)

#1

[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "feDRqfFRt7EUA28XDdX0w1PpyjcVLWRo",
        "function": {
          "arguments": "{\"a\": \"2\", \"b\": \"3\"}",
          "name": "multiply_number"
        },
        "type": "function"
      }
    ],
    "content": "Pour r\u00e9soudre les \u00e9quations impliquant uniquement des nombres, je vais proc\u00e9der \u00e9tape par \u00e9tape en utilisant les outils disponibles. Voici les \u00e9quations \u00e0 r\u00e9soudre :\n\n1. \\( 6 + 2 \\times 3 \\)\n2. \\( 3a + 7b - 2a \\)\n3. \\( 19 - (8 + 2) + 1 \\)\n4. \\( 5y \\times (6x - 7) \\)\n\nJe vais commencer par la premi\u00e8re \u00e9quation. Pour r\u00e9soudre \\( 6 + 2 \\times 3 \\), je dois d'abord effectuer la multiplication \\( 2 \\times 3 \\). Ensuite, j'ajouterai le r\u00e9sultat \u00e0 6."
  },
  {
    "role": "tool",
    "tool_call_id": "feDRqfFRt7EUA28XDdX0w1PpyjcVLWRo",
    "name": "multiply_number",
    "content": "6.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "gIhIsPB5PfxiMkAAmNqmlROeWJtx0Bed",
        "function": {
          "arguments": "{\"a\": \"6\", \"b\": \"6.0\"}",
          "name": "add_number"
        },
        "type": "function"
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "gIhIsPB5PfxiMkAAmNqmlROeWJtx0Bed",
    "name": "add_number",
    "content": "12.0"
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "Le r\u00e9sultat de la premi\u00e8re \u00e9quation est **12.0**.\n\nPassons maintenant \u00e0 la deuxi\u00e8me \u00e9quation : \\\\( 3a + 7b - 2a \\\\).\n\nPour r\u00e9soudre cette \u00e9quation, je dois d'abord combiner les termes semblables. Je vais calculer \\\\( 3a - 2a \\\\) en utilisant l'outil de soustraction."
  }
]

#2

[
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "Pour r\u00e9soudre les \u00e9quations impliquant uniquement des nombres, je vais proc\u00e9der \u00e9tape par \u00e9tape en utilisant les outils disponibles. Voici les \u00e9quations \u00e0 r\u00e9soudre :\n\n1. \\( 6 + 2 \\times 3 \\)\n2. \\( 3a + 7b - 2a \\)\n3. \\( 19 - (8 + 2) + 1 \\)\n4. \\( 5y \\times (6x - 7) \\)\n\nJe vais commencer par la premi\u00e8re \u00e9quation."
  }
]

#3

Traceback (most recent call last):
  File "/home/user/temp_working_dir/devstral_test_2.py", line 241, in <module>
    result = MAP_FN[function_name](**json.loads(function_args))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/temp_working_dir/devstral_test_2.py", line 35, in multiply_number
    a, b = float(a), float(b)
                     ^^^^^^^^
ValueError: could not convert string to float: 'y'

#4

Traceback (most recent call last):
  File "/home/user/temp_working_dir/devstral_test_2.py", line 233, in <module>
    result = MAP_FN[function_name](**json.loads(function_args))
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/temp_working_dir/devstral_test_2.py", line 39, in substract_number
    a, b = float(a), float(b)
           ^^^^^^^^
ValueError: could not convert string to float: '3a'

#5

[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "jHSEVVlFW6wd3V6EJMpkYc0bWvLIM5fg",
        "function": {
          "arguments": "{\"a\": \"2\", \"b\": \"3\"}",
          "name": "multiply_number"
        },
        "type": "function"
      }
    ],
    "content": "Pour r\u00e9soudre les \u00e9quations impliquant uniquement des nombres, je vais proc\u00e9der \u00e9tape par \u00e9tape en utilisant les outils disponibles. Voici les \u00e9quations \u00e0 r\u00e9soudre :\n\n1. \\( 6 + 2 \\times 3 \\)\n2. \\( 3a + 7b - 2a \\)\n3. \\( 19 - (8 + 2) + 1 \\)\n4. \\( 5y \\times (6x - 7) \\)\n\nJe vais commencer par la premi\u00e8re \u00e9quation. Pour r\u00e9soudre \\( 6 + 2 \\times 3 \\), je dois d'abord effectuer la multiplication \\( 2 \\times 3 \\)."
  },
  {
    "role": "tool",
    "tool_call_id": "jHSEVVlFW6wd3V6EJMpkYc0bWvLIM5fg",
    "name": "multiply_number",
    "content": "6.0"
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "{\n  \"tool_calls\": [\n    {\n      \"name\": \"add_number\",\n      \"arguments\": {\n        \"a\": \"6\",\n        \"b\": \"6.0\"\n      },\n      \"id\": \"jHSEVVlF224B3V6EJMpkYc0bWvLIM5fg\"\n    }\n  ],\n  \"content\": [\n    {\n      \"type\": \"text\",\n      \"text\": \"Le r\u00e9sultat de \\\\( 2 \\\\times 3 \\\\) est \\\\( 6.0 \\\\). Maintenant, j'ajoute ce r\u00e9sultat \u00e0 6 pour r\u00e9soudre \\\\( 6 + 2 \\\\times 3 \\\\).\"\n    }\n  ]\n}"
  }
]

---

Translating the image to English using nano-banana (so English text + English image)

• Almost always makes all the tool calls, reaching the final step
• Almost always gets it wrong (but, that said, the "expected output" is wrong too, so...)

[ServeurpersoCom]

Hi, I just spun up the same model on a dedicated llama.cpp test server with an 5090. You can use it to determine if this is a hardware or software bug:

CLI :

llama-server -ngl 999 -ctk q8_0 -ctv q8_0 -fa on --mlock -np 4 -kvu --port 8082 --models-max 1 --models-preset config.ini

Presets :

[Dense-Devstral-Small-2-24B-Instruct-2512]
m = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q6_K_XL.gguf
c = 131072

[Dense-Vision-Devstral-Small-2-24B-Instruct-2512]
m = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q5_K_XL.gguf
mm = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/mmproj-BF16.gguf
c = 131072

OAI endpoint :
https://www.serveurperso.com/ia/v1/chat/completions

WebUI (with a MCP client) :
https://www.serveurperso.com/ia/new/

Server log :
https://www.serveurperso.com/ia/models/llama-server.log

My MCP tool calls work fine on this model, but I haven't tested your specific examples yet
If it crashes or restarts, it's because I'm on it

[blakkd]

"If it crashes or restarts, it's because I'm on it" 😂

That's kind! But actually I'd be more comfortable if you run it on your own machine! + I was using a Q4_K_XL so would be unfair to compare with your Q5_K_XL :)

I just used the test python script they give without touching anything apart loading the image locally instead of the web server to avoid flooding it :D And used Mistral's official untouched template.

[ServeurpersoCom]

I'm adding a preset with the Q4_K_XL as well + https://huggingface.co/mistralai/Devstral-Small-2-24B-Instruct-2512?chat_template=default jinja for realistic test

[Test-Devstral-Small-2-24B-Instruct-2512]
m = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf
mm = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/mmproj-BF16.gguf
chat-template-file = unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF/Devstral-Small-2-24B-Instruct-2512.jinja
c = 131072

Even if it works, the goal is to narrow down little by little. I try it (at least I'll have a test venv ready on the machine)

(root|~/17960) python3 -m venv .venv
(root|~/17960) source .venv/bin/activate
(.venv) (root|~/17960) pip3 install openai
(.venv) (root|~/17960) pip install huggingface_hub

in progress...

Ah yes, I need to tweak the Python code quite a bit or bypass the router which only does SSE!

[ServeurpersoCom]

My modded python :

test.py

Result :

(.venv) (root|~/17960) python3 test.py
DEBUG function_name: multiply_number
DEBUG function_args: '{"a":"2","b":"3"}'
DEBUG function_name: substract_number
DEBUG function_args: '{"a":"8","b":"2"}'
DEBUG function_name: add_number
DEBUG function_args: '{"a":"6","b":"6.0"}'
DEBUG function_name: add_number
DEBUG function_args: '{"a":"12.0","b":"1"}'
[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "z8RfRMuy9E2t3xb84KXf72Th7oOfe7yb",
        "type": "function",
        "function": {
          "name": "multiply_number",
          "arguments": "{\"a\":\"2\",\"b\":\"3\"}"
        }
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "z8RfRMuy9E2t3xb84KXf72Th7oOfe7yb",
    "name": "multiply_number",
    "content": "6.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "6Do40gNt1gDi4yS3lRTXiwYZS55nTWeO",
        "type": "function",
        "function": {
          "name": "substract_number",
          "arguments": "{\"a\":\"8\",\"b\":\"2\"}"
        }
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "6Do40gNt1gDi4yS3lRTXiwYZS55nTWeO",
    "name": "substract_number",
    "content": "6.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "3ZYfxPjqzzh2VNzaizbyN6zD43sSMiFH",
        "type": "function",
        "function": {
          "name": "add_number",
          "arguments": "{\"a\":\"6\",\"b\":\"6.0\"}"
        }
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "3ZYfxPjqzzh2VNzaizbyN6zD43sSMiFH",
    "name": "add_number",
    "content": "12.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "dGn8qIaKDPyWZmmU8S0MjsJoa8frK9fl",
        "type": "function",
        "function": {
          "name": "add_number",
          "arguments": "{\"a\":\"12.0\",\"b\":\"1\"}"
        }
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "dGn8qIaKDPyWZmmU8S0MjsJoa8frK9fl",
    "name": "add_number",
    "content": "13.0"
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "The result of the equation 1 is 13."
  }
]
(.venv) (root|~/17960)

Sometimes the model is wrong, it's not very deterministic.

[blakkd]

I ran many times to collect the different cases, maybe you could some more? Because, for me too it was sometimes going through (but giving a wrong or partial answer, like the run you showed).

[blakkd]

And already thanks for testing!

[ServeurpersoCom]

The model sometimes:
Refuses to call tools at all
Stops after 1-2 calls instead of completing all steps
Mixes up different equations
Uses wrong operations (subtract instead of add)...
-> But I'm not sure about my Python hack; I did it in a rush.
-> Also this model give impressive performance with Anthropic Claude Code CLI (lot of tool_calls) !

[aldehir]

Try building from master, #17945 was recently merged which I found helps with the output quality.

I ran the tests last night and they seemed fine and I also have a 3090. I'll run them again and posts the results shortly.

[aldehir]

Seems to fall within acceptable range for me.

https://github.com/aldehir/mistral-3-tests

Test 1

[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "AFPuKo4A5hNchMD5rnSuESkGDziae4Wj",
        "function": {
          "arguments": "{}",
          "name": "write_a_story"
        },
        "type": "function"
      }
    ],
    "content": null
  },
  {
    "role": "tool",
    "tool_call_id": "AFPuKo4A5hNchMD5rnSuESkGDziae4Wj",
    "name": "write_a_story",
    "content": "A long time ago in a galaxy far far away..."
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "A long time ago in a galaxy far, far away, there was a young warrior named Kael. He was known for his incredible skill with a laser saber, a weapon that emitted a brilliant beam of light. Kael was not just any warrior; he was a guardian of peace, sworn to protect the innocent from the forces of darkness.\n\nOne day, Kael received a distress call from a small planet on the outskirts of the galaxy. The planet was under attack by a ruthless warlord named Malrik, who sought to conquer it and use its resources to fuel his army. Kael knew he had to act quickly.\n\nHe boarded his ship, the Starhawk, and set off for the planet. Along the way, he encountered a group of rebels who had also been fighting against Malrik. Among them was a young woman named Lyra, who had a unique ability to manipulate energy fields. She had been captured by Malrik's forces but managed to escape.\n\nKael and Lyra joined forces, their skills complementing each other perfectly. Kael's laser saber was a formidable weapon, but Lyra's ability to manipulate energy fields gave them an edge in battle. Together, they infiltrated Malrik's stronghold, a massive fortress built into the side of a mountain.\n\nAs they made their way through the fortress, they encountered numerous challenges. Malrik had set up traps and guards at every turn, but Kael and Lyra were able to overcome them with their combined skills. Finally, they reached the heart of the fortress, where Malrik was waiting for them.\n\nThe battle that followed was intense. Kael's laser saber clashed against Malrik's own weapon, sending sparks flying in every direction. Lyra used her energy manipulation to disrupt Malrik's defenses, giving Kael the opening he needed to strike.\n\nIn the end, Kael and Lyra emerged victorious. Malrik was defeated, and his forces scattered. The planet was saved, and Kael and Lyra became heroes, their names sung throughout the galaxy.\n\nBut their adventure was far from over. As they stood on the balcony of the fortress, looking out at the stars, Kael knew that there were still many battles to be fought. He turned to Lyra and smiled. \"Ready for the next adventure?\" he asked.\n\nLyra grinned back at him. \"Always,\" she replied.\n\nAnd so, Kael and Lyra set off once more, their laser sabers at the ready, to face whatever challenges the galaxy threw their way. Their story was just beginning, and the universe was their playground."
  }
]

Test 2

[
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "6xc0t9okzOK7n36Xg051TkD9fxIAHxfd",
        "function": {
          "arguments": "{\"a\": \"2\", \"b\": \"3\"}",
          "name": "multiply_number"
        },
        "type": "function"
      }
    ],
    "content": "Let's solve the equations step by step, focusing only on those that involve only numbers.\n\n### Equation 1: 6 + 2 \u00d7 3\nFirst, we perform the multiplication:"
  },
  {
    "role": "tool",
    "tool_call_id": "6xc0t9okzOK7n36Xg051TkD9fxIAHxfd",
    "name": "multiply_number",
    "content": "6.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "NfvGzGN2KiFrY19l6NI69FPEn47gGfOx",
        "function": {
          "arguments": "{\"a\": \"6\", \"b\": \"6.0\"}",
          "name": "add_number"
        },
        "type": "function"
      }
    ],
    "content": "Now, we add the result to 6:"
  },
  {
    "role": "tool",
    "tool_call_id": "NfvGzGN2KiFrY19l6NI69FPEn47gGfOx",
    "name": "add_number",
    "content": "12.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "GI1XGP1dFG2x0YQxYKhebPAerOWOiewl",
        "function": {
          "arguments": "{\"a\": \"8\", \"b\": \"2\"}",
          "name": "add_number"
        },
        "type": "function"
      }
    ],
    "content": "The result of Equation 1 is **12**.\n\n### Equation 3: 19 - (8 + 2) + 1\nFirst, we solve the parentheses:"
  },
  {
    "role": "tool",
    "tool_call_id": "GI1XGP1dFG2x0YQxYKhebPAerOWOiewl",
    "name": "add_number",
    "content": "10.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "b4gPqDVAFhuWWHF50CBuYy8jaP9mBouK",
        "function": {
          "arguments": "{\"a\": \"19\", \"b\": \"10.0\"}",
          "name": "substract_number"
        },
        "type": "function"
      }
    ],
    "content": "Now, we subtract the result from 19:"
  },
  {
    "role": "tool",
    "tool_call_id": "b4gPqDVAFhuWWHF50CBuYy8jaP9mBouK",
    "name": "substract_number",
    "content": "9.0"
  },
  {
    "role": "assistant",
    "tool_calls": [
      {
        "id": "V3uZFXA5DfHqjKrYJj33IJ8ixfm2gGT0",
        "function": {
          "arguments": "{\"a\": \"9.0\", \"b\": \"1\"}",
          "name": "add_number"
        },
        "type": "function"
      }
    ],
    "content": "Finally, we add 1 to the result:"
  },
  {
    "role": "tool",
    "tool_call_id": "V3uZFXA5DfHqjKrYJj33IJ8ixfm2gGT0",
    "name": "add_number",
    "content": "10.0"
  },
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "The result of Equation 3 is **10**.\n\n### Summary of Results:\n- Equation 1: 6 + 2 \u00d7 3 = **12**\n- Equation 3: 19 - (8 + 2) + 1 = **10**"
  }
]

Test 3

[
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "The most powerful being in the universe is Dr. Strange.\n\nHere's a Python function that returns Dr. Strange as the most powerful being in the universe, regardless of the time and location provided:\n\n```python\ndef most_powerful_being(time=None, location=None):\n    return \"The most powerful being in the universe is Dr. Strange.\"\n```\n\nYou can call this function with any time and location arguments, and it will always return the same string:\n\n```python\nprint(most_powerful_being(time=\"2025-12-12\", location=\"Earth\"))\n# Output: The most powerful being in the universe is Dr. Strange.\n```"
  }
]

Test 4

[
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "If you can't use JavaScript and your team dislikes PHP, there are still several robust alternatives for developing a web server. Here are some options, depending on your preferences and requirements:\n\n### 1. **Python (Flask, Django, FastAPI)**\n   - **Why?** Python is a versatile language with excellent web frameworks.\n   - **Frameworks:**\n     - **Flask**: Lightweight and flexible, great for small to medium projects.\n     - **Django**: Full-featured, batteries-included framework for larger applications.\n     - **FastAPI**: Modern, high-performance framework for APIs (uses Python 3.7+).\n   - **Example (Flask):**\n     ```python\n     from flask import Flask\n     app = Flask(__name__)\n\n     @app.route('/')\n     def home():\n         return \"Hello, World!\"\n\n     if __name__ == '__main__':\n         app.run()\n     ```\n\n### 2. **Ruby (Ruby on Rails or Sinatra)**\n   - **Why?** Ruby is developer-friendly, and Rails is a powerful framework.\n   - **Frameworks:**\n     - **Ruby on Rails**: Convention over configuration, great for rapid development.\n     - **Sinatra**: Lightweight, like Flask.\n   - **Example (Sinatra):**\n     ```ruby\n     require 'sinatra'\n     get '/' do\n       \"Hello, World!\"\n     end\n     ```\n\n### 3. **Java (Spring Boot, Jakarta EE)**\n   - **Why?** Java is enterprise-grade and highly scalable.\n   - **Frameworks:**\n     - **Spring Boot**: Modern, easy-to-use framework for building web apps.\n     - **Jakarta EE**: Full-stack Java platform for web applications.\n   - **Example (Spring Boot):**\n     ```java\n     @RestController\n     public class HelloController {\n         @GetMapping(\"/\")\n         public String index() {\n             return \"Hello, World!\";\n         }\n     }\n     ```\n\n### 4. **Go (Gin, Echo, Fiber)**\n   - **Why?** Go is fast, simple, and great for high-performance servers.\n   - **Frameworks:**\n     - **Gin**: Lightweight and fast.\n     - **Echo**: Minimalist and extensible.\n     - **Fiber**: Inspired by Express.js (Node.js).\n   - **Example (Gin):**\n     ```go\n     package main\n     import \"github.com/gin-gonic/gin\"\n     func main() {\n         r := gin.Default()\n         r.GET(\"/\", func(c *gin.Context) {\n             c.String(200, \"Hello, World!\")\n         })\n         r.Run()\n     }\n     ```\n\n### 5. **C# (.NET Core)**\n   - **Why?** .NET Core is cross-platform and high-performance.\n   - **Example:**\n     ```csharp\n     var builder = WebApplication.Create();\n     builder.MapGet(\"/\", () => \"Hello, World!\");\n     builder.Run();\n     ```\n\n### 6. **Rust (Actix, Rocket, Axum)**\n   - **Why?** Rust is fast, safe, and great for performance-critical apps.\n   - **Frameworks:**\n     - **Actix**: Actor-based framework.\n     - **Rocket**: Easy-to-use, with a focus on developer experience.\n     - **Axum**: Modern, inspired by Elixir's Phoenix.\n   - **Example (Actix):**\n     ```rust\n     use actix_web::{get, App, HttpServer, Responder};\n\n     #[get(\"/\")]\n     async fn hello() -> impl Responder {\n         \"Hello, World!\"\n     }\n\n     #[actix_web::main]\n     async fn main() -> std::io::Result<()> {\n         HttpServer::new(|| App::new().service(hello))\n             .bind(\"127.0.0.1:8080\")?\n             .run()\n             .await\n     }\n     ```\n\n### 7. **Elixir/Phoenix**\n   - **Why?** Elixir runs on the Erlang VM, making it highly scalable and fault-tolerant.\n   - **Example (Phoenix):**\n     ```elixir\n     defmodule MyAppWeb.Router do\n       use MyAppWeb, :router\n       get \"/\", PageController, :index\n     end\n     ```\n\n### 8. **Clojure (Ring, Compojure)**\n   - **Why?** Clojure is a modern Lisp on the JVM, great for functional programming.\n   - **Example (Compojure):**\n     ```clojure\n     (defapp app\n       (routes\n         (GET \"/\" [] \"Hello, World!\")))\n     ```\n\n### 9. **Nim**\n   - **Why?** Nim is a fast, compiled language with Python-like syntax.\n   - **Example:**\n     ```nim\n     import nimbleweb\n     router \"/\": get = \"Hello, World!\"\n     ```\n\n### 10. **Bash (for simple cases)**\n   - **Why?** If you just need a quick HTTP server for testing.\n   - **Example:**\n     ```bash\n     python3 -m http.server 8000  # Even Python can be used for this!\n     ```\n\n### Recommendation:\n- For **quick prototyping**: Flask (Python) or Sinatra (Ruby).\n- For **enterprise-scale**: Spring Boot (Java) or .NET Core (C#).\n- For **high performance**: Go or Rust.\n- For **scalability**: Elixir/Phoenix or Java/Spring Boot.\n\nWould you like a deeper dive into any of these options?"
  }
]

Test 5

[
  {
    "role": "assistant",
    "tool_calls": null,
    "content": "I am Devstral-Small-2-24B-Instruct-2512, a Large Language Model (LLM) created by Mistral AI, a French startup headquartered in Paris. I power an AI assistant called Le Chat.\n\nToday's date is 2025-12-12."
  }
]

[bennny674]

I tested Devstral-Small-2-24B-Instruct-2512 with mistral-vibe on both llama-server locally and on the Mistral API, and it seems to have repetition problems on both (repeating file names in code files endlessly), so it might not be llama.cpp's fault?

[blakkd]

@aldehir it's mainly the test 2 that is often failing

[blakkd]

I tested Devstral-Small-2-24B-Instruct-2512 with mistral-vibe on both llama-server locally and on the Mistral API, and it seems to have repetition problems on both (repeating file names in code files endlessly), so it might not be llama.cpp's fault?

same

[aldehir]

@aldehir it's mainly the test 2 that is often failing

If you haven't already, please try it on master. Also, test-2 is testing both multimodal and tool calling capabilities. It's hard to tell if it's failing at tool calling or to understand the image--my guess is the latter.

[blakkd]

I think it understands the image, because the answer switches from French to English language when I tried with the English translated image