[Bug] Chat Completions Vision Format Crash

[A795da]

Describe the issue

Foundry Local's /v1/chat/completions endpoint severely mishandles standard OpenAI-style image_url inputs. Sending an image in the standard Chat Completions shape triggers a massive request-format bug. The engine attempts to allocate tens of gigabytes of memory, immediately crashing the entire engine.

-Impact: The standard /v1/chat/completions endpoint cannot be used for vision tasks.

(Working alternative for this: route the same image through the proprietary /v1/responses endpoint instead, using a distinct input_image field with raw base64 (no data: URI prefix). This works reliably but is undocumented as the required path for vision and isn't OpenAI-compatible, breaking drop-in compatibility with existing OpenAI client code.)

To reproduce

1. Construct a standard OpenAI Chat Completions request with an image supplied via the image_url content-part shape (as used by the official OpenAI SDK and every OpenAI-compatible client).

2. POST it to Foundry Local's /v1/chat/completions endpoint with the vision model loaded.

3. Observe the engine attempting a massive (tens-of-GB) memory allocation and crashing outright this takes down the whole service, not just the request.

4. Working alternative: route the same image through the proprietary /v1/responses endpoint instead, using a distinct input_image field with raw base64 (no data: URI prefix). This works reliably but is undocumented as the required path for vision and isn't OpenAI-compatible, breaking drop-in compatibility with existing OpenAI client code.

Urgency

Medium. Makes the standard/documented endpoint unusable for vision and forces consumers to use an undocumented, proprietary endpoint. But there is also a working alternative.

Platform and architecture

Windows X64

OS Version

Windows 11

Installation type

Released package/binary

Foundry Local version

0.10.2

API or surface area

REST API

Hardware acceleration/backend

CPU

Backend/runtime version

No response

[metrofun]

The interesting part here is that the same image works through Responses, so this seems narrower than general vision support.

I maintain swobuforge/swobu, which already translates Chat image_url inputs into Responses input_image. This report exposed one remaining compatibility detail for Foundry Local: our Responses encoder currently uses a data:...;base64,... URL for inline images, while this endpoint appears to require raw base64.

So this isn't an immediate Swobu workaround yet, but it looks like a small target-specific lowering rather than an application-level change. Once that representation difference is handled, ordinary Chat clients should be able to use Foundry Local's working Responses vision path transparently.

[A795da]

Thanks for digging into this; good catch narrowing it to a parsing issue specific to /v1/chat/completions rather than vision generally.

I also hit the exact same raw-base64-vs-data:-URI distinction while testing this: sending the image as a data: URI to /v1/responses fails with a generic input_ids size... exceeds max length error, while raw base64 (no prefix) gets past that and fails later with a different, model-internal error (image_grid_thw) - so the raw-base64 requirement is definitely real and specific to this endpoint.

This problem actually surfaced while building Yol Dostu, in the part: offline vision-RAG for plant/wildlife identification. If you are interested, here is the workaround I ended up shipping: vision-model-guard.js.

Sounds like a clean, isolated fix if you end up making that change, and a really interesting project either way. Good luck with the development of it. If it'd help, happy to share the exact request payloads I used to reproduce both variants.