Multimodal tool results lose their image parts on the client round trip

[andrewmcgivery]

TanStack AI version

v0.52.0 (also present in v0.42.0 in a different form — see Notes)

Framework/Library version

React v18.3.1, Node v22.22.3, TypeScript v5.9.3

Describe the bug and the steps to reproduce it

A server tool that returns ContentPart[] containing an image works on the turn it is produced, but the image is silently downgraded to text as soon as the conversation round trips through the client. The base64 payload is then billed as text, which blows the context window:

400 invalid_request_error: prompt is too long: 1302926 tokens > 1000000 maximum

(Possible) Root cause

StreamProcessor.handleToolCallResultEvent parses the tool output correctly but stores the raw JSON string on the tool-result part instead of the parsed value:

packages/ai/src/activities/chat/stream/processor.ts — handleToolCallResultEvent

let output;
try { output = JSON.parse(chunk.content); } catch { output = chunk.content; }

this.messages = updateToolCallWithOutput(this.messages, chunk.toolCallId, output, ...);
//                                                    ^ parsed array, correct
this.messages = updateToolResultPart(this.messages, messageId, chunk.toolCallId, chunk.content, resultState, ...);
//                                                                              ^^^^^^^^^^^^^ raw string

That breaks the metadata side-channel added in 0.52 to carry multimodal tool results:

1. uiMessagesToWire calls rebuiltToolMetadata(part.metadata, part.createdAt, part.id, part.content, true).
2. rebuiltToolMetadata only preserves the payload when it is an array — ...Array.isArray(content) && { content }. It receives a string, so metadata.tanstack.toolResult.content is omitted.
3. restoreToolResultOwnership on the server therefore has nothing to restore, and the tool result falls back to the stringified content.
4. The Anthropic adapter receives a string and emits a tool_result whose content is plain text rather than [text, image] blocks.

The sibling code path is already correct — handleToolResult passes normalizeToolResult(output). Only the AG-UI TOOL_CALL_RESULT path (server-executed tools over SSE) is affected.

Steps to reproduce

1. Define a server tool whose handler returns ContentPart[] — a text part plus an image part with a { type: 'data', value: <base64>, mimeType: 'image/png' } source.
2. Have the model call it, then send a second user message so the history round trips through the client.
3. Inspect the outbound provider payload: the tool_result content is a string, not [text, image].

Your Minimal, Reproducible Example

Standalone Node script, no framework required. npm i @tanstack/ai@0.52.0 @tanstack/ai-anthropic@0.18.3, then run with --input-type=module:

import {
 StreamProcessor,
 uiMessagesToWire,
 chatParamsFromRequestBody,
 convertMessagesToModelMessages,
} from '@tanstack/ai';
import { createAnthropicChat } from '@tanstack/ai-anthropic';

const TC = 'toolu_1';
const toolOutput = [
 { type: 'text', content: 'Layout rule 1.' },
 { type: 'image', source: { type: 'data', value: 'iVBORw0KGgoAAAANSUhEUg==', mimeType: 'image/png' } },
];

const p = new StreamProcessor();
[
 { type: 'RUN_STARTED', runId: 'r1', threadId: 't1' },
 { type: 'TEXT_MESSAGE_START', messageId: 'u1', role: 'user' },
 { type: 'TEXT_MESSAGE_CONTENT', messageId: 'u1', delta: 'hi' },
 { type: 'TEXT_MESSAGE_END', messageId: 'u1' },
 { type: 'TOOL_CALL_START', toolCallId: TC, toolCallName: 'getLayoutRules', parentMessageId: 'a1' },
 { type: 'TOOL_CALL_ARGS', toolCallId: TC, delta: '', args: '{}' },
 { type: 'TOOL_CALL_END', toolCallId: TC, metadata: { tanstack: { toolCallName: 'getLayoutRules', input: {} } } },
 { type: 'TOOL_CALL_RESULT', toolCallId: TC, messageId: 'a1', content: JSON.stringify(toolOutput) },
].forEach((c) => {
 try {
  p.processChunk(c);
 } catch {}
});

// client -> wire -> HTTP (JSON) -> server
const wire = uiMessagesToWire(p.getMessages ? p.getMessages() : p.messages);
const body = JSON.parse(
 JSON.stringify({
  threadId: 't1',
  runId: 'r1',
  state: {},
  messages: wire,
  tools: [],
  context: [],
  forwardedProps: {},
 }),
);
const toolWire = body.messages.find((m) => m.role === 'tool');
console.log('[wire]   metadata carries array :', Array.isArray(toolWire?.metadata?.tanstack?.toolResult?.content));

const params = await chatParamsFromRequestBody(body);
const model = convertMessagesToModelMessages(params.messages);
console.log('[model]  content is array       :', Array.isArray(model.find((m) => m.role === 'tool')?.content));

const adapter = createAnthropicChat('claude-opus-4-8', 'sk-fake');
const block = adapter
 .formatMessages(model)
 .flatMap((m) => (Array.isArray(m.content) ? m.content : []))
 .find((b) => b.type === 'tool_result');
console.log(
 '[anthropic] tool_result content :',
 Array.isArray(block?.content) ? block.content.map((b) => b.type).join(',') : typeof block?.content,
);

Actual output

[wire]   metadata carries array : false
[model]  content is array       : false
[anthropic] tool_result content : string     <- base64 billed as text

Expected output

[wire]   metadata carries array : true
[model]  content is array       : true
[anthropic] tool_result content : text,image

Suggested fix

One line in handleToolCallResultEvent — pass the parsed output through the existing normalizer instead of the raw string. normalizeToolResult is already imported in this module, and it returns strings unchanged, so non-multimodal results are unaffected:

- this.messages = updateToolResultPart(this.messages, messageId, chunk.toolCallId, chunk.content, resultState, ...);
+ this.messages = updateToolResultPart(this.messages, messageId, chunk.toolCallId, normalizeToolResult(output), resultState, ...);

Applying this patch to node_modules makes the reproduction above print the expected output.

Notes

• On v0.42.0 the same symptom occurs for a different reason: uiMessagesToWire stringifies tool-result content with no metadata channel and no server-side restore path, so multimodal tool results cannot round trip at all. v0.52.0 added the channel; this bug prevents it from ever being populated.
• The failure is silent. There is no type error and no warning — the first symptom is a provider-side token-limit or content error, which makes it hard to attribute.

Do you intend to try to help solve this bug with your own PR?

No, because I do not have time to dig into it

Terms & Code of Conduct

• I agree to follow this project's Code of Conduct
• I understand that if my bug cannot be reliable reproduced in a debuggable environment, it will probably not be fixed and this issue may even be closed.