fix(agent-harness): accept MCP image/audio/resource tool results

[EnesYilmazcode]

Why

A successful MCP tool call that returns a non-text content block reaches the model as the literal string Error: [.

ToolCallOutputContentItemSchema accepted {type:'text'} and {type:'image', image_url:{url}}. That second shape comes from the OpenAI chat format; MCP does not use it. The sandbox returns MCP blocks unchanged (agent_environment/main.py:177-183 returns result.content), so an image block arrives as {type:'image', data, mimeType} and fails the parse at agent-eval.ts:259. The catch block does .split('\n')[0] on the message, and a ZodError's message is pretty-printed JSON starting with [\n, so the first line is a bare [. The model is told a working tool call failed and is given nothing to recover from.

It is reachable on the current pins:

• desktop-commander@0.2.7 read_file and read_multiple_files return text plus image for any image path (dist/handlers/filesystem-handlers.js:41-53, :89-101). Nothing suppresses them. They are enabled in 57 of the 500 public tasks, and PNGs ship in the sandbox at /data/repos/storyteller/example/*.png and /data/repos/slackr/man/figures/logo.png via the pinned submodules.
• filesystem@2026.7.10 read_media_file returns image, audio and resource (dist/index.js:246-252), enabled in 39 tasks.

Narrower than it first looks: sandbox-client.ts:257-263 requires a non-empty text block before passing a result through, so an image-only result becomes the string success before the schema ever sees it. Only a mixed result with non-empty text reaches the parse.

Changes

• types.ts widens ToolCallOutputContentItemSchema to the MCP content-block union (text, image, audio, resource_link, resource) for tool responses.
• schema.ts imports that union for CallToolResponseSchema, and keeps a separate narrow InboundToolContentItemSchema for ToolCallOutputMessageSchema. Those two shared one symbol before. Widening both would have loosened the POST /v2/mcp_eval/run_agent request contract so a caller could submit raw base64 that flows to the provider unvalidated, so the inbound side stays exactly as it was.
• errors.ts adds formatToolCallError. Zod 3 collapses a failed union into a single invalid_union issue whose message is literally "Invalid input", with the real detail in issue.unionErrors, so this recurses into those, prefers the branch whose type literal matched, and caps the result at 300 chars.
• helpers/tool-content.ts adds describeNonTextContent, applied before capToolContent. Tool results go verbatim into a Chat Completions payload, which carries text parts only, so widening the schema on its own would replace Error: [ with a provider 400 that ends the agent loop. Non-text blocks become a descriptor like [image content omitted: image/png]. Text items are returned by reference, so text-only results are byte-for-byte unchanged.
• tests/tool-output-schema.test.ts, a typecheck script with tsconfig.test.json, and .github/workflows/agent-harness-tests.yml. The harness has no CI today, so the tests would otherwise never run. tsconfig.json only includes src/**/* and tsx strips types without checking, so the build alone does not typecheck tests. Drop the workflow if you would rather wire up CI yourself.

The union is hand-written rather than imported from @modelcontextprotocol/sdk, which would be the obvious thing to do. It is not possible here: the SDK's ContentBlockSchema is a Zod 4 schema and this harness pins zod@^3.22.4 (resolved 3.25.76). CB instanceof z.ZodType is false and z.union([ContentBlockSchema, ...]) throws option._parseSync is not a function.

Test plan

cd services/agent-harness && npm ci && npm run build && npm test   # 10 passed
npm run typecheck                                                  # clean
cd services/agent-environment && python -m pytest tests/ -v        # 2 passed, unaffected

What the model actually receives for the original failing payload:

	output
before	Error: [
after	Error: content.1: unrecognized type "nonsense", expected one of "text", "image", "audio", "resource_link", "resource"

Forty bad blocks go from a 1068-char error string to 313 chars ending in … (40 issues).

• Reverting errors.ts and schema.ts while keeping the tests fails exactly the four new ones, including actual: 'content.1: Invalid input'.
• Reverting only the schema widening fails the content-block tests with Invalid literal value, expected "text".
• Dropping any single union variant, or any line of describeNonTextContent, fails a test.
• Text passthrough is asserted by reference identity (assert.equal(described[0], textItem)), so a refactor that copies text items fails the test.
• The typecheck gap is proven: adding a deliberate TS2322 to the test file leaves npm run build green and is caught only by npm run typecheck.

Not run: the Docker sandbox and a live LLM call, so there is no end-to-end trace, and the provider-400 reasoning behind describeNonTextContent follows from the Chat Completions contract rather than a measurement. The three-line wiring in agent-eval.ts is covered by typecheck and reading only, since exercising it needs an MCP client, a LiteLLM endpoint and a live sandbox.

Notes

prunedTools at agent-eval.ts:333 force-sets returnImage = false for met-museum_get-museum-object, commented prevent images from being returned. That workaround looks like it exists because of this bug. I left it in place, since removing it would re-enable images on 85 tasks and change token cost, so that is a maintainer decision.

Separate and not fixed here: sandbox-client.ts:257-263 replaces an image-only result with the string success before the schema sees it, so that result is lost silently rather than erroring. The comment there suggests it is deliberate.

Image bytes are described rather than forwarded, since there is no vision path today and Chat Completions does not carry images in a tool message.

🤖 Generated with Claude Code

Greptile Summary

The PR expands MCP tool-result validation and converts non-text results into provider-compatible text descriptions while improving validation errors and adding harness CI coverage.

• Adds MCP image, audio, resource-link, and embedded-resource schemas.
• Converts unsupported non-text blocks before constructing tool messages.
• Adds focused schema, formatting, and conversion tests.
• Adds typechecking and test commands for the agent harness.

Confidence Score: 3/5

The PR is not yet safe to merge because its new workflow still executes mutable GitHub Action references on pull requests and pushes.

The tool-result handling changes do not leave a newly eligible blocking failure, but the previously reported CI supply-chain issue remains: both action dependencies are still selected through movable v4 tags rather than immutable commit SHAs.

Files Needing Attention: .github/workflows/agent-harness-tests.yml

Important Files Changed

Filename	Overview
services/agent-harness/src/mcp-agent/types.ts	Expands tool-result validation to cover MCP text, image, audio, resource-link, and resource blocks while retaining the OpenAI image shape.
services/agent-harness/src/mcp-agent/helpers/tool-content.ts	Converts MCP non-text blocks into text descriptors and preserves embedded textual resources.
services/agent-harness/src/mcp-agent/errors.ts	Produces bounded, actionable summaries for nested Zod union validation failures.
services/agent-harness/src/mcp-agent/agent-evals/agent-eval.ts	Normalizes validated tool content before output capping and uses the new error formatter.
services/agent-harness/src/mcp-agent/schema.ts	Keeps inbound conversation history restricted to provider-compatible content while importing the widened response schema.
services/agent-harness/tests/tool-output-schema.test.ts	Covers MCP content variants, malformed unions, bounded errors, inbound-history restrictions, and non-text conversion.
.github/workflows/agent-harness-tests.yml	Adds path-filtered build, typecheck, and test coverage for the agent harness.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[MCP tool result] --> B[Validate content blocks]
    B --> C[Describe non-text content]
    C --> D[Apply output cap]
    D --> E[Send tool message to model]
    B -->|Validation failure| F[Format concise error]
    F --> E

Loading

_{Reviews (2): Last reviewed commit: "fix(agent-harness): keep inbound tool co..." | Re-trigger Greptile}

[greptile-apps]

Mutable CI action references

The new workflow executes actions/checkout and actions/setup-node through mutable v4 tags on qualifying pull requests and pushes to main. Pinning these actions to full commit SHAs prevents upstream tag movement from changing executable CI code without a reviewed repository change.

How this was verified: Both newly added uses entries reference major-version tags rather than immutable commit SHAs.

Prompt To Fix With AI

This is a comment left during a code review.
Path: .github/workflows/agent-harness-tests.yml
Line: 22-23

Comment:
**Mutable CI action references**

The new workflow executes `actions/checkout` and `actions/setup-node` through mutable `v4` tags on qualifying pull requests and pushes to `main`. Pinning these actions to full commit SHAs prevents upstream tag movement from changing executable CI code without a reviewed repository change.

**How this was verified:** Both newly added `uses` entries reference major-version tags rather than immutable commit SHAs.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[EnesYilmazcode]

Thanks. I left these as major-version tags deliberately, to match the workflow they sit beside: agent-environment-tests.yml uses actions/checkout@v4 and actions/setup-python@v5. Pinning only the new file would make it the one inconsistent workflow in the repo.

Happy to go either way. If you would rather pin to SHAs, I think it is worth doing across both workflows in one change rather than just this one, and I am glad to do that here or in a separate PR, whichever the maintainers prefer.