Add MCP server so agent harnesses can play browser games - #30
Conversation
Runwave's CLI runs its own VLM loop and produces a playtest video. This adds an MCP surface where the connected agent is the player instead: observe a frame, send a timed input sequence, observe the next frame. The server embeds the controller rather than driving the daemon. That is possible because BrowserSession and runStep already take their dependencies as arguments, and it avoids the daemon's session files, unauthenticated loopback port, process.exit on stop, and cwd captured at module load. The normalize/timeline/execute chain, session factory, state reader, and grid overlay are reused unchanged. Recording is out of scope for interactive play, so Chromium runs headless and the gstreamer, PulseAudio, and Xvfb requirements are gone. Adaptations for an agent player rather than a playtest bot: - Frames return as MCP image blocks, downscaled by default. A 1280x720 PNG is ~1200 tokens against ~300 at half scale, which dominates cost over a long navigation. zoom covers detail without a full frame. - act reports whether the frame changed. A byte-identical frame means the input did not reach the game, not that the game ignored it. - Grid overlay off by default: it enlarges the PNG with a label margin, desyncing image coordinates from input coordinates. The offset is reported when it is on. - Grid cells resolve to the cell centre via a new opt-in markGridSampleMode. Playtest scatter varies footage but misses precise targets and makes runs unreproducible; the default is unchanged. - Per-turn state drops the WebGL probe and keeps the largest canvas as the game area. - Calls serialize per session, since one page and one step counter mean concurrent calls would interleave keypresses. - Signal handlers and a 30 minute idle timeout close detached Chromium and game processes. Tests: 12 unit tests, plus an integration test that drives real headless Chromium against a fixture game and skips when Chromium cannot launch. The integration path is unverified here: this machine is missing libnss3/libnspr4, so it has only been exercised via unit tests so far. Co-Authored-By: Claude <noreply@anthropic.com>
TerminalBench Bot CommandsRun tasks: Check: Debug: Full Check: Grok Trace Run: Trace Run: Re-verify (re-score existing agent attempts against updated tests): Harbor format checker: Offline-search reviewer: Online-search reviewer: Sapphire format checker: Multiturn format checker: For detailed parameter descriptions, run Job Management: Review: Remove default flags: Use Aliases: Get help: |
1 participant
Adds an MCP surface where the connected agent is the player. The existing CLI runs its own VLM loop and produces a playtest video; here Claude Code (or any MCP harness) observes a frame, sends a timed input sequence, and observes the next one. Aimed at goal-directed play — reach a location, capture it — rather than footage generation.
Approach
The server embeds the controller instead of driving the daemon over HTTP. That works because
BrowserSessiontakes(config, paths, profiler)as constructor args and only readsthis.paths.runDir, andrunSteptakes everything by parameter. Bypassingdaemon.js/api.jsdrops the session files, the unauthenticated loopback port, theprocess.exitinstop, and the cwd-captured-at-require-time inpaths.jsin one move.Reused unchanged:
session-factory,browser-session, the normalize → timeline → execute chain,state-reader,grid-overlay,output-writer,protocol/.Recording is out of scope for interactive play, so Chromium runs headless and the gstreamer, PulseAudio, and Xvfb requirements are gone. Requirements are now Node 20+ and Chromium's system libs.
mcp/Dockerfilestarts from the Playwright base image, which already has them.Tools
launch_game,observe,act,zoom,capture,reset_game,journal,list_sessions,end_game.actcarries the action schema as real JSON Schema derived fromprotocol/src/action.js, soMAX_ACTION_SPAN_MSand the field allowlists can't drift from what the executor enforces, and malformed sequences are rejected before any input is sent.Adaptations for an agent player
zoomgives full-resolution detail on a region without paying for a full frame.actreports whether the frame changed, from a file hash. A byte-identical frame almost always means the input never reached the game rather than the game ignoring it, which is otherwise easy to misdiagnose.x/y. When enabled, the offset is stated in the response.markGridSampleMode. Playtest scatter varies footage but misses precise targets and makes runs unreproducible. Verified: 1 distinct point across 300 runs with the flag, 287 without. The playtest default is unchanged —agent/'s separate normalizer copy also still scatters.game_area. The probe is useful in a playtest report and pure overhead on every turn of a navigation.session_idsafely.SIGINT/SIGTERM/SIGHUP/uncaughtExceptionclose every browser and game process group; a 30-minute idle timeout covers a forgottenend_game. Chromium and game processes are detached children, so without this a host shutdown orphans the tree.Changes to existing code
Deliberately minimal and additive: 4 lines plus one new
gridSampleModefunction inprotocol/src/mark-grid.jsand its call site incontroller/src/action-normalizer.js. Both default to current behaviour.Testing
121/123 pass, 0 fail (
npm test). 12 new unit tests cover config defaults, cell determinism, scatter preservation, image resize/crop, state compaction, and the action schema.The integration test has not run. It drives real headless Chromium against a fixture game — held-key movement, a grid-cell click onto an 80x80 target, frame-change detection — but the machine I built this on is missing
libnss3/libnspr4, so it skips itself with a message rather than passing vacuously. Everything above the browser boundary is unit-tested; the play loop end to end is not verified yet.npm run test:mcpon a host with the libs (or in the provided Dockerfile) is the real go/no-go, and worth doing before merge.🤖 Generated with Claude Code