Add MCP server so agent harnesses can play browser games

mcp/Dockerfile

@@ -0,0 +1,33 @@
+# Runwave MCP server: lets an agent harness play browser games.
+#
+# The Playwright base image already carries every system library Chromium needs
+# (libnss3, libnspr4, libgbm, the X client libs), which is the whole reason to
+# start from it rather than a plain node image.
+#
+# No recording means no gstreamer, no PulseAudio, and no Xvfb: Chromium runs
+# headless, so this image stays far smaller than the playtest runner's.
+FROM mcr.microsoft.com/playwright:v1.61.1-noble
+
+ENV NODE_ENV=production \
+    RUNWAVE_MCP_WORKSPACE=/var/lib/runwave-mcp
+
+WORKDIR /opt/runwave
+
+# Dependencies are copied first so a source edit does not invalidate the
+# install layer.
+COPY package.json package-lock.json ./
+RUN npm ci --omit=dev
+
+COPY runwave ./runwave
+COPY mcp ./mcp
+
+RUN mkdir -p "$RUNWAVE_MCP_WORKSPACE"
+
+# Chromium's sandbox needs privileges a default container does not grant. The
+# base image ships a non-root user with the right setup; use it rather than
+# disabling the sandbox.
+USER pwuser
+
+# stdio transport: the host speaks MCP over stdin/stdout, so nothing is exposed
+# on the network and no port is published.
+ENTRYPOINT ["node", "/opt/runwave/mcp/bin/runwave-mcp.js"]

mcp/README.md

@@ -0,0 +1,131 @@
+# Runwave MCP
+
+An MCP server that lets an agent harness such as Claude Code play a browser game
+directly: look at a frame, send a timed sequence of inputs, look at the next
+frame.
+
+This is the interactive counterpart to the `runwave` CLI. The CLI runs a VLM in a
+loop by itself and produces a recorded video for playtesting. Here the connected
+agent *is* the player, so the OpenRouter agent loop is not used and there is no
+recording — which is also why this needs none of runwave's gstreamer,
+PulseAudio, or Xvfb setup.
+
+## Requirements
+
+- Node 20+
+- Chromium's system libraries. On a bare Linux host:
+  ```sh
+  npx playwright install --with-deps chromium   # needs sudo for the libs
+  ```
+  In Docker, use `mcp/Dockerfile`, which starts from the Playwright base image
+  and already has them.
+
+No X server, no audio, no display. Chromium runs headless.
+
+## Run
+
+```sh
+node mcp/bin/runwave-mcp.js
+```
+
+Artifacts (screenshots, per-step JSON) are written under
+`RUNWAVE_MCP_WORKSPACE`, defaulting to a directory in the system temp dir. Set
+it explicitly if you want to keep them:
+
+```sh
+RUNWAVE_MCP_WORKSPACE=./artifacts node mcp/bin/runwave-mcp.js
+```
+
+Register it with Claude Code:
+
+```sh
+claude mcp add runwave -- node /absolute/path/to/mcp/bin/runwave-mcp.js
+```
+
+Or in Docker:
+
+```sh
+docker build -f mcp/Dockerfile -t runwave-mcp .
+claude mcp add runwave -- docker run --rm -i runwave-mcp
+```
+
+## Tools
+
+| Tool | Purpose |
+| --- | --- |
+| `launch_game` | Start a session from a `url`, or a `game_dir` containing `start.sh` plus a `port`. Returns the first frame. |
+| `observe` | Fresh frame and state, no input sent. |
+| `act` | Send a timed input sequence, return the resulting frame. |
+| `zoom` | Full-resolution crop of a region, to read small UI without a full frame. |
+| `capture` | Save a clean, full-resolution, un-annotated screenshot. The deliverable. |
+| `reset_game` | Reload at the launch URL. |
+| `journal` | Text log of what has been tried this session. |
+| `list_sessions` | Running sessions. |
+| `end_game` | Close the browser and stop the game process. |
+
+## How `act` works
+
+An action sequence is timed, not a single keypress. Offsets are milliseconds
+from the start of the sequence and actions may overlap, so one call can express
+"hold right for 900ms and jump at 150ms":
+
+```json
+{
+  "session_id": "game-...",
+  "actions": [
+    { "type": "key", "start": 0, "end": 900, "key": "ArrowRight" },
+    { "type": "key", "start": 150, "end": 230, "key": "Space" }
+  ]
+}
+```
+
+This matters because an agent turn costs seconds. Committing to a move beats
+sending one tap per turn.
+
+Action types: `key`, `click`, `multi_click`, `drag`, `cursor_move`, `view_move`.
+Pointer actions take either `x`/`y` in viewport pixels or an
+`overlay_row`/`overlay_col` grid cell. The full schema is enforced on the tool
+input, so a malformed sequence is rejected before any input is sent.
+
+## Notes on the design
+
+**Frames are downscaled by default.** A 1280x720 PNG is roughly 1200 tokens; at
+half scale it is about 300. Over a long navigation that difference dominates
+everything else. Pass `full_res: true` when detail genuinely matters, or use
+`zoom` on the region you care about — usually cheaper than a full-resolution
+frame.
+
+**One frame per turn.** Interval captures are off. Pass `captures` with explicit
+offsets to see a trajectory within a sequence.
+
+**`act` reports whether the frame changed.** A byte-identical frame almost always
+means the input never reached the game, rather than the game ignoring it. The
+tool says so instead of leaving the agent to guess.
+
+**The grid overlay is off by default.** When enabled it enlarges the PNG with a
+label margin on every side, so pixels read off the image no longer match pixels
+sent back as `x`/`y`. The offset is reported in the response when the grid is on,
+but exact coordinates or grid cells are the better targets.
+
+**Grid cells resolve to the cell centre.** Runwave's playtest path deliberately
+scatters clicks inside a cell to vary footage; that is wrong when aiming at a
+specific target, and it makes a run unreproducible. Playtest behaviour is
+unchanged — this server opts into `markGridSampleMode: 'center'`.
+
+**Calls are serialized per session.** There is one Playwright page and one shared
+step counter, so concurrent calls would interleave keypresses. Subagents may
+share a `session_id` safely.
+
+**Sessions close on shutdown.** Chromium and any spawned game process are
+detached children; `SIGINT`, `SIGTERM`, `SIGHUP`, and an uncaught exception all
+close them. Sessions also close after 30 minutes idle, so a forgotten
+`end_game` does not leak a browser.
+
+## Tests
+
+```sh
+npm run test:mcp
+```
+
+The integration test drives a real headless Chromium against a fixture game and
+skips itself when Chromium cannot launch.

mcp/bin/runwave-mcp.js

@@ -0,0 +1,41 @@
+#!/usr/bin/env node
+'use strict';
+
+const { StdioServerTransport } = require('@modelcontextprotocol/sdk/server/stdio.js');
+const { createServer } = require('../src/server');
+
+async function main() {
+  const { server, registry, workspace } = createServer();
+  // stdout is the MCP transport, so diagnostics go to stderr only.
+  process.stderr.write(`runwave-mcp workspace: ${workspace}\n`);
+
+  // Chromium and any game process are detached children. Without this a host
+  // shutting the server down orphans the whole tree, leaving browsers running.
+  let shuttingDown = false;
+  const shutdown = async (signal) => {
+    if (shuttingDown) return;
+    shuttingDown = true;
+    process.stderr.write(`runwave-mcp shutting down on ${signal}\n`);
+    await registry.closeAll();
+    await server.close().catch(() => {});
+    process.exit(0);
+  };
+
+  for (const signal of ['SIGINT', 'SIGTERM', 'SIGHUP']) {
+    process.on(signal, () => { shutdown(signal); });
+  }
+  process.on('uncaughtException', async (error) => {
+    process.stderr.write(`runwave-mcp uncaught: ${error.stack || error.message}\n`);
+    await shutdown('uncaughtException');
+  });
+  process.on('unhandledRejection', (reason) => {
+    process.stderr.write(`runwave-mcp unhandled rejection: ${reason}\n`);
+  });
+
+  await server.connect(new StdioServerTransport());
+}
+
+main().catch((error) => {
+  process.stderr.write(`${error.stack || error.message}\n`);
+  process.exit(1);
+});

mcp/src/config.js

@@ -0,0 +1,56 @@
+'use strict';
+
+const path = require('path');
+
+const DEFAULT_VIEWPORT = { width: 1280, height: 720 };
+
+// Runwave scatters clicks within a cell to vary playtest footage. An agent
+// aiming at a target needs the opposite: the same cell must mean the same pixel
+// so a saved action trace replays identically.
+const CELL_SAMPLE_MODE = 'center';
+
+function normalizeViewport(viewport) {
+  const width = Number(viewport && viewport.width);
+  const height = Number(viewport && viewport.height);
+  return {
+    width: Number.isFinite(width) && width > 0 ? Math.round(width) : DEFAULT_VIEWPORT.width,
+    height: Number.isFinite(height) && height > 0 ? Math.round(height) : DEFAULT_VIEWPORT.height,
+  };
+}
+
+// The daemon hands createSession its raw CLI input, which is why grid-cell
+// actions fail there when no viewport was passed. Building the config
+// explicitly closes that gap: viewport is always present and always numeric.
+function buildSessionConfig(options = {}) {
+  const viewport = normalizeViewport(options.viewport);
+  return {
+    kind: 'web',
+    ...(options.url ? { url: options.url } : {}),
+    ...(options.gameDir ? { gameDir: path.resolve(options.gameDir) } : {}),
+    ...(options.port ? { port: Number(options.port) } : {}),
+    viewport,
+    deviceScaleFactor: 1,
+    // No recording: no gstreamer, no PulseAudio, no headed Chromium.
+    record: false,
+    headless: true,
+    // Overlay is opt-in per call. It enlarges the PNG by a margin per side,
+    // which desyncs image coordinates from input coordinates.
+    gridScreenshots: false,
+    fullPageScreenshots: false,
+    // The agent asks for frames explicitly; interval captures would spend
+    // context on frames nobody requested.
+    autoCaptures: false,
+    markGridSampleMode: CELL_SAMPLE_MODE,
+    ...(options.markGridRows ? { markGridRows: Number(options.markGridRows) } : {}),
+    ...(options.markGridCols ? { markGridCols: Number(options.markGridCols) } : {}),
+    ...(options.stateExpression ? { stateExpression: String(options.stateExpression) } : {}),
+    waitAfterLoad: Number(options.waitAfterLoad ?? 700),
+  };
+}
+
+module.exports = {
+  CELL_SAMPLE_MODE,
+  DEFAULT_VIEWPORT,
+  buildSessionConfig,
+  normalizeViewport,
+};

mcp/src/diff.js

@@ -0,0 +1,29 @@
+'use strict';
+
+const crypto = require('crypto');
+const fs = require('fs');
+
+function hashFile(file) {
+  try {
+    return crypto.createHash('sha256').update(fs.readFileSync(file)).digest('hex');
+  } catch {
+    return null;
+  }
+}
+
+// "Did anything happen?" is the single most useful signal after an input, and
+// the cheapest: no second screenshot, no pixel walk. A false here usually means
+// the input never reached the game, which is otherwise easy to misread as the
+// game ignoring the move.
+function changedSince(previousFile, nextFile) {
+  if (!previousFile || !nextFile) return null;
+  const before = hashFile(previousFile);
+  const after = hashFile(nextFile);
+  if (!before || !after) return null;
+  return before !== after;
+}
+
+module.exports = {
+  changedSince,
+  hashFile,
+};

mcp/src/frame.js

@@ -0,0 +1,54 @@
+'use strict';
+
+const { drawMarkGridOnScreenshot } = require('../../runwave/controller/src/grid-overlay');
+const { readPng } = require('./image');
+const { frameBlocks, gridNote, stateText, textBlock } = require('./result');
+
+// The overlay writes the PNG larger than the capture by a fixed margin per
+// side. gridLabelStyle is private, so the margin is recovered from the file
+// itself rather than reimplementing the label metrics.
+function overlayMargin(file, viewport) {
+  try {
+    const png = readPng(file);
+    const margin = Math.round((png.width - Number(viewport.width)) / 2);
+    return margin > 0 ? margin : 0;
+  } catch {
+    return 0;
+  }
+}
+
+// Draws the overlay onto an existing capture and reports the margin it added.
+function applyGrid(session, file) {
+  drawMarkGridOnScreenshot(file, session.config);
+  return overlayMargin(file, session.config.viewport);
+}
+
+// Screenshots are taken clean. The grid is drawn only when a caller asks for
+// it, so the deliverable frame is never annotated.
+async function captureFrame(session, { name, grid = false }) {
+  const dir = session.actionDir(name);
+  const file = await session.browser.screenshot(dir, name);
+  if (!grid) return { file, margin: 0 };
+  return { file, margin: applyGrid(session, file) };
+}
+
+// Assembles the per-turn payload: frame, trimmed state, and any coordinate
+// caveat the model needs in order to aim correctly.
+function frameResult({ file, margin, state, scale, fullRes, region, label, extra = [] }) {
+  const { blocks, image } = frameBlocks(file, { scale, fullRes, region, label });
+  const notes = [stateText(state)];
+  const caveat = gridNote(image, margin);
+  if (caveat) notes.push(caveat);
+  for (const note of extra) if (note) notes.push(note);
+  return {
+    content: [...blocks, textBlock(notes.join('\n'))],
+    image,
+  };
+}
+
+module.exports = {
+  applyGrid,
+  captureFrame,
+  frameResult,
+  overlayMargin,
+};