fix(transport): derive the gRPC message ceiling from the output cap (#173 item 2)

docs/adrs/0024-sandbox-transport-remote-exec.md

@@ -217,6 +217,60 @@ transport.ts` hardcodes `streaming: true` — so a scripted `End{truncated: true
 runner's accounting and the session's frame. Equality of the two caps is still pinned, but
 it now pins memory parity rather than detectability, and the test says so.
 
+### 2026-09-08 — the gRPC message ceiling is derived from the output cap (#173 item 2)
+
+`MaxCallRecvMsgSize` was unconfigured, so both ends sat at gRPC's 4 MiB receive default.
+
+**The reported severity was wrong, and tracing it changed the fix.** The item claimed an
+oversized write "kills the whole stream rather than one exec, triggering a reconnect". It does
+not: `ExecRequest` is `{sandbox_id, exec}` and the relay forwards `{exec: {…}}` **without**
+`sandbox_id`, so the frame the worker receives is strictly smaller than the request that
+carried it in. With both limits equal, the relay's ingress always trips first and the Attach
+stream never sees the payload — one exec fails with `RESOURCE_EXHAUSTED`.
+
+**The real defect was a read/write asymmetry the item never mentioned.** Read is capped at
+`DEFAULT_OUTPUT_CAP` (8 MiB); a write costs 4/3 of the file in `Exec.stdin`, so ~3 MiB was the
+write ceiling. Files between ~3 and 8 MiB were **readable but not writable**, and Pi's Edit
+composes read with write. `KubectlTransport` has no such ceiling, so the same write succeeded
+on the pod path — the divergence class of the three revisions above.
+
+**Decided:** 16 MiB on both receive limits, **derived** as
+`base64EncodedLength(DEFAULT_OUTPUT_CAP) + EXEC_FRAMING_HEADROOM` (11,184,812 + 65,536), so
+write capacity ≥ read capacity by construction. Pinned equal across the language boundary by
+`message-size-coupling.test.ts`, which asserts the derivation too, so raising the output cap
+alone cannot restore the asymmetry. Send limits were already unlimited on both implementations
+and are untouched.
+
+The floor uses the **exact** encoded length `4·⌈n/3⌉` rather than the ×4/3 ratio. Caught in
+review: the ratio yields 11,184,810.67 against a real base64 length of 11,184,812, so a bound
+written against it sat 1.33 bytes _below_ the smallest payload it existed to admit and would
+have certified a ceiling of 11,184,811 that cannot carry an 8 MiB file. **A guard derived from
+an approximation can fail at exactly the boundary it was written for.**
+
+**Memory budget**, stated because moving a ceiling in this repo comes with one (`BufferCap`
+gives `2 × MaxConcurrent × BufferCap`; the 2026-08-28 revision pins the caps equal for memory
+parity). Worst-case ingress buffering rises 4×, as
+`concurrently decoding ExecRequests × MAX_EXEC_MESSAGE_BYTES`. It is a transient rather than a
+residency — the relay forwards `exec` and keeps no copy — and nothing in the contract bounds
+the count, so it sizes the relay rather than proving a bound. One such request is a quarter of
+the worker's 64 MiB budget, and only a write near the read cap reaches it.
+
+**Rejected:** raising only the relay, the hop that visibly rejects today. It would forward the
+payload and move the rejection onto the worker's Attach stream — killing every concurrent and
+queued exec and forcing a re-dial. That is the failure the item wrongly claimed already
+existed, so fixing the relay alone would have created it. Hence an equality between the two
+limits, not a floor, and `TestContractAcceptsAnOversizedWritePayload` fails if the worker is
+left at the default.
+
+**Rejected:** treating "needs a decision on both ends plus a documented max write size" as a
+prerequisite. That framing kept the item untouched for a month, but once the asymmetry is the
+frame the number follows from `DEFAULT_OUTPUT_CAP` and needs no cross-team agreement.
+
+**Consequence:** the worker's dial options moved into `session.DialOptions` so the contract
+tests dial exactly as `main.go` does. An option that production does not apply is
+indistinguishable from an absent one, and the coupling test asserts `main.go` reaches the wire
+through that function.
+
 ---
 
 _Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>_

docs/specs/2026-07-08-sandbox-transport-grpc-design.md

@@ -408,6 +408,43 @@ The frame _semantics_ are carried from the superseded design verbatim — only t
   seam a second truncation concept for bytes it does not return would buy nothing, and the Go
   worker logs the event instead.
 
+- **Message-size ceiling, and why it is derived rather than chosen.** gRPC defaults every
+  receive limit to 4 MiB on both implementations (grpc-js
+  `DEFAULT_MAX_RECEIVE_MESSAGE_LENGTH`, grpc-go `defaultClient/ServerMaxReceiveMessageSize`);
+  send limits default to unlimited. 4 MiB is **smaller than this contract's own write path
+  needs.** A write travels as base64 in `Exec.stdin`, inflating the file by 4/3, so the
+  largest _readable_ file — `DEFAULT_OUTPUT_CAP`, 8 MiB — becomes a ~10.7 MiB `Exec`. Left at
+  the default, every file between ~3 MiB and 8 MiB was **readable but not writable**, and Pi's
+  Edit composes read with write, so editing one succeeded at reading and then failed.
+  `KubectlTransport` pipes base64 through `kubectl exec` stdin with no such ceiling, making
+  this one more divergence decided by which backend was leased.
+
+  The ceiling is therefore **16 MiB**, clearing
+  `base64EncodedLength(DEFAULT_OUTPUT_CAP) + EXEC_FRAMING_HEADROOM` = 11,184,812 + 65,536, so
+  write capacity ≥ read capacity by construction. It is `MAX_EXEC_MESSAGE_BYTES`
+  (`transport.ts`) and `session.MaxRecvMsgBytes` (`dial.go`), pinned equal by
+  `message-size-coupling.test.ts` — which also asserts the derivation, so raising the output
+  cap alone cannot silently restore the asymmetry. The floor uses the **exact** encoded
+  length `4·⌈n/3⌉`, not the ×4/3 ratio: the ratio gives 11,184,810.67, which sits _below_ the
+  real payload and would admit a ceiling too small to carry an 8 MiB file.
+
+  **Memory budget.** Raising a receive limit raises worst-case ingress buffering with it, 4×
+  here, so it is stated as a formula the way `BufferCap` states its own:
+  `concurrently decoding ExecRequests × MAX_EXEC_MESSAGE_BYTES`. It is a **transient, not a
+  residency** — the relay forwards `exec` onto the Attach stream and keeps no copy — so the
+  peak is however many oversized requests are mid-decode, not however many execs are in
+  flight. Nothing in the contract bounds that count (the relay is single-replica and serves
+  every harness replica), so this is a ceiling to size the relay against rather than a proof.
+  For scale, one such request costs a quarter of the worker's own 64 MiB
+  (`2 × MaxConcurrent × BufferCap`) budget, and only a write near the read cap reaches it.
+
+  **Both ends must move together, and the worker's must not be lower.** The relay's ingress
+  limit rejects an oversized `ExecRequest` with `RESOURCE_EXHAUSTED`, and that failure is
+  contained to one exec. A relay that accepts more than the worker forwards a payload the
+  worker then refuses **on the Attach stream**, whose death takes every concurrent and queued
+  exec with it and forces a re-dial — strictly worse than both sitting at the default. That
+  ordering is why the limits are pinned as an equality rather than as a floor.
+
 - **Abort/end races.** A late `End` for an aborted `req_id` is dropped; an `Abort` for an
   already-ended `req_id` is a no-op.
 

packages/k8s-sandbox/src/index.ts

@@ -2,6 +2,9 @@ export { k8sSandboxExtension } from './extension.js';
 export { resolveConfig, type K8sSandboxConfig } from './config.js';
 export { buildKubectlArgs, KubectlTransport, type ExecInPod, type ExecResult } from './exec.js';
 export type { SandboxTransport } from './transport.js';
+// Exported for the relay, which must set its ingress limit to the SAME value the Go
+// worker uses — see MAX_EXEC_MESSAGE_BYTES on why the two cannot move independently.
+export { MAX_EXEC_MESSAGE_BYTES } from './transport.js';
 export { buildPersistentKubectlArgs, persistentExecInPod } from './persistent-exec.js';
 export {
   buildSelectorArgs,

packages/k8s-sandbox/src/transport.ts

@@ -75,6 +75,76 @@ export const DEFAULT_OUTPUT_CAP = 8 * 1024 * 1024; // 8 MiB
 /** Appended to returned stdout when the cap trips, so Pi sees the truncation. */
 export const OUTPUT_TRUNCATED_MARKER = '\n[output truncated]';
 
+/**
+ * base64EncodedLength is what a write costs on the wire, EXACTLY. `createPodWriteOps`
+ * sends file content as base64 in `Exec.stdin` (operations.ts), and base64 emits 4
+ * bytes per 3 consumed, padding the final group — so the length is `4·⌈n/3⌉`, which is
+ * up to 2 bytes more than the ×4/3 ratio suggests.
+ *
+ * The ratio is fine for prose and wrong for a bound: at DEFAULT_OUTPUT_CAP the two
+ * differ by 1.33 bytes, and 8 MiB × 4/3 = 11184810.67 sits BELOW the real 11184812. A
+ * guard written against the ratio therefore admits a ceiling that cannot actually carry
+ * the largest readable file — it fails to certify the one property it exists for, at
+ * exactly its own boundary. Use this instead of multiplying.
+ */
+export const base64EncodedLength = (bytes: number): number => 4 * Math.ceil(bytes / 3);
+
+/**
+ * EXEC_FRAMING_HEADROOM is what an `Exec` costs BEYOND its base64 stdin: the command
+ * string, the protobuf field tags and length prefixes, and `ExecRequest`'s `sandbox_id`.
+ *
+ * Measured, not guessed: a 4 MiB write arrived as 5592440 bytes against a base64 payload
+ * of 5592408 — a delta of **32 bytes**. 64 KiB is three orders of magnitude above that,
+ * which is deliberate: the command string is the only unbounded term (`base64 -d > <path>`
+ * today, a few dozen bytes) and nothing in the contract caps it.
+ */
+export const EXEC_FRAMING_HEADROOM = 64 * 1024;
+
+/**
+ * MAX_EXEC_MESSAGE_BYTES raises gRPC's 4 MiB default receive limit, which is smaller
+ * than this contract's own write path needs (#173 item 2).
+ *
+ * THE DERIVATION, because the number must not be arbitrary. The largest readable file
+ * is DEFAULT_OUTPUT_CAP, and writing it back costs
+ * `base64EncodedLength(DEFAULT_OUTPUT_CAP) + EXEC_FRAMING_HEADROOM` = 11184812 + 65536
+ * ≈ 10.7 MiB of `Exec`. At the 4 MiB default, every file between ~3 MiB and 8 MiB was
+ * READABLE BUT NOT WRITABLE — and Pi's Edit composes read with write, so editing one
+ * succeeded at reading and then failed. 16 MiB clears that floor with ~5 MiB to spare,
+ * making write capacity >= read capacity by construction. `KubectlTransport` pipes
+ * base64 through `kubectl exec` stdin with no such ceiling, so this also removes a
+ * divergence where the same write succeeded or failed depending on which backend was
+ * leased.
+ *
+ * The floor is asserted against the EXACT encoded length, not the ×4/3 ratio — see
+ * base64EncodedLength for why the ratio cannot certify this property at its own
+ * boundary.
+ *
+ * BOTH ENDS MUST MOVE TOGETHER, and the worker's limit must be at least this one.
+ * The relay's ingress is what rejects an oversized `ExecRequest` today, and that
+ * rejection is contained to a single exec. Raising the relay alone would forward the
+ * payload and move the rejection onto the worker's Attach stream, whose death takes
+ * every concurrent and queued exec with it. The Go side is
+ * `session.MaxRecvMsgBytes`, pinned to this value by
+ * test/message-size-coupling.test.ts — change one and change the other.
+ *
+ * MEMORY BUDGET — raising a receive limit raises worst-case ingress buffering with it,
+ * 4x here, so state it the way `BufferCap` states its own (runner.go):
+ *
+ *	concurrently decoding ExecRequests × MAX_EXEC_MESSAGE_BYTES
+ *
+ * It is a TRANSIENT, not a residency: the relay forwards `exec` to the Attach stream and
+ * keeps no copy (relay.ts routeExec), so the peak is however many oversized requests are
+ * mid-decode at once rather than however many execs are in flight. Nothing in the
+ * contract bounds that count — the relay is single-replica and serves every harness
+ * replica — so this is a ceiling to size the relay's limits against, not a proof. For
+ * scale: one such request costs a quarter of the worker's own 64 MiB
+ * (2 × MaxConcurrent × BufferCap) budget, and only a write near the read cap reaches it.
+ *
+ * Send limits need no change: grpc-js defaults max_send_message_length to -1 and
+ * grpc-go defaults MaxCallSendMsgSize to MaxInt32, both effectively unlimited.
+ */
+export const MAX_EXEC_MESSAGE_BYTES = 16 * 1024 * 1024; // 16 MiB
+
 /**
  * Ceiling on one exec when the caller names no `timeout` (spec §3; issue #182). Shared by
  * ALL THREE implementations, for the same reason DEFAULT_OUTPUT_CAP is: an exec that

packages/k8s-sandbox/test/message-size-coupling.test.ts

@@ -0,0 +1,94 @@
+import { readFileSync } from 'node:fs';
+import { fileURLToPath } from 'node:url';
+import { dirname, resolve } from 'node:path';
+import { describe, expect, it } from 'vitest';
+import {
+  base64EncodedLength,
+  DEFAULT_OUTPUT_CAP,
+  EXEC_FRAMING_HEADROOM,
+  MAX_EXEC_MESSAGE_BYTES,
+} from '../src/transport.js';
+
+const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../..');
+const DIAL_GO = resolve(REPO_ROOT, 'remote-worker/internal/session/dial.go');
+const MAIN_GO = resolve(REPO_ROOT, 'remote-worker/cmd/worker/main.go');
+
+/**
+ * Loud-throw reader, same shape as output-cap-coupling's: a renamed or reformatted Go
+ * constant must fail the extraction rather than limp through as NaN and pass the
+ * comparison by accident.
+ */
+const readMaxRecvMsgBytes = (): number => {
+  const src = readFileSync(DIAL_GO, 'utf8');
+  const match = /MaxRecvMsgBytes = (\d+) \* 1024 \* 1024/.exec(src);
+  if (!match) {
+    throw new Error(
+      'could not find `MaxRecvMsgBytes = N * 1024 * 1024` in dial.go — constant renamed or reformatted?',
+    );
+  }
+  return Number(match[1]) * 1024 * 1024;
+};
+
+describe('gRPC message-size limit is pinned across the language boundary', () => {
+  it("MAX_EXEC_MESSAGE_BYTES equals the Go worker's MaxRecvMsgBytes", () => {
+    // The two ends must move TOGETHER. The relay's ingress limit rejects an oversized
+    // ExecRequest and contains the failure to one exec; if the worker's limit were
+    // lower, the relay would forward a payload the worker then refuses — on the Attach
+    // stream, whose death takes every concurrent and queued exec with it and forces a
+    // re-dial. A worker limit BELOW the relay's is therefore strictly worse than both
+    // being at the 4 MiB default, which is why this is an equality and not a floor.
+    expect(MAX_EXEC_MESSAGE_BYTES).toBe(readMaxRecvMsgBytes());
+  });
+
+  it('leaves room to write back the largest file the read path can return', () => {
+    // The defect this closes: read is capped at DEFAULT_OUTPUT_CAP (8 MiB) while a write
+    // costs base64 of the file in Exec.stdin, so at gRPC's 4 MiB default a file between
+    // ~3 and 8 MiB was readable but not writable — and Pi's Edit composes read with
+    // write. Asserting the RELATIONSHIP rather than the literal means bumping the output
+    // cap alone cannot silently reintroduce the asymmetry.
+    //
+    // Two things this bound gets right that `DEFAULT_OUTPUT_CAP * 4/3` did not. The
+    // encoded length is EXACT: 4·⌈n/3⌉ is 11184812 where the ratio gives 11184810.67, so
+    // the ratio's bound sat 1.33 bytes BELOW the smallest payload it exists to admit and
+    // would have passed a ceiling of 11184811 that cannot carry an 8 MiB file. And the
+    // headroom term is asserted rather than merely claimed in prose: without it, a
+    // ceiling with zero room for the command string or protobuf framing satisfies the
+    // floor while contradicting the constant's own documented derivation.
+    expect(MAX_EXEC_MESSAGE_BYTES).toBeGreaterThanOrEqual(
+      base64EncodedLength(DEFAULT_OUTPUT_CAP) + EXEC_FRAMING_HEADROOM,
+    );
+  });
+
+  it('encodes lengths the way base64 actually does, padding included', () => {
+    // base64EncodedLength is load-bearing for the floor above, so pin it directly rather
+    // than trusting the formula: Node's own encoder is the oracle.
+    for (const n of [0, 1, 2, 3, 4, 3000, DEFAULT_OUTPUT_CAP]) {
+      expect(base64EncodedLength(n)).toBe(Buffer.alloc(n).toString('base64').length);
+    }
+  });
+
+  it('is actually applied by the worker, not merely defined', () => {
+    // A constant nothing dials with is indistinguishable from no constant. main.go must
+    // reach the wire through session.DialOptions, which is where the limit is attached
+    // and what the contract tests exercise.
+    //
+    // `[^)]*` for the address argument, NOT the literal `relayAddr`: pinning a local
+    // variable name would make renaming it report "the option is not applied" while the
+    // option is in fact applied — a false negative on the one assertion whose entire job
+    // is telling applied from merely defined. The requirement is that the DialOptions
+    // call sits inside NewClient's arguments; what the address is called is not this
+    // test's business. A bare /session\.DialOptions\(/ would be too loose in the other
+    // direction, since main.go's own comment names the function.
+    const mainGo = readFileSync(MAIN_GO, 'utf8');
+    if (!/grpc\.NewClient\([^)]*session\.DialOptions\(/.test(mainGo)) {
+      // Loud throw naming the cause, like readMaxRecvMsgBytes above: a bare toMatch
+      // failure here reads as "the limit is not applied" when the likelier truth is that
+      // the dial was restructured and this assertion needs re-aiming.
+      throw new Error(
+        'main.go does not dial through session.DialOptions — either the receive limit is ' +
+          'no longer applied in production (the defect this guards), or the dial was ' +
+          'restructured and this assertion needs updating. Check which before "fixing" it.',
+      );
+    }
+  });
+});

packages/sandbox-relay/src/main.ts

@@ -17,13 +17,20 @@ import {
   type ExecEvent,
   type AbortRequest,
   type AbortResponse,
+  MAX_EXEC_MESSAGE_BYTES,
 } from '@sh/k8s-sandbox';
 import { RedisRecordStore } from '@sh/harness';
 import { createRelay, type RelayDeps, type AttachStream } from './relay.js';
 
 export function buildServer(deps: RelayDeps): { server: Server } {
   const relay = createRelay(deps);
-  const server = new Server();
+  // Raise the ingress limit above gRPC's 4 MiB default. This is the hop that rejects an
+  // oversized write today: the harness's ExecRequest carries base64 stdin at 4/3 of the
+  // file, so a file the read path can return (DEFAULT_OUTPUT_CAP, 8 MiB) needs ~10.7 MiB
+  // here. MAX_EXEC_MESSAGE_BYTES is shared with the Go worker's session.MaxRecvMsgBytes
+  // and pinned equal to it — a relay that accepts more than the worker would forward a
+  // payload the worker refuses on its Attach stream, killing every exec on it (#173 item 2).
+  const server = new Server({ 'grpc.max_receive_message_length': MAX_EXEC_MESSAGE_BYTES });
 
   const workerImpl: SandboxWorkerServer = {
     // AttachStream types metadata.get() as returning string[]; grpc-js's real