OOM on lockfile-regen PRs: Phase 6+6.7 harness fan-out has no shared concurrency budget

[AbirAbbas]

Incident

2026-08-05 ~17:34 UTC, Railway deployment 749d6546 (commit 8f1abeb). While reviewing Agent-Field/agentfield#883 — a Dependabot security bump whose regenerated lockfiles produce diffs on the order of 60k lines — the node's memory climbed until the container hit the Railway plan memory ceiling (~24 GB) and went down. Service metrics (10-min sampling) show ≥8.8 GB on the way up; the final spike outran the sampling window.

Before the container died, every in-flight opencode child hit thread-spawn failure and aborted:

opencode finished: returncode=-6 stdout=0 chars elapsed=1s
thread '<unnamed>' panicked at crates/fff-core/src/scan.rs:148:14:
failed to spawn fff-scan thread: Os { code: 11, kind: WouldBlock, message: "Resource temporarily unavailable" }
fatal runtime error: failed to initiate panic, error 5, aborting

~15 of these SIGABRTs within 20 seconds, followed by schema-retry churn ("The output file was NOT created" / "Process killed by signal 6") that re-spawned children into the same exhausted container, then node restart.

Root cause: the peak-concurrency phase has no shared limiter

Phase 6 ‖ 6.7 run concurrently, and each side brings its own (or no) limiter:

1. Consistency-verify is unbounded. Up to 12 verify_obligation harness calls go out via a bare asyncio.gather — no semaphore at all (cap literal, gather).
2. The reviewer semaphore is per-call, not review-wide. _run_parallel_review constructs a fresh Semaphore(max_concurrent_reviewers=8) on every invocation, so the coverage loop's gap reviewers don't share a budget with anything else.
3. The only global cap is the SDK's OPENCODE_MAX_CONCURRENT (default 10, process-wide, in agentfield's opencode provider). In the crash window 12 + 4 = 16 harness calls contended for 10 slots. Each admitted opencode/Bun process holds hundreds of MB scanning a workspace with a 60k-line diff, and the SDK path buffers each child's full stdout with no truncation (observed 291k chars), effectively three times over: raw chunk list, joined string, parsed JSONL event list.
4. Evidence extraction stacks uncapped children on top. A real grep -RInE is forked per mentioned identifier per finding (subprocess.run, plus a per-file import-context grep at L400), with no cap on identifier count — this PID/thread pressure is what surfaced as EAGAIN before the memory kill completed.
5. The degenerate diff got maximum effort. Generated-lockfile churn counted toward depth-escalation signals (quick → standard fired) and toward obligation extraction — so the least reviewable diffs trigger the most fan-out.
6. Baseline is inflated by an unbounded-by-bytes cache. _FILE_CACHE is module-global and holds full line-lists of up to 2000 files across all reviews in the process — bounded by entry count, not bytes (evidence.py#L19).

Ruled out: the Go port. The deployed service is the Python orchestrator (root Dockerfile builds src/ only); commit 8f1abeb changed packaging/go/* only, zero src/ changes vs the deployment that had been running since 2026-07-29.

Proposed fix (prioritized)

1. One review-wide concurrency budget. A single orchestrator-owned semaphore that every fan-out acquires — review dimensions, recursive sub-reviews, coverage iterations, verify_obligation, adversary batches, polish/merge-gate .ai() calls. Env-configurable (PR_AF_MAX_CONCURRENT_AGENTS), defaulting to something the deployment's memory can actually sustain.
2. Bound consistency-verify under that budget and make the 12-obligation literal configurable.
3. Treat generated files as generated. Exclude lockfiles (package-lock.json, yarn.lock, pnpm-lock.yaml, uv.lock, poetry.lock, …) from depth-escalation signals and obligation extraction; review dependency changes from the manifest diff instead.
4. Cap evidence fan-out (identifiers per finding) and byte-bound _FILE_CACHE.
5. Truncate/stream harness stdout (upstream in agentfield's opencode provider): keep parsed events, drop the redundant raw copies.
6. Ops stopgaps: set OPENCODE_MAX_CONCURRENT=4–6 on the deployment until (1) lands, and pin agentfield==<version> in the Dockerfile so rebuilds stop silently floating the SDK (the Aug 4 rebuild jumped ≤0.1.117 → 0.1.120 as a side effect).

Latent issues in the same code worth fixing while here:

• Recursive semaphore acquisition: a parent dimension holds its slot (L700) while awaiting its sub-review children, which need slots from the same semaphore — at deep depth (12 dims) this can deadlock outright.
• build_dimension_pack runs blocking subprocess.run on the event-loop thread.
• /workspaces/<repo>-pr<N> clones are never cleaned up (persistent-volume disk growth).

Validation contract

• Reviewing a lockfile-regen PR (agentfield#883 makes a good fixture) never exceeds the configured agent cap at any instant — including when consistency-verify and the coverage loop overlap (cap, not cap + 12).
• A lockfile-only PR does not escalate review depth.
• Under the fixture on a memory-limited container, the review completes with no EAGAIN/SIGABRT cascade and no OOM kill.
• Cap is tunable via env without a code change.

Longer-term direction: furrow

From team discussion (2026-08-06, internal Slack): this is a candidate spot for furrow — byte-exact copy-on-write forks of the whole workspace, so the N concurrent reviewer agents each get an isolated fork of one materialized checkout (shared package state included) instead of all contending over the same /workspaces checkout. That handles the shared-package concern and bounds per-agent workspace overhead as the fan-out grows. Suggested by @santoshkumarradha; makes sense as a follow-on once the concurrency budget (fix 1) exists.