Skip to content

Commit a80ce26

Browse files
norandomclaude
andcommitted
research(annotation-driver): run batches in parallel worker copies (R62, gate Amendment 9)
Worker copies with a shared queue, time limits scaled from the unmutated run, a long-lived Java container that compiles one file and runs one test class, and one shared outcome store. Same 40-mutant batches, nothing reused: Java 1,875 s to 178 s, TypeScript 412 s to 70 s, Python 97 s to 32 s, every class identical to the serial receipts. Rust stays serial (concurrent builds exhaust memory). Launch failures and out-of-memory builds are now class error instead of killed or invalid. Co-Authored-By: Claude <noreply@anthropic.com>
1 parent 41cbe05 commit a80ce26

19 files changed

Lines changed: 11238 additions & 64 deletions

‎.kiro/handover.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,8 @@ has the result table, the open decisions, the owner's rules and the commands.
3131
5. **Open from R55:** Python `deconvolute analyze` is a stub that reports
3232
`passed: true`; the pre-push hook's refactor-branch check does nothing;
3333
the hard-coded mutation operators stay until a design passes the gate.
34+
Batches now run in parallel worker copies (R62): a full pass over the
35+
four scopes takes about 8 minutes instead of 43, with identical results.
3436
6. **Working rules from the owner today:** the agent decides strategy and
3537
records it; clear implementation goes to Opus at xhigh effort; do not
3638
manage the Podman machine, only run containers; the runs were too slow,

‎.kiro/specs/agnostic-annotation-driver/prototype-gate.md‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -409,6 +409,30 @@ mutant; timeouts are reported with that caveat.
409409
Applies to Java arm A run 2 and Java D2, which are rerun or run with it.
410410
`harness.py` SHA-256 `337b47e75921d3e94cca7aa5d272ec57f9708eda5447c26bca0af84fdbfd13c6`.
411411

412+
## Amendment 9, 2026-11-10 (operational; after attempt 1, owner: the serial runs make no sense)
413+
414+
How batches run, from now on ([R62](../../../docs/research/2026-11-10/08-annotation-driver-parallel-batches.md)):
415+
416+
- Mutants of a scope run concurrently in independent worker copies that
417+
pull from one queue (`run_arm.py --workers N`). Rust is limited to one
418+
worker: concurrent builds of that scope exhaust memory.
419+
- A run may take five times what the unmutated run needed on that worker,
420+
with a floor per scope, in place of the fixed limits of Amendment 5.
421+
- Java runs in one long-lived container per worker, compiling only the
422+
scope file and running the one test class.
423+
- One outcome store is shared by all workers and the serial path
424+
(Amendment 8).
425+
- New class `error`: a test command that did not start, or a build that ran
426+
out of memory, is not a result about the mutant. It is retried once, never
427+
stored, and reported. A failing exit counts as killed only when the test
428+
runner's own output shows that it ran. `error` mutants are outside every
429+
row's numerator and denominator, and a batch with any left is incomplete.
430+
431+
Verified before use: with reuse off, every D1 mutant of the four scopes gets
432+
the same class as in the serial receipts of R60. None of the 240 kills of
433+
attempt 1 was a launch failure. Classification of real test runs is
434+
unchanged.
435+
412436
## Result of attempt 1, 2026-11-10
413437

414438
Outcome `revise`; full tables and limits in R61. No design approval and no

‎.kiro/specs/agnostic-annotation-driver/spec.json‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,8 @@
3333
"docs/research/2026-11-10/04-annotation-driver-validator-controls.md",
3434
"docs/research/2026-11-10/05-annotation-driver-strategy-and-harness.md",
3535
"docs/research/2026-11-10/06-annotation-driver-d1-results.md",
36-
"docs/research/2026-11-10/07-annotation-driver-gate-result.md"
36+
"docs/research/2026-11-10/07-annotation-driver-gate-result.md",
37+
"docs/research/2026-11-10/08-annotation-driver-parallel-batches.md"
3738
],
3839
"notes": "2026-11-10: owner set priorities (agnostic driver first, inference-drift hardening second) and approved a new spec with mutation as the first annotation consumer. Brief and prototype gate written for review; gate not run. No requirements, design or tasks generated. Owner go (proceed) 2026-11-10: gate checklist step 1 scope screening done (R57, gate Amendment 1: four scopes admitted, control expectation corrected). Arm D (rules as data, model as offline rule author) approved by the owner 2026-11-10. Owner requirement 2026-11-10: hard-coded operators to be removed completely (gate Amendment 2b). Step 2 done (R58): validator controls. Gate attempt 1 completed 2026-11-10 with outcome revise (R61): Rust, Python and Java pass in at least one arm, TypeScript fails in every arm; model-written packs admitted and equivalent to hand-written ones. 21 model calls. Attempt 2 awaits the owner."
3940
}

‎.kiro/task_28_handover.md‎

Lines changed: 8 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -89,6 +89,11 @@ uv run --no-sync python annotate.py --scope NAME --scopes-dir $scopes --guide gu
8989
uv run --no-sync python run_arm.py --scope NAME --scopes-dir $scopes --out-dir OUT --annotations OUT\annotations.json
9090
```
9191

92+
- **Run batches in parallel:** add `--workers 6` to `run_arm.py` (Rust is
93+
capped at 1 by `scopes.json`). A D1 batch then takes about 30 s (Python),
94+
70 s (TypeScript), 3 min (Java) and 3.5 min (Rust); see R62. Worker copies
95+
live in `...\annotation-driver\workers`. A mutant classed `error` means
96+
the test command did not run; the batch is then incomplete.
9297
- Scope names are in `scopes.json`. The scopes are isolated clones under
9398
`D:\BuildArtifacts\lesscode\annotation-driver\scopes`; pristine read-only
9499
copies are in `...\scopes-ro` (use these for checks while a batch runs).
@@ -98,9 +103,9 @@ uv run --no-sync python run_arm.py --scope NAME --scopes-dir $scopes --out-dir O
98103
- Model endpoint: `%USERPROFILE%\.lesscode\model.env` (Anthropic style,
99104
`claude-opus-5-5`). Every real call appends to a `ledger.jsonl` next to
100105
its output directory. Real calls cost money.
101-
- Java tests run through `docker compose -f compose.yaml run --rm java-tests`
102-
with `SCOPES_DIR` set; each run starts a fresh Maven container and takes
103-
20 to 60 seconds. Rust tests for the `mmhmem` scope must use the restricted
106+
- Java tests run by `docker compose exec` in a long-lived `java-worker`
107+
container per worker (started by the batch; stop idle ones with
108+
`docker container stop`). Rust tests for the `mmhmem` scope must use the restricted
104109
target list in `scopes.json` (other test targets do not compile at that
105110
revision).
106111

‎docs/experiments/annotation-driver/README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ Research entries: [R56](../../research/2026-11-10/02-agnostic-annotation-driver-
1616
| `engine.py` | Finds eligible sites and applies the generic operators; reads only the packs |
1717
| `harness.py`, `run_arm.py` | Runs a batch of mutants against a scope's own tests and writes a receipt |
1818
| `run_engine_checks.py` | Model-free checks of packs, positions, operators, counts, selection and the harness |
19+
| `pool.py` | Runs a batch in parallel worker copies (`run_arm.py --workers N`); see R62 |
20+
| `java-fast-test.sh` | Compiles one Java file and runs one test class inside the long-lived test container |
1921
| `results/` | Retained receipts |
2022
| `compose.yaml` | Linux-side tools (Maven for the Java scope) |
2123

‎docs/experiments/annotation-driver/compose.yaml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,5 +37,19 @@ services:
3737
- -Dtest=${JAVA_TEST:-LexerTest}
3838
- test
3939

40+
# Long-lived variant: started once per batch, each test run is an exec in it,
41+
# so the container and Maven's start-up are not paid per mutant.
42+
java-worker:
43+
image: maven:3.9-eclipse-temurin-21
44+
working_dir: /scope
45+
volumes:
46+
- ${SCOPES_DIR:?set SCOPES_DIR to the isolated scopes directory}/commons-csv:/scope
47+
- maven-repo:/root/.m2
48+
- ./java-fast-test.sh:/opt/ad/java-fast-test.sh:ro
49+
command: ["sleep", "infinity"]
50+
4051
volumes:
52+
# One fixed name, so every project (one per worker copy, COMPOSE_PROJECT_NAME
53+
# in scopes.json) uses the same populated repository for `mvn -o`.
4154
maven-repo:
55+
name: lesscode-annotation-driver_maven-repo

0 commit comments

Comments
 (0)