fix(sdd): batch small same-shape tasks into one dispatch - #2078
Conversation
Plans sometimes enumerate many tiny, same-shape edits (one-line fixes, constant changes, a field added across files) as separate tasks. The current loop dispatches a fresh implementer plus review per task, so a 12-micro-task plan costs ~24 subagent seats for what one subagent could do in a single pass. In controlled evals on a micro-task plan, batching cut cost 73% and dispatches 87% with better completion than control; on a 5-non-trivial-task plan the rule correctly never batched (dispatch counts and completion identical to control). Claude-Session: https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy
|
@arittr — as with #2077, we'd value an independent eval run on this one. Three behaviors worth testing (this is Claude, writing at Jesse's direction):
Full eval records are public in the autoresearch repo (linked in the PR body): the micro-task battery (73% cost / 87% dispatch reduction, better completion) and the boundary gate (3/3 no-batching, full parity). |
Batching moves N edits under one review, which changes the review's failure profile: an implementer that silently skips one file of twelve produces a diff full of correct, uniform edits — nothing conspicuous is missing, and no seat in the pipeline was assigned to notice. The single combined review is the only net for a dropped edit, but the reviewer template never told it to count. The batch brief already lists every file with its change, so the reviewer reconciles the diff against that list file by file; a listed file with no hunk is a Missing finding regardless of how clean the rest of the batch looks. Conditional on a multi-file brief, so single-task reviews are unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Independent validation from my environment, as requested — posting post-merge for the record. Setup: quorum A/B (3 reps × 7 scenarios, codex lane, dev vs the full 7-PR stack, both arms on the same binary — host codex 0.144.4, so V2-specific claims were validated separately in live probes on codex-cli 0.146.0 with subscription auth), plus a fresh 2-task SDD run on 0.146. Full numbers below for this PR. This was the one PR in the set with no independent verification available — no should-bind scenario exists in any eval corpus we have. What we can say: across the full A/B, no false batching on non-trivial plans (dispatch structure at parity with dev), and the dropped-edit net you named as the top failure mode is now closed at the reviewer seat ( Missing eval asset if you want the bind rate measured: a should-bind battery (10–15 same-shape micro-tasks → expect 1 implementer + 1 review, every edit present, one deliberately wrong edit spec that the review must catch) plus a must-not-bind boundary battery (individually non-trivial tasks → expect zero batching). Validation performed by Claude Fable 5 (Claude Code) at my direction. |
Who is submitting this PR? (required)
claude-fable-5)What problem are you trying to solve?
Subagent-driven-development pays a fixed overhead per task: a fresh implementer dispatch plus a task-review dispatch, each carrying the brief, the diff package, and a full agent context. That price is right for tasks that need their own judgment and test cycle. But real plans sometimes enumerate many tiny, same-shape edits — the same one-line fix across N files, a constant change, a field addition — as separate tasks, and the current skill text dispatches one implementer + one review per micro-task. In our mined-session baseline (174 SDD sessions), the median dispatch consumes roughly 79,500× more tokens in agent overhead than the useful output it produces; micro-task plans are where that ratio does the most damage.
What does this PR change?
skills/subagent-driven-development/SKILL.md(+8 lines, one file): a rule at the top of the Task Loop — when the plan lists several small, independent, same-shape edits, compose ONE dispatch brief listing every file and its change, send the batch to a single subagent, and review the result as one diff. The rule's own boundary reserves per-task dispatch for anything needing its own judgment, its own tests, or its own review surface.Is this change appropriate for the core library?
Yes — it modifies core SDD controller behavior that applies to any project using subagent-driven-development. No dependencies, no domain-specific content.
What alternatives did you consider?
Does this PR contain multiple unrelated changes?
No — one file, one rule.
Existing PRs
devindependent of all of them; merge order is free). No open or closed PR adds dispatch batching to SDD.Environment tested
New harness support (required if this PR adds a new harness)
N/A — no new harness.
Evaluation
Two pre-registered batteries, containerized quorum evals (fresh clone per rep, pinned scripted user replies, mechanical scoring):
logs/2026-08-01-queue-campaign.md(Task 7 fixture + MINE baseline, Task 12 battery verdict, 2026-08-02 ship-gate pre-registration and verdict) andreports/2026-08-queue-campaign.md§2.https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy