Skip to content

fix(sdd): batch small same-shape tasks into one dispatch - #2078

Merged
arittr merged 2 commits into
devfrom
fix/x6a-sdd-batch-small-tasks
Aug 4, 2026
Merged

fix(sdd): batch small same-shape tasks into one dispatch#2078
arittr merged 2 commits into
devfrom
fix/x6a-sdd-batch-small-tasks

Conversation

@obra

@obra obra commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Who is submitting this PR? (required)

Field Value
Your model + version Claude Fable 5 (claude-fable-5)
Harness + version Claude Code 2.1.220
All plugins installed superpowers 6.2.0, episodic-memory, linear, context7, superpowers-chrome, agent-sdk-dev, code-simplifier, github-triage, plugin-dev
Human partner who reviewed this diff Jesse Vincent (@obra) — directed the treatment, the eval program, and this submission; reviewing the diff here

What problem are you trying to solve?

Subagent-driven-development pays a fixed overhead per task: a fresh implementer dispatch plus a task-review dispatch, each carrying the brief, the diff package, and a full agent context. That price is right for tasks that need their own judgment and test cycle. But real plans sometimes enumerate many tiny, same-shape edits — the same one-line fix across N files, a constant change, a field addition — as separate tasks, and the current skill text dispatches one implementer + one review per micro-task. In our mined-session baseline (174 SDD sessions), the median dispatch consumes roughly 79,500× more tokens in agent overhead than the useful output it produces; micro-task plans are where that ratio does the most damage.

What does this PR change?

skills/subagent-driven-development/SKILL.md (+8 lines, one file): a rule at the top of the Task Loop — when the plan lists several small, independent, same-shape edits, compose ONE dispatch brief listing every file and its change, send the batch to a single subagent, and review the result as one diff. The rule's own boundary reserves per-task dispatch for anything needing its own judgment, its own tests, or its own review surface.

Is this change appropriate for the core library?

Yes — it modifies core SDD controller behavior that applies to any project using subagent-driven-development. No dependencies, no domain-specific content.

What alternatives did you consider?

  • Fix it at plan-authoring instead (writing-plans right-sizes tasks so micro-task plans never exist): registered as a competing hypothesis in our eval program and worth pursuing — but SDD also executes plans it did not author (imported, legacy, human-written), so the execution-side guard retains value regardless. Defense in depth, not either/or.
  • Cheaper models per micro-task (already in the skill's Model Selection): reduces per-dispatch price but keeps the seat count — every micro-task still costs an implementer + reviewer round-trip with full context. Measured effect is much smaller than removing the seats.
  • Do nothing: the mined baseline above says this is among the largest recurring cost shapes in real SDD sessions.

Does this PR contain multiple unrelated changes?

No — one file, one rule.

Existing PRs

Environment tested

Harness (e.g. Claude Code, Cursor) Harness version Model Model version/ID
Claude Code (authoring) 2.1.220 Claude Fable 5 claude-fable-5
Codex CLI (containerized evals, quorum harness) 0.46 lineage GPT-5.6-Codex gpt-5.6-codex

New harness support (required if this PR adds a new harness)

N/A — no new harness.

Evaluation

Two pre-registered batteries, containerized quorum evals (fresh clone per rep, pinned scripted user replies, mechanical scoring):

  • Where batching SHOULD bind — a 12-micro-task plan (same-shape small edits): treatment cut cost 73% and dispatches 87% vs control, with better completion (control reps dropped edits; the batched pass completed them all).
  • Where batching must NOT bind (the ship gate) — a 5-task, 1100-line Go plan whose tasks are individually non-trivial (multi-file renderers with test cycles): treatment dispatch counts 14/14/13 vs control 16/13, i.e. zero batching; completion parity (5/5 reps pass, 7/7 post-checks each); cost parity ($7.96 vs $8.49 mean). The boundary language held 3/3.
  • Full records (public): https://github.com/prime-radiant-inc/superpowers-autoresearchlogs/2026-08-01-queue-campaign.md (Task 7 fixture + MINE baseline, Task 12 battery verdict, 2026-08-02 ship-gate pre-registration and verdict) and reports/2026-08-queue-campaign.md §2.

https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy

Plans sometimes enumerate many tiny, same-shape edits (one-line fixes,
constant changes, a field added across files) as separate tasks. The
current loop dispatches a fresh implementer plus review per task, so a
12-micro-task plan costs ~24 subagent seats for what one subagent could
do in a single pass. In controlled evals on a micro-task plan, batching
cut cost 73% and dispatches 87% with better completion than control; on
a 5-non-trivial-task plan the rule correctly never batched (dispatch
counts and completion identical to control).

Claude-Session: https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy
@obra

obra commented Aug 3, 2026

Copy link
Copy Markdown
Owner Author

@arittr — as with #2077, we'd value an independent eval run on this one. Three behaviors worth testing (this is Claude, writing at Jesse's direction):

  1. The batching case: an SDD plan of many tiny same-shape tasks (e.g. the same one-line fix across 10+ files as 10+ tasks). Success = ONE implementer dispatch covering the batch, all edits actually landing, and one review of the combined diff. Failure modes to watch: edits silently dropped from the batch, or the controller still dispatching per-task.
  2. The boundary: a plan of individually non-trivial tasks (own tests, multi-file, judgment). Success = zero batching — dispatch counts identical to what you'd see without this text. In our gate battery (5-task Go plan) treatment matched control 14/14/13 vs 16/13.
  3. Review integrity under batching: the batched diff must still get a genuine task review as one unit — batching the implementation must not become an excuse to skip or thin the review.

Full eval records are public in the autoresearch repo (linked in the PR body): the micro-task battery (73% cost / 87% dispatch reduction, better completion) and the boundary gate (3/3 no-batching, full parity).

Batching moves N edits under one review, which changes the review's
failure profile: an implementer that silently skips one file of twelve
produces a diff full of correct, uniform edits — nothing conspicuous is
missing, and no seat in the pipeline was assigned to notice. The single
combined review is the only net for a dropped edit, but the reviewer
template never told it to count.

The batch brief already lists every file with its change, so the reviewer
reconciles the diff against that list file by file; a listed file with no
hunk is a Missing finding regardless of how clean the rest of the batch
looks. Conditional on a multi-file brief, so single-task reviews are
unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@arittr arittr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arittr
arittr merged commit 05c2393 into dev Aug 4, 2026
@arittr

arittr commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

Independent validation from my environment, as requested — posting post-merge for the record. Setup: quorum A/B (3 reps × 7 scenarios, codex lane, dev vs the full 7-PR stack, both arms on the same binary — host codex 0.144.4, so V2-specific claims were validated separately in live probes on codex-cli 0.146.0 with subscription auth), plus a fresh 2-task SDD run on 0.146. Full numbers below for this PR.

This was the one PR in the set with no independent verification available — no should-bind scenario exists in any eval corpus we have. What we can say: across the full A/B, no false batching on non-trivial plans (dispatch structure at parity with dev), and the dropped-edit net you named as the top failure mode is now closed at the reviewer seat (78cc1892: batch reviews reconcile the diff against the brief's file list; a listed file with no hunk is a Missing finding).

Missing eval asset if you want the bind rate measured: a should-bind battery (10–15 same-shape micro-tasks → expect 1 implementer + 1 review, every edit present, one deliberately wrong edit spec that the review must catch) plus a must-not-bind boundary battery (individually non-trivial tasks → expect zero batching).

Validation performed by Claude Fable 5 (Claude Code) at my direction.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants