fix(planning): the spec travels with the plan - #2086
Conversation
…+ SDD reads it at setup In controlled evals, an identical seeded-incoherence plan yielded 0-1/5 correct conflict resolutions when executed specless (controllers ruled the conflicts 'internally explained') and 4-5/5 with the spec merely present and named — even with no other skill-text changes. Cross-task coherence turns out to be adjudicable only against ground truth above the plan; this change makes that ground truth travel with the plan. Claude-Session: https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy
|
@arittr Ready for your eval treatment — same program as the merged stack and #2089. Evidence summary: the specless wall (0-1/5 seeded plan-incoherence conflicts resolved, four different text arms) vs spec-present 4-5/5 with stock skills, on gpt-5.6 (terra-uniform battery); the exact texts in this diff ran verbatim in a composed battery (5/5 ×3, no regression); and the mechanism replicates on claude-sonnet-5 via the quorum claude adapter (spec 5/5 ×2, specless 0/5 ×2). Scenario fixtures (cp-x10-spec / cp-x10-consistency, seeded-value answer key) are public in prime-radiant-inc/superpowers-autoresearch if you want to run them directly — happy to walk you through the setup. — Claude (Fable 5), on the eval program with @obra |
|
Post-merge cross-model validation, for the record: the spec-in-the-room contrast was re-run on two more model families in the same containerized eval harness (superpowers at the merged dev base, full SDD sessions, same seeded 5-defect plan).
That makes four model families with the same shape — gpt-5.6 (0-1/5 → 4-5/5), claude-sonnet-5 (0/5 → 5/5), kimi and glm above. All 8 new reps passed the LLM-verifier gate; the seeded-defect scans are tree-resolving and were hand-checked. — Claude (Fable 5, Claude Code session; evidence log in superpowers-autoresearch logs/2026-08-05-adjudication-battery.md, 2026-08-07 entry) |
Who is submitting this PR? (required)
claude-fable-5)What problem are you trying to solve?
Plans executed without the spec they were derived from are unauditable for cross-task coherence. In our eval program this failure recurred at every layer we instrumented: whole-branch reviewers detected 0/45 seeded cross-module inconsistencies (praising drift as design), a mechanical consistency scanner's findings were 100% dismissed at adjudication ("the plan explicitly requires both values"), and a preflight value-table with amendment authority enumerated the exact seeded conflicts and ruled them "internally explained." The root cause is structural: each side of a plan-internal contradiction is individually plan-mandated, so once the plan is the only authority in scope, incoherence is unfalsifiable — every instrument reaches the same "intentional per-module policy" conclusion.
What does this PR change?
Two navigational additions (+7/−1 across two files):
skills/writing-plans/SKILL.md: the plan header template gains a**Spec:**line pointing at the spec/design doc the plan implements — the spec travels with the plan.skills/subagent-driven-development/SKILL.md: the setup step reads the plan's named spec ("the spec is the authority the plan argues from; conflicts inside the plan resolve against it"), and a plan with no reachable spec gets a ledger note that rulings are provisional.Is this change appropriate for the core library?
Yes — core planning/execution plumbing, project-agnostic, no dependencies. The superpowers flow already produces specs (brainstorming → spec → plan); this closes the loop so execution can see them.
What alternatives did you consider?
Does this PR contain multiple unrelated changes?
No — one mechanism (the spec travels with the plan), two files that each carry half of it.
Existing PRs
Environment tested
New harness support (required if this PR adds a new harness)
N/A.
Evaluation
logs/2026-08-03-plan-decomposition-campaign.md(P2-on-cp-x10 FAIL verdict, P2' three-arm battery pre-registration + verdict, dismisser-interrogation entry) and the cp-x10-spec scenario + answer key undercampaigns/cost-pathologies/scenarios/.https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy