Skip to content

fix(planning): the spec travels with the plan - #2086

Merged
arittr merged 1 commit into
devfrom
fix/spec-travels-with-plan
Aug 6, 2026
Merged

fix(planning): the spec travels with the plan#2086
arittr merged 1 commit into
devfrom
fix/spec-travels-with-plan

Conversation

@obra

@obra obra commented Aug 4, 2026

Copy link
Copy Markdown
Owner

Who is submitting this PR? (required)

Field Value
Your model + version Claude Fable 5 (claude-fable-5)
Harness + version Claude Code 2.1.220
All plugins installed superpowers 6.2.0, episodic-memory, linear, context7, superpowers-chrome, agent-sdk-dev, code-simplifier, github-triage, plugin-dev
Human partner who reviewed this diff Jesse Vincent (@obra) — directed the eval program and this submission; reviewing the diff here (opened as DRAFT pending that review)

What problem are you trying to solve?

Plans executed without the spec they were derived from are unauditable for cross-task coherence. In our eval program this failure recurred at every layer we instrumented: whole-branch reviewers detected 0/45 seeded cross-module inconsistencies (praising drift as design), a mechanical consistency scanner's findings were 100% dismissed at adjudication ("the plan explicitly requires both values"), and a preflight value-table with amendment authority enumerated the exact seeded conflicts and ruled them "internally explained." The root cause is structural: each side of a plan-internal contradiction is individually plan-mandated, so once the plan is the only authority in scope, incoherence is unfalsifiable — every instrument reaches the same "intentional per-module policy" conclusion.

What does this PR change?

Two navigational additions (+7/−1 across two files):

  • skills/writing-plans/SKILL.md: the plan header template gains a **Spec:** line pointing at the spec/design doc the plan implements — the spec travels with the plan.
  • skills/subagent-driven-development/SKILL.md: the setup step reads the plan's named spec ("the spec is the authority the plan argues from; conflicts inside the plan resolve against it"), and a plan with no reachable spec gets a ledger note that rulings are provisional.

Is this change appropriate for the core library?

Yes — core planning/execution plumbing, project-agnostic, no dependencies. The superpowers flow already produces specs (brainstorming → spec → plan); this closes the loop so execution can see them.

What alternatives did you consider?

  • Stronger gate text instead of spec plumbing: tested — a preflight value-table with plan-amendment authority (an arm building on fix(sdd): preflight emits its checks as a ledger table and rules on what it surfaces #2080's evidence-bearing preflight) scored 0-1/5 on a seeded-incoherence plan when no spec was in scope. Text cannot substitute for ground truth.
  • Mechanical consistency scanning at final review: built and tested — the scanner surfaced 14/14 candidates with file:line and adjudication dismissed 100% under plan authority. Same root cause.
  • Anti-rationalization wording alone (naming the dismissal pattern in the skill text): under test in a running battery; interrogation of the dismissing sessions suggests the operative driver is "tests are green so the divergences must be fine," which ground-truth availability addresses directly.

Does this PR contain multiple unrelated changes?

No — one mechanism (the spec travels with the plan), two files that each carry half of it.

Existing PRs

Environment tested

Harness (e.g. Claude Code, Cursor) Harness version Model Model version/ID
Claude Code (authoring) 2.1.220 Claude Fable 5 claude-fable-5
Codex CLI (containerized evals, quorum harness) 0.46 lineage GPT-5.6-Codex gpt-5.6-codex

New harness support (required if this PR adds a new harness)

N/A.

Evaluation

  • Method: containerized quorum evals; one seeded-incoherence fixture (a 6-task plan whose task briefs pin 5 deliberately incompatible shared values), run in two scenario variants: specless, and with the product spec the plan derives from present and named in the kickoff message. Pre-registered criteria; mechanical spec-resolution scoring + controller hand-reads; raw records public.
  • Results: specless execution: 0-1/5 conflicts resolved correctly, with controllers ruling the divergences "deliberate module-local differences... internally explained" (3/3 reps). Spec present, unmodified stock skills (the exact configuration this PR ships): 4/5, 5/5, 4/5 — detection came free; controllers enumerated all five plan/spec conflicts immediately. Gate-text arms on the same spec'd scenario reached uniform 5/5 and resolved autonomously (no human ask), which is why the SDD half of this change also anchors preflight rulings to the spec.
  • Cross-model (2026-08-05): the mechanism replicates on claude-sonnet-5 via the quorum claude adapter, stock skills: spec present 5/5 ×2, specless 0/5 ×2 (hand-verified divergent values in fully built trees). With the gpt-5.6 batteries this covers the two largest user segments; kimi/glm cells deferred with triggers.
  • Disclosure/update: the SDD sentence in this diff has now ALSO run verbatim as a composed battery arm (with the reviewer-template paragraph from fix(sdd): reviewers re-read illegible evidence instead of re-running to regenerate it #2089 co-present): 3/3 reps resolved 5/5 seeded plan/spec conflicts (hand-verified), post-checks 19/19, no regression from the stock-with-spec baseline. The specless ablation also completed after this PR was opened: three further specless text arms (including a counter elicited from the dismissing sessions themselves) all scored 0/5 — the spec's presence, not any wording, is the active ingredient.
  • Full records (public): https://github.com/prime-radiant-inc/superpowers-autoresearchlogs/2026-08-03-plan-decomposition-campaign.md (P2-on-cp-x10 FAIL verdict, P2' three-arm battery pre-registration + verdict, dismisser-interrogation entry) and the cp-x10-spec scenario + answer key under campaigns/cost-pathologies/scenarios/.

https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy

…+ SDD reads it at setup

In controlled evals, an identical seeded-incoherence plan yielded 0-1/5
correct conflict resolutions when executed specless (controllers ruled
the conflicts 'internally explained') and 4-5/5 with the spec merely
present and named — even with no other skill-text changes. Cross-task
coherence turns out to be adjudicable only against ground truth above
the plan; this change makes that ground truth travel with the plan.

Claude-Session: https://claude.ai/code/session_0185AJr98gHx5EmwqNeft4Sy
@obra

obra commented Aug 6, 2026

Copy link
Copy Markdown
Owner Author

@arittr Ready for your eval treatment — same program as the merged stack and #2089. Evidence summary: the specless wall (0-1/5 seeded plan-incoherence conflicts resolved, four different text arms) vs spec-present 4-5/5 with stock skills, on gpt-5.6 (terra-uniform battery); the exact texts in this diff ran verbatim in a composed battery (5/5 ×3, no regression); and the mechanism replicates on claude-sonnet-5 via the quorum claude adapter (spec 5/5 ×2, specless 0/5 ×2). Scenario fixtures (cp-x10-spec / cp-x10-consistency, seeded-value answer key) are public in prime-radiant-inc/superpowers-autoresearch if you want to run them directly — happy to walk you through the setup.

— Claude (Fable 5), on the eval program with @obra

@arittr arittr left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@arittr
arittr merged commit fdd1763 into dev Aug 6, 2026
@obra

obra commented Aug 7, 2026

Copy link
Copy Markdown
Owner Author

Post-merge cross-model validation, for the record: the spec-in-the-room contrast was re-run on two more model families in the same containerized eval harness (superpowers at the merged dev base, full SDD sessions, same seeded 5-defect plan).

  • kimi (Kimi Code CLI, oauth lane), 2 reps/condition: spec-present 5/5 + 5/5 seeded inconsistencies resolved to spec values; specless 0/5 + 0/5.
  • glm-5.2 (pi harness via OpenRouter), 2 reps/condition: spec-present 5/5 + 5/5; specless 0/5 + 0/5.

That makes four model families with the same shape — gpt-5.6 (0-1/5 → 4-5/5), claude-sonnet-5 (0/5 → 5/5), kimi and glm above. All 8 new reps passed the LLM-verifier gate; the seeded-defect scans are tree-resolving and were hand-checked.

— Claude (Fable 5, Claude Code session; evidence log in superpowers-autoresearch logs/2026-08-05-adjudication-battery.md, 2026-08-07 entry)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants