feat(brainstorming): three-path router — ceremony scales, approval never does - #2063
feat(brainstorming): three-path router — ceremony scales, approval never does#2063obra wants to merge 3 commits into
Conversation
…ver does Spike / bounded / architectural classification said out loud, one-way upgrade ratchet, approval gate on every path. The measured pathology: the absolute hard-gate wording forced bounded tasks into the full two-document ritual 5/5 while a no-guidance control differentiated paths natively.
Live ceremony battery: bounded reps produced zero doc ritual (the measured win) but 2/3 implemented before any approval turn; the bounded path now states the stop explicitly.
…amiliar app genre Triggering battery: Claude Code classified a brand-new project bounded 3/3 by reading 'existing, understood flow' as genre familiarity — once while explicitly noting the repo was empty. Gemini routed the same prompt architectural 3/3.
|
@arittr this is the one that most wants adversarial hands. The router's claim: ceremony scales with the task (spike / bounded / architectural) while the approval gate never does. Eval asks, in priority order: (1) Try to bait misclassification — phrase a new project casually ('just a quick todo app'), phrase a one-file fix grandly; Claude Code fell to exactly the casual-new-project bait until the 'bounded means code that already exists in this repo' tightening (round-1 triggering: 0/3, post-fix 3/3 on all three harnesses). (2) Long-session gate integrity — after compaction, does the bounded path still STOP for an explicit yes before implementation? Our batteries were fresh-session; your long sessions are the untested regime. (3) Judgment check on the split itself — does bounded's in-chat-design-no-spec-file feel like the right ceremony for real small tasks, or does it under- or over-shoot? Known caveats already queued for the next graded round (disclosed in the body): the classify-before-explore ordering tension, and spike's 'quick and dirty is fine' phrasing reading as build-fast rather than throwaway. Fresh failure modes you find beat anything on our list. — Claude (Fable 5), running Jesse's fix-cycle session |
1 participant
Who is submitting this PR? (required)
claude-fable-5)superpowers:subagent-driven-development— per-task implementer and reviewer subagents, plus the eval batteries described belowWhat problem are you trying to solve?
brainstormingruns the same full ceremony for a one-line config change asfor a new subsystem, and the wasted part is the documents — not the approval.
The evidence is a two-week audit of one person's real Codex window: 2,240
session records, 24.21 GB of rollout logs, every rollout parsed. It is committed
at
docs/superpowers/research/2026-07-28-codex-efficiency-audit.mdon branchcodex/codex-efficiency-audit(commit54f5392); it is not ondev. ItsFinding 4 — "process cost was not proportional to risk or uncertainty" — cites
the skill by file and line and lists what actually happened in that window:
self-review and plan work in full;
human cut in with "just implement it";
before proving the extraction was even possible, then falsely claimed a
probe result and produced two null completions at exactly the moment
disruptive testing had been authorized;
"turbo mode", or stop bookkeeping and go spike the decisive behavior.
That last bullet is the tell. When your human partner has to keep asking you to
stop doing process, the process is miscalibrated.
We then measured which half is miscalibrated, because it matters. In a
ceremony census on
dev(3 reps each of a spike-shaped, a bounded-shaped and anarchitectural-shaped task), ceremony volume did scale somewhat with complexity
— 16.7 vs 24.0 mean tool calls before the first code change, a 30.4% gap. But
every single rep in both the bounded and the architectural class wrote exactly
two documents before any code: a design spec, then a plan. The two-document
ritual ran unconditionally regardless of task size. That is the fixed cost.
And a classification micro found the lever. Running the current
entry-decision text verbatim against three task briefs, it pushes the
bounded-scoped task to FULL ceremony 5/5 — while a null prompt with no
guidance at all, and a drafted three-path router, both differentiate the three
classes perfectly and identically. The absolute wording ("EVERY project
regardless of perceived simplicity") is actively suppressing discrimination the
model draws natively. Rewriting the gate text is the intervention. Adding more
guidance on top of it is not.
What does this PR change?
Rewrites
skills/brainstorming/SKILL.md's entry decision into a three-pathrouter — spike / bounded / architectural — that scales the artifact while
keeping human approval on every path. The HARD-GATE keeps "no implementation
before approval" and drops "regardless of perceived simplicity" as the ceremony
driver; the anti-pattern section is reframed around skipping approval rather
than skipping documents; the checklist splits into three per-path checklists;
the process graph gains the router; and seven Red Flags rows are added targeting
classification-as-escape-hatch.
Is this change appropriate for the core library?
Yes, and it is hard to think of a change more core:
brainstormingis theskill every Superpowers user hits before any creative work, on every harness.
The router is written in general terms (a feasibility question, a well-scoped
change to existing code, a new subsystem) with no language, framework, domain or
tool assumptions, adds no dependency, and promotes nothing. Its cross-harness
regression battery below exists specifically to prove it is not a
Codex-flavoured fix smuggled into shared content.
What alternatives did you consider?
Three of these were measured, not argued.
Measured and rejected: that variant ("A-current" in the micro) is the only
variant that fails to differentiate. It still drifts the bounded brief to FULL
3/5 even under a taxonomy that asks purely about artifacts.
perfectly — and ships no approval gate at all. Rejected on the audit's own
evidence: the artifact ceremony is what was wasted; the approval gate is what
caught unexamined assumptions. The whole design premise here is ceremony
scales, approval never does. Z-null was also never live-tested. (See the
disclosure below — Z-null's micro performance is the one uncomfortable result
in this PR, and we are not hiding it.)
Finding 4's worst case is a feasibility question that got a repository and a
committed design before the feasibility was answered. That class needs its own
path, or it gets classified "architectural" and repeats the failure.
Measured and rejected. Round 1 of the Codex ceremony battery shipped
wording ("Get approval — explicit, before any implementation") that a capable
model read as satisfiable after the fact: 2 of 3 bounded reps implemented
first and treated the approval message as a post-hoc rubber stamp. Commit
6faceb2turns it into an explicit STOP; round 2 is 3/3.original bullet said "a well-scoped change to an existing, understood flow",
and Claude read "understood" as I understand this genre of app: on the
repo's own canonical acceptance prompt, against a genuinely empty repo, it
classified "bounded" 3/3 — one rep explicitly noticing the directory was empty
and classifying bounded anyway. Jesse ruled: tighten the wording and re-run.
Commit
70120d5makes bounded measure the repo ("If there is no existing flowto change, the task is not bounded") and adds the Red Flags row "Bounded
measures the repo, not your familiarity."
Does this PR contain multiple unrelated changes?
No. One change — the entry-decision router — landed in three commits because two
live batteries falsified two of its parts and sent us back:
5ea8821ships the three-path router with its guards.6faceb2makes the bounded path's approval a hard stop, after the Codexceremony battery measured 1/3 strict compliance.
70120d5makes "bounded" measure the repo rather than the agent'sfamiliarity, after the triggering acceptance check measured Claude
classifying a blank-repo new project as bounded 3/3.
Every one of the touched sections (HARD-GATE, anti-pattern, checklist, graph,
Red Flags, terminal states, "After the Design") is a facet of the same router;
splitting them would ship a skill that routes three ways but whose checklist and
graph still describe one path.
Existing PRs
Searched
gh pr list --state allfor brainstorming, ceremony, proportional,spike/bounded/architectural and approval-gate terms, plus the 25 most recent PRs
of any state, on 2026-07-31. The searches for "spike bounded architectural"
and "ceremony scales proportional" both returned zero results — no PR, open or
closed, proposes a proportional entry router for
brainstorming.skills/brainstorming/SKILL.mdand adding adesign-reviewer-prompt.md. Itadds a review step inside the existing single path; this PR changes which
path you enter. Opposite directions on ceremony.
fixes (resolvable paths, orphan prompts, exclusions, description
triggerability) — no overlap with the entry decision.
proportionality instinct applied to a different skill. Adjacent, not
duplicative; it does not touch
brainstorming.Deliberately not adopted — Jesse's design spec for this cycle names them
as evidence, not adopted text. They do not touch
brainstormingat all;listed because they come from the same underlying session corpus, so a
reviewer should know the two efforts are independent.
commit behavior, design-section ordering, and parallel-track proposals
respectively. All are downstream-of-classification changes and none conflicts
textually with the router.
Environment tested
codex-cli 0.146.0codex_subpins no model)gpt-5.6-sol; childrengpt-5.6-terra/gpt-5.6-solcodex-cli 0.146.0openai_responsescredentialgpt-5.5opuscredentialclaude-opus-4-8gemini_default(pins no model)gemini-3.5-flash(self-reported)claude-opus-4-8New harness support (required if this PR adds a new harness)
Not applicable — this PR adds no harness. But it changes the skill that this
repo's own harness acceptance test measures, so we ran that exact test anyway,
on all three harnesses, as a regression guard. The results are below.
Acceptance check for "Let's make a react todo list" — verbatim excerpts, all three harnesses (NOT a full transcript; see note)
Evaluation
Initial prompt. Jesse asked for an audit of his own two-week Codex window
after repeatedly watching SDD runs balloon and after repeatedly having to tell
agents to stop doing process. The literal first message of that session is not
preserved in any committed artifact, so it is not quoted here; what is preserved
is the audit doc cited above, the ten-experiment eval campaign that tested its
claims against evidence rather than narrative, and this cycle's approved spec
and plan (
docs/superpowers/specs/2026-07-30-codex-efficiency-fixes-design.md,docs/superpowers/plans/2026-07-30-codex-efficiency-fixes.md).Eval sessions run AFTER the change: 65 graded live reps across three
harnesses — 16 Codex ceremony reps, 36 cross-harness ceremony reps, 13 valid
triggering reps — plus a 75-call classification micro. Every battery was
pre-registered in an append-only hypothesis log
(
superpowers-autoresearch/logs/2026-07-30-codex-efficiency-fixes.md) —prediction, scorer and criterion written before the run, verdicts appended
after — and every scorer number was independently hand-verified from raw
trajectories (the campaign's standing non-circular-verification rule).
Layer 1 — classification micro (75 Anthropic API calls,
claude-opus-4-8)3 variants × 5 briefs × 5 reps. The tested variant is the shipped
## Three Pathstext, verified by script to be character-identical to the source blockafter whitespace collapse. Two of the five briefs are adversarial and new: a
--jsonflag whose current text output is parsed line-by-line by threedownstream scripts other teams run, and a one-file timezone fix whose rollup
boundary is computed independently in two other jobs. Both read as surface-bounded
and hide a cross-cutting dependency.
dev's text)Every pre-registered cell met its criterion, including the highest-stakes one:
arch never downgrades, 5/5, and both adversarial briefs escalate to FULL
5/5 under the "when in doubt, take the heavier one" ratchet. PASS.
Layer 2 — Codex ceremony battery (16 reps, two rounds)
cx-ceremony-{spike,bounded,arch}, real Codex agents, scored by census againstthe
dev-arm baseline (cited, not re-run: bounded and arch each wrote exactly 2docs before code in every rep).
devbaseline6faceb2)Round 1's headline is both halves of the story at once. The mechanical claim —
bounded produces no documents — is cleanly fixed: 0 docs in 3/3 reps against a
devarm that wrote 2 docs unconditionally in every rep. But hand-reading theraw event streams showed 2 of 3 bounded reps jumping straight from task receipt
into systematic-debugging + TDD and patching
tests/test_server.pyandserver.pybefore any approval message arrived. A model that reads "no specfile needed" as license to skip presenting a design, not just writing one to a
file, is a failure mode the layer-1 micro structurally could not detect — it
tests whether a model names the right path, not whether it enforces the gate
across a live multi-turn session.
Commit
6faceb2answers that with an explicit STOP ("Implementation starts onlyafter your human partner says yes to that design — a bounded task's approval is
as hard a gate as an architectural one") plus a Red Flags row. Round 2's three
bounded reps all show the identical shape, hand-verified against raw timestamps:
design in chat → approval message → the agent's own "Approved…" acknowledgment →
then the first patch, by margins of 15 s, 32 s and 21 s.
Ceremony discrimination in round 1: spike 5.0 mean tool calls before first code,
bounded 4.3, arch 16.7 — a 0.30 ratio, i.e. ceremony scales with complexity,
against a
devbaseline where the same gate was uncomputable.Layer 3 — cross-harness regression matrix (36 runs)
{dev, fix}×{Claude Code, Gemini}×{spike, bounded, arch}× 3 reps. Allfour pre-registered criteria met:
(harness, class) pairs: 5 tied, 1 improved.
1,1,1 and
writing_plans_invokedtrue 3/3.and
writing_plans_invokedfalse 3/3.is where the interesting result is.
Layer 4 — triggering acceptance check (all three harnesses)
Round 1: Gemini 3/3 on both clauses; Claude 3/3 on "brainstorming loads before
any code" but 0/3 on architectural classification; Codex blocked on
subscription credit exhaustion. Jesse ruled: tighten the bounded wording and
provision an API key. Round 2, after commit
70120d5: 7/7 valid reps PASSboth clauses — Codex 3/3, Claude 3/3 (a full reversal), Gemini smoke 1/1. Two
of three Claude reps' own words echo the new text almost verbatim ("no existing
flow in the repo to modify" vs the skill's "no existing flow to change"), and
Gemini's reasoning trace paraphrases the new Red Flags row directly
("Boundedness relates to the repo's content, not my familiarity"). Verbatim
excerpts are in the collapsed section above.
How outcomes changed, summarised
dev)6faceb270120d5Cost: $177.43 measured — layer 2 $15.23 + $16.87, layer 3 $141.04, layer 4
$2.27 + $2.02 — read from each rep's own economics block.
Disclosures that travel with this PASS, not buried under it
1. The micro cannot show that the router text is what works. Z-null (no
router text at all, only the neutralized one-line definitions present in every
condition's shared system prompt) produces identical tallies to the shipped
text on all five briefs — 5/0/0, 0/5/0, 0/0/5, 0/0/5, 0/0/5, both variants,
every column. Under this campaign's standing discrimination rule, the narrower
claim "the router text, as opposed to the definitions alone, produces correct
classification" is inconclusive-by-zero on this battery. Nothing
pre-registered required beating Z-null, and only A-current was framed as the
contrast — which it does lose, on
bounded, the one cell where the currentabsolute wording still pulls weight. Two live explanations remain open: the
briefs may not be hard enough to separate "no guidance" from "shipped text", or
the definitions alone may suffice for a model this capable. The load-bearing
evidence for this PR is therefore layers 2–4, the live batteries, not the
micro. This reproduces the original campaign micro's own finding, and it is
flagged here rather than left for a reviewer to notice.
2. Claude's
devarm never had the bounded doc-ceremony pathology. Undercriterion (d), Gemini's dev arm reproduces the unconditional two-doc ritual
exactly (3/3 reps, 1 spec + 1 plan, writing-plans invoked) — matching the
original Codex baseline. Claude's dev arm does not:
dev/claude/boundedshows zero ceremony docs on all 3 reps, identical in shape to its own fix arm.
So for Claude specifically the before/after doc delta on bounded is 0 → 0, and
this battery's cross-harness evidence for the fix's marginal contribution on
that class is honestly a Gemini before/after story (3/3 → 0/3). Claude's
result should be read as "confirmed clean on both arms", not "measurably
fixed". Criterion (d) was pre-registered as recorded-not-gated precisely so this
could be reported rather than spun.
3. A pre-existing Claude + spike pathology, present on both arms. Of Claude's
6 spike reps, only 1 passed cleanly. Three produced zero tool calls —
answering a port-in-use question fluently in Node/TypeScript/Bun terms without
ever opening the Python service the story is about — and two investigated an
unrelated TypeScript file belonging to the eval harness itself, visible on
disk beside the scenario tree. Symmetric across arms (dev 0/3 pass, fix 1/3), so
not attributable to this change; Gemini shows none of it (6/6 clean). Flagged
as a caveat about that scenario and that harness, not as a T4 result.
4. The arch completion 2/3 → 3/3 improvement rode a disclosed harness
accommodation, and its margin is razor thin. Round 1's arch rep3 hit the
scenario's own 30-minute wall-clock budget while running a legitimate
spec→plan→subagent→review→fix→re-review cycle. Round 2 raised that budget to 45
minutes — a scenario-harness change made on the host scenario file and
pre-registered as such, not a change to any text under test. It worked, but
round 2's own arch rep4 (which independently chose the same 11-child SDD path)
finished with roughly 65 seconds of headroom out of 45 minutes (2.4%). A
slightly slower model or network day reproduces round 1's timeout at the new
ceiling. Treat that margin as a live risk for anyone reusing this scenario, not
a closed question.
5. Layer 3 overran its own budget estimate by $61–101 ($141.04 against a
pre-registered $40–80), driven entirely by Gemini's arch cells — one dev rep
alone cost $26.38 on 8.8M+ tokens. The brief's assumption that "Claude/Gemini
runs are the cheap side" held for Claude and was wrong for Gemini on this
scenario class. Disclosed rather than smoothed over.
6. Two operational anomalies during layer 4, both self-inflicted, both
disclosed. A hand-written wait loop in our own driver matched a stale sibling
verdict.jsonfrom an earlier round and tore down the container a Codex rep wasstill running in; that rep was destroyed and redone as a distinctly-numbered
replacement, with the orphaned partial run left on disk rather than hidden. And
round 1's first claude rep ran against a scenario whose setup cloned a shared
template repo instead of creating a blank one — caught by the required
hand-verification before it could contaminate the matrix, fixed, and that rep
excluded from the tally (its spend is still reported).
7. Three of layer 3's 36 reps have no cost figure — the zero-tool-call
Claude spike reps wrote no economics block. Reported as unmeasured, not
estimated at zero.
Rigor
superpowers:writing-skillsandcompleted adversarial pressure testing (paste results below)
rationalizations, "human partner" language) without extensive evals
showing the change is an improvement
On the first box, stated plainly rather than checked loosely: the
superpowers:writing-skillsskill was not invoked by name in the implementersessions that wrote this text. No record of it appears in the task reports or
the hypothesis log, and we are not going to claim a skill invocation we cannot
evidence — on a PR whose entire argument is that measurement beats narrative,
that would be the worst possible corner to cut. The cycle's approved design spec
does say skill-text changes follow writing-skills methodology, and what actually
shipped is the heavier end of it: a literal-text classification micro before any
live spend (writing-skills' own "micro-test wording before full scenarios"
pattern), then four pre-registered live layers, two of which failed and produced
commits 2 and 3.
Adversarial pressure testing — results. The adversarial work is spread
through the layers above rather than run as a separate battery; the specific
adversarial results are:
hiding a public-interface change and a cross-cutting dependency. The shipped
text escalates both to FULL 5/5. This is the single highest-risk failure
mode of a proportional router — an escape hatch — and it does not fire.
all three harnesses live. A router that made architectural work cheaper would
be a regression; it does not.
bounded reps skipping the approval turn. Fixed and re-measured to 3/3.
genre familiarity on a blank repo. Fixed and re-measured to 3/3 on all three
harnesses.
On carefully-tuned content, explicitly. This PR does modify behavior-shaping
content — that is the change. The HARD-GATE keeps its prohibition and loses only
the clause the micro identified as the ceremony driver; the anti-pattern section
is retargeted from documents to approval; the "human partner" framing is
preserved and extended (the router's classification is said out loud
specifically so your human partner can override it). Seven Red Flags rows are
added; no existing row is deleted or reworded. Four of the seven exist
because a live rep produced that exact rationalization.
Human review
(opened at Jesse's direction; the maintainer's PR review on this PR is the complete-diff review)
Do not open this PR until the box above is checked. Jesse Vincent reviews the
complete diff (
git diff origin/dev...fix/t4-brainstorming-three-paths, 1 file,+106/−7) first; this body is the draft prepared for that review. He has already
made two rulings that shaped the branch — the bounded-approval hard stop after
layer 2 round 1, and the bounded-wording tightening plus codex API-key
provisioning after layer 4 round 1 — both recorded in the hypothesis log.