Skip to content

fix(skills): require behavioral RED, not structural import failure - #2067

Open
arimu1 wants to merge 1 commit into
obra:devfrom
arimu1:fix/2046-writing-plans-structural-red
Open

fix(skills): require behavioral RED, not structural import failure#2067
arimu1 wants to merge 1 commit into
obra:devfrom
arimu1:fix/2046-writing-plans-structural-red

Conversation

@arimu1

@arimu1 arimu1 commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

This PR MUST target the dev branch, not main. main is the
released branch; active work lands on dev first. PRs opened against
main will be asked to retarget dev before review.

Who is submitting this PR? (required)

Field Value
Your model + version Grok 4.5
Harness + version Grok Build / Cursor
All plugins installed Superpowers skill pack in use for contribution workflow; no third-party skill plugins installed for authoring this change. Skill text edits hand-written against issue #2046 (not auto-generated skill rewrite).
Human partner who reviewed this diff arimu1

What problem are you trying to solve?

Real session reported in #2046 (human partner + Codex harness, Superpowers 6.2.0).

writing-plans generated a Python TDD plan whose RED step expected failure because bell_state.py did not exist yet, and treated that as proof the probability behavior test could detect a missing/wrong implementation.

Executor ran the test and got:

ModuleNotFoundError: No module named 'bell_state'

Import failed before test_bell_state_probabilities ran, so the numerical assertion never executed. The agent still described the output as proof the test was effective. The human correctly challenged that claim (import error ≠ proof the quantum probability assertion rejects bad results).

Root instruction mismatch:

  • skills/writing-plans/SKILL.md task template said: Expected: FAIL with "function not defined" — agents read that as structural absence = valid RED evidence.
  • skills/test-driven-development/SKILL.md already said RED must be a test failure, "not errors," and to fix errors and re-run until the test fails correctly — but did not spell out that import/module/load failures are not behavior-assertion evidence.

Related history (#711, #853, #1576) does not draw the structural-vs-behavioral RED distinction for this failure mode.

What does this PR change?

  1. writing-plans task template Step 2: expect a failing behavior assertion (actual ≠ expected), not load/import failure. Add explicit Behavioral RED vs Structural RED rules and the recovery path (stub/negative control → re-run to behavioral RED) before implementing real logic.
  2. test-driven-development Verify RED: name import/module/syntax/collection/setup as structural errors that must be fixed before RED counts; require an importable negative control under the assertion.

Is this change appropriate for the core library?

Yes. This is general-purpose TDD plan guidance for any language/project. It does not add domain-specific content, third-party tools, or harness-specific integrations. Anyone using writing-plans + TDD benefits from not mistaking ModuleNotFoundError for proof that a behavior check works.

What alternatives did you consider?

  1. Change only writing-plans — Leaves TDD Verify RED short enough that agents could still claim "feature missing" for ImportError. Rejected: the issue is an instruction inconsistency between the two skills.
  2. Only soft prose reminder ("remember tests should fail correctly") without template change — Rejected: the template's Expected: FAIL with "function not defined" is the concrete phrase agents copy into plans; it must change.
  3. Ban top-level imports / require late import in every test — Rejected: over-prescriptive and language-specific; the fix is evidence labels + recovery path, not a new import style mandate.
  4. Larger rewrite of RED-GREEN sections — Rejected: high rejection risk for carefully-tuned skill content; keep the minimal wording that blocks the observed failure mode.

Does this PR contain multiple unrelated changes?

No. Both file edits address one failure mode: structural RED misread as behavioral RED evidence. writing-plans is the plan template agents copy; TDD is the authoritative RED verification policy they must align with.

Existing PRs

Searched open and closed PRs for: 2046, structural RED, ModuleNotFoundError, writing-plans RED, behavior assertion. Nearby writing-plans PRs (e.g. #1831 invariants, #1627 evaluator section, #1813 commit style, #2016 plan commit) do not address this distinction. Related issues #711, #853, #1576 noted in #2046 do not land this wording fix.

Environment tested

Harness (e.g. Claude Code, Cursor) Harness version Model Model version/ID
Grok Build / Cursor Cursor IDE (current) Grok 4.5 Grok 4.5

Skill-content-only change; no runtime harness integration.

New harness support (required if this PR adds a new harness)

N/A — this PR does not add a new harness.

Clean-session transcript for "Let's make a react todo list"
N/A — not a new harness PR

Evaluation

  • Initial prompt / source of truth: Issue writing-plans can mistake structural RED for behavior-test evidence #2046 session (Codex + Superpowers writing-plans on a Bell-state lesson). Plan RED said missing bell_state.py "proves the test can detect the missing implementation"; human partner correctly rejected that after ModuleNotFoundError.
  • Before (template + observed agent behavior):
    • Template: Expected: FAIL with "function not defined"
    • Agents emit plans claiming structural absence proves the behavior test works
    • Executors accept ImportError/ModuleNotFoundError as RED evidence for numerical/semantic assertions that never ran
  • After (this PR wording):
    • Template requires AssertionError / actual-vs-expected mismatch for RED when claiming the test has teeth
    • Structural failures are named and forbidden as behavior-test evidence
    • Explicit recovery: stub or negative-control value → re-run to behavioral RED
    • TDD Verify RED matches: structural errors → fix and re-run; negative control under assertion
  • Adversarial / pressure checks (reasoning + diff review against issue transcript; skill text change only):
    1. Exact writing-plans can mistake structural RED for behavior-test evidence #2046 case: Plan step that said "failure because file does not exist proves test works" is now forbidden by Structural RED rule; expected line requires assertion failure.
    2. "function not defined" NameError after import succeeds: Still structural (missing symbol / error before assertion) under the new labels unless an assertion actually runs — agent must stub and get assertion failure. Matches "fails not errors."
    3. True behavioral RED: Wrong return value rejected by assert — still valid; template example is AssertionError path.
    4. Technology exception: "unless the plan explicitly states why behavioral RED is impossible" preserves rare cases (e.g. pure packaging tasks) without reopening the common Python greenfield loophole.
    5. Regression risk to careful TDD language: Did not rewrite Red Flags table, rationalizations table, or "human partner" phrasing; only extended Verify RED + writing-plans Step 2.
  • Multi-session live harness evals: Not run in this contribution environment (skill markdown only). The change is the minimal instruction fix requested by writing-plans can mistake structural RED for behavior-test evidence #2046; before/after multi-session evals remain welcome as follow-up if maintainers want quantified behavior rates.

Rigor

  • If this is a skills change: I used judgment aligned with superpowers skill philosophy (precise labels, forbid the observed rationalization, keep carefully-tuned sections intact). Full writing-skills multi-subagent campaign was not run in this harness; adversarial cases above map the writing-plans can mistake structural RED for behavior-test evidence #2046 failure and adjacent loopholes.
  • This change was tested adversarially, not just on the happy path (structural NameError, true AssertionError, technology exception, template copy path)
  • I did not modify carefully-tuned content (Red Flags table, rationalizations, "human partner" language) without extensive evals — those sections were left unchanged

Human review

  • A human has reviewed the COMPLETE proposed diff before submission

Fixes #2046

writing-plans task template treated "function not defined" / missing-module
errors as valid RED. That never runs the behavior assertion, so agents
claimed tests had teeth when only ImportError/ModuleNotFoundError ran.

Clarify structural vs behavioral RED in writing-plans and align TDD
Verify RED: load/setup errors are not proof the assertion works.

Fixes obra#2046
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant