fix(skills): require behavioral RED, not structural import failure - #2067
Open
arimu1 wants to merge 1 commit into
Open
fix(skills): require behavioral RED, not structural import failure#2067arimu1 wants to merge 1 commit into
arimu1 wants to merge 1 commit into
Conversation
writing-plans task template treated "function not defined" / missing-module errors as valid RED. That never runs the behavior assertion, so agents claimed tests had teeth when only ImportError/ModuleNotFoundError ran. Clarify structural vs behavioral RED in writing-plans and align TDD Verify RED: load/setup errors are not proof the assertion works. Fixes obra#2046
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Who is submitting this PR? (required)
What problem are you trying to solve?
Real session reported in #2046 (human partner + Codex harness, Superpowers 6.2.0).
writing-plansgenerated a Python TDD plan whose RED step expected failure becausebell_state.pydid not exist yet, and treated that as proof the probability behavior test could detect a missing/wrong implementation.Executor ran the test and got:
Import failed before
test_bell_state_probabilitiesran, so the numerical assertion never executed. The agent still described the output as proof the test was effective. The human correctly challenged that claim (import error ≠ proof the quantum probability assertion rejects bad results).Root instruction mismatch:
skills/writing-plans/SKILL.mdtask template said:Expected: FAIL with "function not defined"— agents read that as structural absence = valid RED evidence.skills/test-driven-development/SKILL.mdalready said RED must be a test failure, "not errors," and to fix errors and re-run until the test fails correctly — but did not spell out that import/module/load failures are not behavior-assertion evidence.Related history (#711, #853, #1576) does not draw the structural-vs-behavioral RED distinction for this failure mode.
What does this PR change?
Is this change appropriate for the core library?
Yes. This is general-purpose TDD plan guidance for any language/project. It does not add domain-specific content, third-party tools, or harness-specific integrations. Anyone using writing-plans + TDD benefits from not mistaking ModuleNotFoundError for proof that a behavior check works.
What alternatives did you consider?
Expected: FAIL with "function not defined"is the concrete phrase agents copy into plans; it must change.Does this PR contain multiple unrelated changes?
No. Both file edits address one failure mode: structural RED misread as behavioral RED evidence. writing-plans is the plan template agents copy; TDD is the authoritative RED verification policy they must align with.
Existing PRs
Searched open and closed PRs for:
2046,structural RED,ModuleNotFoundError,writing-plans RED,behavior assertion. Nearby writing-plans PRs (e.g. #1831 invariants, #1627 evaluator section, #1813 commit style, #2016 plan commit) do not address this distinction. Related issues #711, #853, #1576 noted in #2046 do not land this wording fix.Environment tested
Skill-content-only change; no runtime harness integration.
New harness support (required if this PR adds a new harness)
N/A — this PR does not add a new harness.
Clean-session transcript for "Let's make a react todo list"
Evaluation
bell_state.py"proves the test can detect the missing implementation"; human partner correctly rejected that afterModuleNotFoundError.Expected: FAIL with "function not defined"Rigor
Human review
Fixes #2046