Fault-inject both directions: the rule was half-written - #609
Merged
Conversation
Kaizen pass over today. The dominant failure was not slop or wrong numbers - it was building an instrument, trusting its verdict, and nearly changing correct work. Four times: case-study marker accused our own real jt_site case study generalized-actor rule fired on "anyone on the team could remember" phrase marker flagged a quoted exhibit inside a ```diff as our claim single-line img grep reported a working diagram as not rendering One rule already covers half of this - "a new test isn't done until you've BROKEN THE CODE and watched it fail". It is one-sided. It says inject the defect and watch it fail; it never says inject a legitimate LOOKALIKE and watch it pass. All four misses were that missing half. Smallest change that prevents the class: one clause on the rule that already exists, rather than a new rule or a new tool. Verified rather than asserted, on the ratchet I modified an hour ago: legitimate exhibit (quoted bad writing inside a fence) -> 0 failures PASS the real claim (same phrase, in prose) -> Expected 10 to be <= 9 FAIL Both directions confirmed on a throwaway probe post, which was then deleted; suite back to 5 runs / 13 assertions / 0 failures. That is the first gate today tested in both directions, and it is the one that had a false positive this afternoon. The operational half of the clause is the part that would have saved the time: **when a gate accuses something, reproduce the accusation by hand before believing it.** Three of the four above survived because the verdict looked authoritative and I acted on it instead of checking it - the same shape as trusting a green run, inverted. Kept deliberately small. No new script, no new gate, no framework - one clause on an existing BLOCKING rule, with the four cases named so the next reader knows it is describing something that happened rather than a hypothetical. Gates: marketing_copy_test 5/13/0. bin/hugo-build green. Instruction-layer diff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg
Contributor
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fault-inject both directions: the rule was half-written
Kaizen pass over today. The dominant failure was not slop or wrong numbers - it
was building an instrument, trusting its verdict, and nearly changing correct
work. Four times:
case-study marker accused our own real jt_site case study
generalized-actor rule fired on "anyone on the team could remember"
phrase marker flagged a quoted exhibit inside a ```diff as our claim
single-line img grep reported a working diagram as not rendering
One rule already covers half of this - "a new test isn't done until you've
BROKEN THE CODE and watched it fail". It is one-sided. It says inject the defect
and watch it fail; it never says inject a legitimate LOOKALIKE and watch it
pass. All four misses were that missing half.
Smallest change that prevents the class: one clause on the rule that already
exists, rather than a new rule or a new tool.
Verified rather than asserted, on the ratchet I modified an hour ago:
legitimate exhibit (quoted bad writing inside a fence) -> 0 failures PASS
the real claim (same phrase, in prose) -> Expected 10 to be <= 9 FAIL
Both directions confirmed on a throwaway probe post, which was then deleted;
suite back to 5 runs / 13 assertions / 0 failures. That is the first gate today
tested in both directions, and it is the one that had a false positive this
afternoon.
The operational half of the clause is the part that would have saved the time:
when a gate accuses something, reproduce the accusation by hand before
believing it. Three of the four above survived because the verdict looked
authoritative and I acted on it instead of checking it - the same shape as
trusting a green run, inverted.
Kept deliberately small. No new script, no new gate, no framework - one clause
on an existing BLOCKING rule, with the four cases named so the next reader knows
it is describing something that happened rather than a hypothetical.
Gates: marketing_copy_test 5/13/0. bin/hugo-build green. Instruction-layer diff.
Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg