Skip to content

Fault-inject both directions: the rule was half-written #609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
pftg merged 1 commit into from
Aug 22, 2026
Merged

Fault-inject both directions: the rule was half-written #609

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CLAUDE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@ Operational knowledge lives in `.okf/` (`build/`, `content/`, `design/`, `workfl

- **TDD**: RED → GREEN → REFACTOR. Doctrine: `docs/20-29-testing-qa/` + `docs/incidents/25.0x`.
- **Behavior-focused tests ONLY** — reject implementation/existence/config tests. Don't hardcode tunables (`q=90`, exact sizes); assert the shape (`q=\d+`, has `<picture>`). A test that breaks on a knob change is testing config — relax it.
- **Fault-inject BOTH directions, or the gate is half-tested (BLOCKING, extended 2026-08-22).** Plant the defect and watch it FAIL, then plant a legitimate lookalike and watch it PASS. The second half was missing and cost four false positives in one day: a marker flagged our own real `jt_site` case study as fabrication, an AI-tell rule fired on "anyone on **the team** could remember", a phrase marker flagged a quoted exhibit inside a ```diff fence as our own claim, and a single-line `<img>` grep reported a working diagram as missing because Hugo wraps the tag. Each time the instrument's verdict was trusted and correct work was nearly changed. **When a gate accuses something, reproduce the accusation by hand before believing it.**
- **A new test isn't done until you've BROKEN THE CODE and watched it fail (BLOCKING).** Green proves nothing on its own — a test that passes because it asserts nothing passes forever. So: inject the exact defect the test exists to catch, quote the failure message in your handback, then revert and re-run green. Both halves are the evidence. Where a gate can't discriminate yet, say so in the test's own comment rather than pretending. Measured 2026-08-22 (`docs/20-29-testing-qa/20.11`): fault-injecting 8 realistic defects caught only 3 — the testimonial gate passed a *vanished* blockquote (`""` is a substring of everything), the rendered ratchet carried 3 hits of slack that swallowed a planted banned phrase, and the link job excluded 90% of links while reporting green for a year.
- **Framework**: Minitest (`test/system/`, `test/unit/`). Test runner matrix: the header **Test** line — don't restate it.
- **Visual regression + gates**: full matrix, tolerances, baseline procedure → `.okf/build/test-gates.md` (canonical). Tolerance 0.0 for refactoring, ≤0.03 for new features.
Expand Down