From 6b0a0c56619fd9d6664cc6befc5211b86ada35cb Mon Sep 17 00:00:00 2001 From: Paul Keen <125715+pftg@users.noreply.github.com> Date: Sat, 22 Aug 2026 20:09:17 +0200 Subject: [PATCH] Fault-inject both directions: the rule was half-written Kaizen pass over today. The dominant failure was not slop or wrong numbers - it was building an instrument, trusting its verdict, and nearly changing correct work. Four times: case-study marker accused our own real jt_site case study generalized-actor rule fired on "anyone on the team could remember" phrase marker flagged a quoted exhibit inside a ```diff as our claim single-line img grep reported a working diagram as not rendering One rule already covers half of this - "a new test isn't done until you've BROKEN THE CODE and watched it fail". It is one-sided. It says inject the defect and watch it fail; it never says inject a legitimate LOOKALIKE and watch it pass. All four misses were that missing half. Smallest change that prevents the class: one clause on the rule that already exists, rather than a new rule or a new tool. Verified rather than asserted, on the ratchet I modified an hour ago: legitimate exhibit (quoted bad writing inside a fence) -> 0 failures PASS the real claim (same phrase, in prose) -> Expected 10 to be <= 9 FAIL Both directions confirmed on a throwaway probe post, which was then deleted; suite back to 5 runs / 13 assertions / 0 failures. That is the first gate today tested in both directions, and it is the one that had a false positive this afternoon. The operational half of the clause is the part that would have saved the time: **when a gate accuses something, reproduce the accusation by hand before believing it.** Three of the four above survived because the verdict looked authoritative and I acted on it instead of checking it - the same shape as trusting a green run, inverted. Kept deliberately small. No new script, no new gate, no framework - one clause on an existing BLOCKING rule, with the four cases named so the next reader knows it is describing something that happened rather than a hypothetical. Gates: marketing_copy_test 5/13/0. bin/hugo-build green. Instruction-layer diff. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg --- CLAUDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/CLAUDE.md b/CLAUDE.md index e0fd293cd..a31fcf386 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -60,6 +60,7 @@ Operational knowledge lives in `.okf/` (`build/`, `content/`, `design/`, `workfl - **TDD**: RED → GREEN → REFACTOR. Doctrine: `docs/20-29-testing-qa/` + `docs/incidents/25.0x`. - **Behavior-focused tests ONLY** — reject implementation/existence/config tests. Don't hardcode tunables (`q=90`, exact sizes); assert the shape (`q=\d+`, has ``). A test that breaks on a knob change is testing config — relax it. +- **Fault-inject BOTH directions, or the gate is half-tested (BLOCKING, extended 2026-08-22).** Plant the defect and watch it FAIL, then plant a legitimate lookalike and watch it PASS. The second half was missing and cost four false positives in one day: a marker flagged our own real `jt_site` case study as fabrication, an AI-tell rule fired on "anyone on **the team** could remember", a phrase marker flagged a quoted exhibit inside a ```diff fence as our own claim, and a single-line `` grep reported a working diagram as missing because Hugo wraps the tag. Each time the instrument's verdict was trusted and correct work was nearly changed. **When a gate accuses something, reproduce the accusation by hand before believing it.** - **A new test isn't done until you've BROKEN THE CODE and watched it fail (BLOCKING).** Green proves nothing on its own — a test that passes because it asserts nothing passes forever. So: inject the exact defect the test exists to catch, quote the failure message in your handback, then revert and re-run green. Both halves are the evidence. Where a gate can't discriminate yet, say so in the test's own comment rather than pretending. Measured 2026-08-22 (`docs/20-29-testing-qa/20.11`): fault-injecting 8 realistic defects caught only 3 — the testimonial gate passed a *vanished* blockquote (`""` is a substring of everything), the rendered ratchet carried 3 hits of slack that swallowed a planted banned phrase, and the link job excluded 90% of links while reporting green for a year. - **Framework**: Minitest (`test/system/`, `test/unit/`). Test runner matrix: the header **Test** line — don't restate it. - **Visual regression + gates**: full matrix, tolerances, baseline procedure → `.okf/build/test-gates.md` (canonical). Tolerance 0.0 for refactoring, ≤0.03 for new features.