feat(guard): a sorry cannot exempt itself by writing a comment (#385) - #396
Merged
Conversation
REQ-GUARD-GATE-EVIDENCE-002 (f).
`Fail on sorry (post-build gate)` is part of the REQUIRED `Lean proof typecheck
(lake build)` context. It permitted every sorry in the tree while printing
No sorrys in proofs/Proofs/ — gate green.
Its pipeline excluded any sorry carrying a same-line `-- TODO`, and all twelve
are `sorry -- TODO(v1.0.0)`. Twelve in, zero out, the `if` never fires. The gate
was satisfied by exactly the thing it exists to prevent.
HOW IT DRIFTED, which matters more than the filter.
The step's own comment said:
# The five tracked sorrys do NOT carry a same-line comment; the
# TODO sits on the prior line, so this gate flags all five.
That was TRUE when written. The exclusion was a deliberate no-op — a defensive
filter against a form nobody used. The sorries then grew 5 -> 12 and moved their
TODO onto the same line, and the no-op silently became universal. Nobody edited
the guard. The code drifted out from under its comment.
So the defect is not a careless filter. It is that A SELF-EXEMPTION KEYED ON A
PATTERN THE EXEMPTED CODE ITSELF CONTROLS CANNOT HOLD: the proof author writes
both the `sorry` and the `-- TODO` that excuses it, in the same commit, and no
review step sees the exemption grow. An exemption has to be declared somewhere
the exempted party does not edit.
THE REPLACEMENT is a ratchet, the same shape as the mutants gate. Count every
bare sorry with NO comment-based exemption — a `-- TODO` is a note to humans,
not a permit — and compare against `--max-sorries`, declared in proofs.yml
rather than in the proofs.
count > floor FAIL. New sorries cannot be admitted by writing a comment.
count < floor PASS, and say loudly to lower the floor. Improving must never
fail, or the ratchet gets walked back instead of tightened.
empty scan exit 2. Zero sorries because no .lean file was read is the
error-path-yields-ideal-reading shape this requirement names.
The count and a per-file, per-line breakdown print on EVERY path including
success, so "12 permitted" is visible rather than inferred from silence.
MEASURED on ae27ff1: 11 .lean files scanned, 12 sorries (MinPlus.lean 4,
MinPlusPwa.lean 8). Floor set to 12 — green and honest rather than green and
blind. Verified it can go red: at `--max-sorries 11` it exits 1 naming the
breach.
Self-test, 8 cases, runs before the gate judges anything. The discriminating
pair is the regression: `sorry -- TODO` COUNTS and breaches a floor of 0, and
the SAME file passes at a floor of 1 — the comment is signal for neither
verdict. A checker still honouring the exemption passes the first case and
cannot fail it. Also pinned: `sorryAx` and the word "sorry" in prose are not
bare sorries, so the count cannot be inflated by unrelated text.
NOT claimed: that the 12 remaining sorries are acceptable. The floor records how
many exist, not that they are fine. Discharging them is REQ-PROOF-NC-MINPLUS-001
(v0.38.0). This only stops the number rising unnoticed — which is precisely what
it did between 5 and 12.
Refs #385
Rivet verification gate✅ 20/20 passed
Filter: Failed artifacts(none) Updated automatically by |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
#395 (obligations g+h) and this branch (f) both appended a TEST-GUARD-* artifact at the same insertion point in artifacts/verification.yaml, so the three-way merge conflicted on adjacency alone. Both sides kept; nothing was edited or dropped. Verified on the MERGED tree, not on either parent: 168 artifacts, no duplicate ids, all three new entries present, check_release_plane / check_human_scoped / check_required_contexts green, and the four guard self-tests (lean_sorries 8, clippy_workspaces 3, fuzz_targets 7, fmt_workspaces) all pass together. Live gates on the merged tree: "All 3 declared fuzz targets are run" and "12 sorries, at or under the declared floor of 12". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REQ-GUARD-GATE-EVIDENCE-002 (f) — v0.36.0 bug-fix track. Written during the GitHub Actions outage, so CI here will be red for infrastructure reasons until the incident clears; see #395 for the same situation.
The defect
Fail on sorry (post-build gate)is part of the requiredLean proof typecheck (lake build)context. It permitted every sorry in the tree while printing:It excluded any sorry with a same-line
-- TODO, and all twelve aresorry -- TODO(v1.0.0). Twelve in, zero out, theifnever fires. The gate was satisfied by exactly the thing it exists to prevent.How it drifted — the part worth keeping
The step's own comment said:
That was true when written. The exclusion was a deliberate no-op against a form nobody used. The sorries then grew 5 → 12 and moved their TODO onto the same line — and the no-op silently became universal. Nobody edited the guard; the code drifted out from under its comment.
So the defect isn't a careless filter. It's that a self-exemption keyed on a pattern the exempted code itself controls cannot hold — the proof author writes both the
sorryand the-- TODOthat excuses it, in one commit, and no review step sees the exemption grow.The replacement
A ratchet, same shape as the mutants gate. No comment-based exemption at all — a
-- TODOis a note to humans, not a permit — and the floor lives inproofs.yml, not in the proofs..leanwas read is the error-path-yields-ideal-reading shape this requirement namesCount plus per-file, per-line breakdown prints on every path including success, so "12 permitted" is visible rather than inferred from silence.
Measured
On
ae27ff1: 11.leanfiles scanned, 12 sorries (MinPlus.lean4,MinPlusPwa.lean8). Floor set to 12 — green and honest rather than green and blind. Verified it can go red: at--max-sorries 11it exits 1 naming the breach.8-case self-test runs before the gate judges anything. The discriminating pair is the regression itself:
sorry -- TODOcounts and breaches a floor of 0, and the same file passes at a floor of 1 — the comment is signal for neither verdict. A checker still honouring the exemption passes the first case and cannot fail it. Also pinned:sorryAxand "sorry" in prose are not bare sorries.Not claimed
That the 12 remaining sorries are acceptable. The floor records how many exist, not that they're fine — discharging them is REQ-PROOF-NC-MINPLUS-001 (v0.38.0). This only stops the number rising unnoticed, which is exactly what it did between 5 and 12.
Refs #385
🤖 Generated with Claude Code