feat(guard): a truncated cargo-mutants run cannot score (#389) - #401
Merged
Conversation
A cargo-mutants run killed part-way is INTERNALLY CONSISTENT, which is why -001's cross-check could not see it. Run 30563879100 records 210+603+793+2 == 1608 == total_mutants, every count agreeing with outcomes.json, while the log shows it had found 1737 and lost its worker trees 3h02m in. total_mutants is written incrementally, so the report is a truthful account of the 92% that ran. The failure is flattering, which is what makes it dangerous: a truncated run UNDER-reports survivors, so 210 sits under the 292 threshold and the gate goes green. A threshold ratcheted onto such a number is a ceiling set below the floor. end_time is the discriminator — null while running, populated on a clean finish, and the one field that does not degrade gracefully under truncation. Absence of outcomes.json is fatal here, stricter than _cross_check's tolerance, because "cannot prove it finished" must not read as "finished". Arming this was deliberately deferred until the cause was fixed; the issue said so before either half existed. The cause turned out to be smithy's tmpfiles policy sweeping cargo-mutants-*.tmp older than 2h, hourly, on the stated assumption that a running job keeps mtime fresh. That is false for cargo-mutants, which copies each worker's source tree once at start and never touches it again, so mtime is frozen while the job is alive — and the four workers dying within one second of each other was one sweep, not four faults. Age raised to 8h, above GitHub's 6h job ceiling, with the aggressive reclaim moved into the post-job hook (smithy PR #7, verified 12/12 rules and 12/12 hooks). Residual coupling noted in the docstring: 8h holds only while this repo's mutants job stays under ~7h. 24 self-test cases. The discriminating pair is the point — the same report, same counts, same cross-check, same under-threshold verdict, differing only in end_time, must produce different outcomes. Run 30563879100's exact numbers appear as a case in both forms. Mutation-tested, 5 mutants, all caught — but two survived the first pass and the reason is worth recording. Tolerating a missing outcomes.json and accepting a missing end_time KEY each still exit 1 via a fallback (a missing file raises on json.load; a missing key reads as None), so the verdict was over-determined and the cases could not tell which check fired. Fixed by pinning the DIAGNOSIS as well as the exit code — the same principle the gate rests on, applied to its own tests. REQ-GUARD-GATE-EVIDENCE-002 stays `proposed`. It is tagged human-scoped, and REQ-GUARD-HUMAN-SCOPED-001 caught an attempt to promote it to `implemented` while preparing this change. Whether six implementations mean the requirement is MET is the judgement that tag reserves for a human; promoting it means dropping the tag in the same diff. Closes #389. REQ-GUARD-GATE-EVIDENCE-002 (d). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Rivet verification gate✅ 20/20 passed
Filter: Failed artifacts(none) Updated automatically by |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #389. REQ-GUARD-GATE-EVIDENCE-002 obligation (d) — the last of six.
Unblocked by the runner-side fix applied on #389 earlier today (smithy PR #7).
The issue was explicit that the detector and the infrastructure fix had to land
together, and that arming (2) without (1) "turns a required context into an
infrastructure coin-flip". (1) now holds, so this is (2).
The defect
A cargo-mutants run killed part-way is internally consistent, which is
exactly why -001's cross-check obligation could not see it. Run 30563879100:
Every count agrees with
outcomes.json. Meanwhile the log shows cargo-mutantshad found 1737 mutants and lost its worker trees 3h02m in.
total_mutantsis written incrementally, so the report is a truthful account of the 92% that
ran — and truthful about the wrong population.
The failure is flattering, which is what makes it dangerous. A truncated run
under-reports survivors, so 210 sits comfortably under the 292 threshold and the
gate goes green. Any threshold ratcheted down onto such a number is a ceiling set
below the floor.
The discriminator
end_time— null while running, populated on a clean finish. It is the onefield that does not degrade gracefully under truncation.
Absence of
outcomes.jsonis fatal here, which is stricter than_cross_check(which tolerates absence and says so). That asymmetry isdeliberate: losing the cross-check costs a detector, but losing
end_timemeanscompletion cannot be established at all — and "cannot prove it finished" must not
read as "finished".
Why the root cause is worth reading
Not a disk-pressure hook, which is what the issue originally suspected — that
was a red herring, since
ACTIONS_RUNNER_HOOK_JOB_COMPLETEDonly runs betweenjobs. The actual reaper was smithy's tmpfiles policy:
resting on its own stated assumption — "a job that's still running keeps mtime
fresh". That is false for cargo-mutants specifically. It copies the source
tree into each worker's temp dir once, at start, and never touches those files
again, so their mtime is frozen while the job is very much alive. A 3h02m run
aged past 2h and the next hourly sweep deleted its sources. The four workers
dying "within one second of each other" was one sweep, not four faults.
Age is now 8h, above GitHub's 6h job ceiling. Residual coupling, documented
in the docstring: that only holds while this repo's
mutantsjob stays under~7h; it is currently
timeout-minutes: 240.Evidence
same report, same counts, same cross-check, same under-threshold verdict,
differing only in
end_time, must produce different outcomes. Run30563879100's exact numbers appear as a case in both truncated and completed
form.
and that is the part worth keeping. Tolerating a missing
outcomes.jsonandaccepting a missing
end_timekey each still exit 1 through a fallback (amissing file raises on
json.load; a missing key reads asNone). The verdictwas over-determined, so the cases could not tell which check fired. Fixed by
pinning the diagnosis as well as the exit code — the same principle the
gate rests on, turned on its own tests. Both are now caught by "exit 1
correct, but the diagnosis is wrong".
One thing deliberately NOT done
REQ-GUARD-GATE-EVIDENCE-002staysproposed. I moved it toimplementedwhilepreparing this change and
check_human_scoped.pyfailed the tree — theartifact carries the
human-scopedtag, and REQ-GUARD-HUMAN-SCOPED-001 exists tostop an agent promoting exactly this kind of thing. Whether six implementations
mean the requirement is met is the judgement that tag reserves for a human.
Promoting it means dropping the tag in the same diff, so the decision is one
explicit line rather than a status that drifted upward on its own.
NOT claimed: that the surviving-mutant threshold is right. This only ensures
the number being compared came from a complete run.
🤖 Generated with Claude Code