Skip to content

feat(guard): a truncated cargo-mutants run cannot score (#389) - #401

Merged
avrabe merged 1 commit into
mainfrom
feat/v036-mutants-truncation-389
Aug 7, 2026
Merged

feat(guard): a truncated cargo-mutants run cannot score (#389)#401
avrabe merged 1 commit into
mainfrom
feat/v036-mutants-truncation-389

Conversation

@avrabe

@avrabe avrabe commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Closes #389. REQ-GUARD-GATE-EVIDENCE-002 obligation (d) — the last of six.

Unblocked by the runner-side fix applied on #389 earlier today (smithy PR #7).
The issue was explicit that the detector and the infrastructure fix had to land
together, and that arming (2) without (1) "turns a required context into an
infrastructure coin-flip". (1) now holds, so this is (2).

The defect

A cargo-mutants run killed part-way is internally consistent, which is
exactly why -001's cross-check obligation could not see it. Run 30563879100:

missed 210 + caught 603 + unviable 793 + timeout 2  ==  1608  ==  total_mutants

Every count agrees with outcomes.json. Meanwhile the log shows cargo-mutants
had found 1737 mutants and lost its worker trees 3h02m in. total_mutants
is written incrementally, so the report is a truthful account of the 92% that
ran — and truthful about the wrong population.

The failure is flattering, which is what makes it dangerous. A truncated run
under-reports survivors, so 210 sits comfortably under the 292 threshold and the
gate goes green. Any threshold ratcheted down onto such a number is a ceiling set
below the floor.

The discriminator

end_time — null while running, populated on a clean finish. It is the one
field that does not degrade gracefully under truncation.

Absence of outcomes.json is fatal here, which is stricter than
_cross_check (which tolerates absence and says so). That asymmetry is
deliberate: losing the cross-check costs a detector, but losing end_time means
completion cannot be established at all — and "cannot prove it finished" must not
read as "finished".

Why the root cause is worth reading

Not a disk-pressure hook, which is what the issue originally suspected — that
was a red herring, since ACTIONS_RUNNER_HOOK_JOB_COMPLETED only runs between
jobs. The actual reaper was smithy's tmpfiles policy:

e /var/lib/runners/runnerN/_tmp/cargo-mutants-*.tmp - - - 2h -

resting on its own stated assumption — "a job that's still running keeps mtime
fresh"
. That is false for cargo-mutants specifically. It copies the source
tree into each worker's temp dir once, at start, and never touches those files
again, so their mtime is frozen while the job is very much alive. A 3h02m run
aged past 2h and the next hourly sweep deleted its sources. The four workers
dying "within one second of each other" was one sweep, not four faults.

Age is now 8h, above GitHub's 6h job ceiling. Residual coupling, documented
in the docstring: that only holds while this repo's mutants job stays under
~7h; it is currently timeout-minutes: 240.

Evidence

  • 24 self-test cases (was 19). The discriminating pair is the point: the
    same report, same counts, same cross-check, same under-threshold verdict,
    differing only in end_time, must produce different outcomes. Run
    30563879100's exact numbers appear as a case in both truncated and completed
    form.
  • Mutation-tested, 5 mutants, all caught — but two survived the first pass,
    and that is the part worth keeping. Tolerating a missing outcomes.json and
    accepting a missing end_time key each still exit 1 through a fallback (a
    missing file raises on json.load; a missing key reads as None). The verdict
    was over-determined, so the cases could not tell which check fired. Fixed by
    pinning the diagnosis as well as the exit code — the same principle the
    gate rests on, turned on its own tests. Both are now caught by "exit 1
    correct, but the diagnosis is wrong"
    .

One thing deliberately NOT done

REQ-GUARD-GATE-EVIDENCE-002 stays proposed. I moved it to implemented while
preparing this change and check_human_scoped.py failed the tree — the
artifact carries the human-scoped tag, and REQ-GUARD-HUMAN-SCOPED-001 exists to
stop an agent promoting exactly this kind of thing. Whether six implementations
mean the requirement is met is the judgement that tag reserves for a human.
Promoting it means dropping the tag in the same diff, so the decision is one
explicit line rather than a status that drifted upward on its own.

NOT claimed: that the surviving-mutant threshold is right. This only ensures
the number being compared came from a complete run.

🤖 Generated with Claude Code

A cargo-mutants run killed part-way is INTERNALLY CONSISTENT, which is why
-001's cross-check could not see it. Run 30563879100 records
210+603+793+2 == 1608 == total_mutants, every count agreeing with
outcomes.json, while the log shows it had found 1737 and lost its worker
trees 3h02m in. total_mutants is written incrementally, so the report is a
truthful account of the 92% that ran.

The failure is flattering, which is what makes it dangerous: a truncated
run UNDER-reports survivors, so 210 sits under the 292 threshold and the
gate goes green. A threshold ratcheted onto such a number is a ceiling set
below the floor.

end_time is the discriminator — null while running, populated on a clean
finish, and the one field that does not degrade gracefully under
truncation. Absence of outcomes.json is fatal here, stricter than
_cross_check's tolerance, because "cannot prove it finished" must not read
as "finished".

Arming this was deliberately deferred until the cause was fixed; the issue
said so before either half existed. The cause turned out to be smithy's
tmpfiles policy sweeping cargo-mutants-*.tmp older than 2h, hourly, on the
stated assumption that a running job keeps mtime fresh. That is false for
cargo-mutants, which copies each worker's source tree once at start and
never touches it again, so mtime is frozen while the job is alive — and
the four workers dying within one second of each other was one sweep, not
four faults. Age raised to 8h, above GitHub's 6h job ceiling, with the
aggressive reclaim moved into the post-job hook (smithy PR #7, verified
12/12 rules and 12/12 hooks). Residual coupling noted in the docstring:
8h holds only while this repo's mutants job stays under ~7h.

24 self-test cases. The discriminating pair is the point — the same
report, same counts, same cross-check, same under-threshold verdict,
differing only in end_time, must produce different outcomes. Run
30563879100's exact numbers appear as a case in both forms.

Mutation-tested, 5 mutants, all caught — but two survived the first pass
and the reason is worth recording. Tolerating a missing outcomes.json and
accepting a missing end_time KEY each still exit 1 via a fallback (a
missing file raises on json.load; a missing key reads as None), so the
verdict was over-determined and the cases could not tell which check
fired. Fixed by pinning the DIAGNOSIS as well as the exit code — the same
principle the gate rests on, applied to its own tests.

REQ-GUARD-GATE-EVIDENCE-002 stays `proposed`. It is tagged human-scoped,
and REQ-GUARD-HUMAN-SCOPED-001 caught an attempt to promote it to
`implemented` while preparing this change. Whether six implementations
mean the requirement is MET is the judgement that tag reserves for a
human; promoting it means dropping the tag in the same diff.

Closes #389. REQ-GUARD-GATE-EVIDENCE-002 (d).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 7, 2026

Copy link
Copy Markdown

Rivet verification gate

20/20 passed

count
Passed 20
Failed 0
Skipped (no steps) 0

Filter: (and (= type "feature") (or (has-tag "v093") (has-tag "v0100")))

Failed artifacts

(none)

Updated automatically by tools/post_verification_comment.py. Source of truth: artifacts/verification.yaml.

@codecov

codecov Bot commented Aug 7, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

@avrabe
avrabe merged commit 618b29d into main Aug 7, 2026
21 checks passed
@avrabe
avrabe deleted the feat/v036-mutants-truncation-389 branch August 7, 2026 06:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutation Testing scores a crashed cargo-mutants run as a good one (end_time null, survivors under-reported)

1 participant