What
test_the_spacer_exemplars_diff_transcript_is_what_the_console_prints
(tests/test_docs.py) now requires every quoted output line to be a line the
tool printed, and #376 added a floor so a step cannot quote nothing. Three
weaknesses remain, all found by adversarial review of #376 and all confirmed
passing on both pinned engines:
- Order and multiplicity are not asserted. The check is set membership,
so swapping the two quoted lines, or claiming one printed twice, passes.
- Leading whitespace is discarded. Both sides are
.strip()ed, so
un-indenting the covered: continuation line passes — and that indent is
how the README shows it is a continuation.
- Only the first
partspec diff console block is replayed. The
exemplar's other four blocks — the --pin, the PASS: 8 pass / echo $?
pair, the exit-4 pin mismatch, and the --pin rewrote claims stderr — are
asserted by nothing. Falsifying one line in each leaves the whole file
green, and just example-spacer does not cover them either.
Why it matters
#361 existed because the exemplar README drifted from what the tool prints and
three gates stayed green over it. The replay closes that for one command. The
other four commands in the same file are still in the state #361 was filed
about.
Smallest fix
For (1) and (2): compare the quoted lines to the tail of
printed.splitlines() in order, without stripping the left side.
For (3): replay every console block, not next() of them.
Found by review of #376, deliberately left out of that PR's scope as wider
than #361.
What
test_the_spacer_exemplars_diff_transcript_is_what_the_console_prints(
tests/test_docs.py) now requires every quoted output line to be a line thetool printed, and #376 added a floor so a step cannot quote nothing. Three
weaknesses remain, all found by adversarial review of #376 and all confirmed
passing on both pinned engines:
so swapping the two quoted lines, or claiming one printed twice, passes.
.strip()ed, soun-indenting the
covered:continuation line passes — and that indent ishow the README shows it is a continuation.
partspec diffconsole block is replayed. Theexemplar's other four blocks — the
--pin, thePASS: 8 pass/echo $?pair, the exit-4 pin mismatch, and the
--pin rewrote claimsstderr — areasserted by nothing. Falsifying one line in each leaves the whole file
green, and
just example-spacerdoes not cover them either.Why it matters
#361 existed because the exemplar README drifted from what the tool prints and
three gates stayed green over it. The replay closes that for one command. The
other four commands in the same file are still in the state #361 was filed
about.
Smallest fix
For (1) and (2): compare the quoted lines to the tail of
printed.splitlines()in order, without stripping the left side.For (3): replay every console block, not
next()of them.Found by review of #376, deliberately left out of that PR's scope as wider
than #361.