Skip to content

The exemplar replay ignores order, indent, and four of five console blocks #379

Description

@CameronBrooks11

What

test_the_spacer_exemplars_diff_transcript_is_what_the_console_prints
(tests/test_docs.py) now requires every quoted output line to be a line the
tool printed, and #376 added a floor so a step cannot quote nothing. Three
weaknesses remain, all found by adversarial review of #376 and all confirmed
passing on both pinned engines:

  1. Order and multiplicity are not asserted. The check is set membership,
    so swapping the two quoted lines, or claiming one printed twice, passes.
  2. Leading whitespace is discarded. Both sides are .strip()ed, so
    un-indenting the covered: continuation line passes — and that indent is
    how the README shows it is a continuation.
  3. Only the first partspec diff console block is replayed. The
    exemplar's other four blocks — the --pin, the PASS: 8 pass / echo $?
    pair, the exit-4 pin mismatch, and the --pin rewrote claims stderr — are
    asserted by nothing. Falsifying one line in each leaves the whole file
    green, and just example-spacer does not cover them either.

Why it matters

#361 existed because the exemplar README drifted from what the tool prints and
three gates stayed green over it. The replay closes that for one command. The
other four commands in the same file are still in the state #361 was filed
about.

Smallest fix

For (1) and (2): compare the quoted lines to the tail of
printed.splitlines() in order, without stripping the left side.
For (3): replay every console block, not next() of them.

Found by review of #376, deliberately left out of that PR's scope as wider
than #361.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions