fix(import): CRC-validate reused landing episodes before stamping receipts - #429
Conversation
…eipts The Hebbian-Robotics#390 resume path stamped conversion-grade receipts over reused bytes it never integrity-checked: _episode_identity_matches compared seven metadata fields and never read the message payloads, so a landing file with payload damage (bit rot, partial external copy) that left the container and metadata intact was reused and given a fresh receipt. In the resume scenario the prior run published no manifest (manifest-last), so nothing recorded the damage. The identity check now ends with a CRC-validated full message pass (the same standard hflow doctor applies). A file that fails falls through to re-conversion: damaged work is not reuse, it is work to redo. The reuse log line names the verification. tests/reuse_test_helpers.py adds flip_chunk_payload_bytes, a surgical corruption helper that flips bytes inside a chunk's records region while leaving the container and metadata intact, shared so the Hebbian-Robotics#423-family tests can reuse it. The identity fixture now writes one chunked message with CompressionType.NONE so the corruption is addressable in place. Controlled result on current main: 400 payload bytes flipped, size and metadata unchanged, identity still matched without the fix, and the reuse receipt carried a different content id (ce5f28aa -> 28ef502d). With the fix, the damaged file is refused and re-converted. Refs Hebbian-Robotics#426
kstonekuan
left a comment
There was a problem hiding this comment.
The measurement is what makes this reviewable. 1.15x on a read that was already happening turns "is this too expensive" from an argument into a number, and 51 ms against a re-conversion is not a real cost.
Validated on your branch: full gate clean, 1587 passed / 6 skipped. Four mutations, because a hand-edited corrupt file can easily be refused for a reason that has nothing to do with the feature:
validate_crcs=Trueflipped toFalse, everything else intact: both tests red. This is the one I most wanted to see, since it proves the flag is doing the work rather than the extra read.- The whole pass removed: red.
- The CRC failure swallowed and reused anyway: red.
- Reader opened but messages never iterated: red. Confirms iteration is load-bearing, which it is, since the CRC is checked when the chunk is decoded.
So the damaged file is refused for the stated reason and not by accident.
Pushed one commit, feat(import): warn when a reused episode fails CRC validation, for the half of #426's second definition-of-done item that was still open. You covered "reused after verification" with the log line. The other side was silent: a damaged episode fell through to re-conversion indistinguishably from a first import, so the import repairs the bit rot and nobody ever learns the landing tree is rotting. The warning fires only in the branch where identity already matched, so it cannot fire for a stranger's file, and it names the path. The test now asserts it, and removing the warning turns that test red on its own.
One thing I left alone: test_reuse_accepts_an_intact_episode_after_the_crc_pass is a strict subset of the damaged test, which already asserts both the intact receipt and the refusal after corruption. It costs nothing and reads as documentation, so it stays, but it is not adding coverage.
Putting the CRC pass inside _episode_identity_matches means a caller cannot tell "not mine" from "mine but damaged". That is fine while both answers mean re-convert, and the warning covers the observability gap it would otherwise leave. Worth remembering if anything ever needs to branch on the difference.
Merging.
…e path (#462) #429 fixed the LeRobot resume path; the same hole was open one stage earlier, on the primary path every ingest takes. transform.py's read of the source file (open_reader) never asked for CRC validation, so a structurally valid MCAP with a damaged chunk payload transcoded without complaint and got a fresh, true receipt over corrupt bytes. open_reader and PythonMcapEpisodeReader gain validate_crcs (default False); only the ingest read in transform.py opts in, so the other three call sites reading HFlow's own canonical output are unaffected. The transcode already decodes every chunk in one pass, so this piggybacks a check on a read that already happens: measured 1.113x on a 30MB source and 1.035x on a 75MB one. CRCValidationError subclasses ValueError, not McapError, so classify_ingest_failure needed its own branch for it -- without one it silently fell through to INFRASTRUCTURE, blaming the platform for a damaged recording. It classifies to the same SOURCE_UNREADABLE kind as the not-MCAP case; error_type (InvalidMagic vs CRCValidationError) keeps the two distinguishable in the ledger without a new enum member. Refs #431 Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
Closes #426.
The #390 resume path stamped conversion-grade receipts over reused bytes it never integrity-checked:
_episode_identity_matchescompared seven metadata fields and never read the message payloads. A landing file with payload damage that left the container and metadata intact (on-disk bit rot, a partial external copy) matched identity and received a fresh receipt. The resume scenario made it quiet: run one failed mid-batch and published no manifest (manifest-last), so nothing recorded the damage.The fix: the identity check now ends with a CRC-validated full message pass, the same standard
hflow doctorapplies to any canonical file. Metadata matching alone is not integrity; the metadata records live outside the chunks, so payload damage never reaches them. A file that fails the CRC pass falls through to re-conversion: damaged work is not reuse, it is work to redo. The reuse log line now says "reusing verified" and names the content id.Measured cost on a real canonical episode: the CRC-validated full message read is about 1.15x the content id hash pass that already ran pre-#426 (2.6 ms vs 2.3 ms at fixture scale, 51 ms vs 40 ms at 8 MB scale). The read was already happening for the receipt; the CRC flag makes the same pass verify.
tests/reuse_test_helpers.py adds
flip_chunk_payload_bytes, a surgical corruption helper that flips bytes inside a chunk's records region while leaving the container, indexes, and metadata intact, shared so the #423-family tests can reuse it. The identity fixture now writes one chunked message (CompressionType.NONE) so the corruption is addressable in place.Controlled result on current main: 400 payload bytes flipped, size and metadata unchanged, identity matched without the fix, and the reuse receipt carried a different content id (ce5f28aa -> 28ef502d). With the fix, the damaged file is refused and re-converted. Mutation: deleting the CRC pass turns the controlled test red.
Gate: ruff, format, ty clean; 1587 passed / 6 skipped.
Refs #426. Fourth brick of the trust thread with #376, #389, and #395.