feat(import): per-episode receipts in prepared-manifest.json - #389
Conversation
prepared-manifest.json (schema version 3) now carries an episodes receipt list: every delivered episode's published URI, its content_episode_id content address, and its byte size. A recipient of a prepared corpus can check the delivery against the manifest without re-running the import, and a truncated, missing, or swapped episode is detectable from the delivery itself. The content id is taken inside _convert_single_episode while the canonical file is still on local disk, before storage.publish: for a bucket root, reading the hash back from the published object means downloading our own upload. content_episode_id stays the single hashing implementation. The v2 top-level keys are unchanged, so old readers keep working, and the entries carry everything a future verify command needs, keeping verify purely additive (Hebbian-Robotics#379 direction). Documented in docs/how-to/import-lerobot-v3.md. Refs Hebbian-Robotics#379
.gitignore had no trailing newline, so the added entry concatenated onto **/node_modules/* and produced **/node_modules/*.zcode/. That stopped ignoring node_modules entirely: git check-ignore reported packages/hflow-server/ui/node_modules as tracked. Both manifest tests build their receipts inside the convert stub, so dropping content_id or size_bytes from the real _convert_single_episode left the suite green. The real-conversion test now checks the receipt against the object it published. Also updates the two Hebbian-Robotics#388 assertions the schema-3 bump moves, which is what that whole-dict pin exists to force.
kstonekuan
left a comment
There was a problem hiding this comment.
Merging with three fixups in 75366de. The design is what the direction asked for and the truncation fixture landed as the right kind of test: it asserts the size and the content id both disagree after truncation, which is the detection the receipt exists for rather than a vague "something changed".
The .gitignore change was a real defect, and not a cosmetic one. That file had no trailing newline, so the added entry concatenated onto the line above it:
-**/node_modules/*
+**/node_modules/*.zcode/
That is one broken pattern, not two rules. It stopped ignoring node_modules at all:
$ git check-ignore -v packages/hflow-server/ui/node_modules/react/index.js
NOT IGNORED (rule broken)
Restored to **/node_modules/* with a trailing newline. .zcode/ looks like local tooling rather than anything this repo needs, so I did not add it back; if you want it ignored, that belongs in your global gitignore rather than the project's.
The receipt fields were not held by any test. Both manifest tests go through _install_publish_through_convert, and that stub builds the receipt itself, so the assertions were checking the test's own dict. Dropping either field from the real _convert_single_episode left everything green:
content_id dropped from the receipt 47 passed, nothing noticed
size_bytes dropped from the receipt 47 passed, nothing noticed
This is the same shape as the gap I flagged on #377, and the same fix: test_video_cache_distinguishes_file_indices_and_reuses_same_source already drives the real function, so the receipt now gets checked there against the object it published. Both mutations fail now.
The rest of the manifest is properly pinned, and #388's whole-dict assertion did its job: your schema bump broke it, which is why it exists. I updated it and the receipt["uri"] call in the provenance test, since those two are consequences of your change rather than mistakes in it.
One correction to a constraint I gave you. I said to hash before storage.publish because reading it back on a bucket root means downloading your own upload. The first half of that was over-specified: both operations run on the same local canonical path inside the same temporary directory, so swapping the two lines is behavior-neutral and nothing tests the order, correctly. What actually mattered was not hashing the published URI, and you did not. Keep the comment; it explains a real hazard. I just should not have phrased it as an ordering requirement.
Keeping episodes_converted alongside episodes is the right call for v2 readers, and it is pinned, so it cannot quietly drift out of agreement with the list length.
Gate on the merged result: ruff check, ruff format --check, ty check clean, 1492 passed / 6 skipped. Your one failure was #385, merged a couple of hours after you opened this.
Closes #379. Verify stays deferred and the entries carry what it will need, so when we take it, it is a reader of this manifest and not another schema bump. That was the whole point of splitting it.
Closes #379.
The manifest recorded how many episodes were converted but not which ones, so a truncated, missing, or swapped episode was undetectable from the delivery (#379's controlled result: one landing file cut to zero bytes, manifest unchanged). This adds the receipt.
prepared-manifest.jsonmoves to schema version 3: every delivered episode carries its published URI, itscontent_episode_idcontent address, and its byte size. The v2 top-level keys are unchanged, so existing readers keep working, and the entries carry everything a future verify command needs, keeping verify purely additive per the issue direction: the entries are done here, verify is deferred, not rejected.Per the constraints on the issue: the hash is taken inside
_convert_single_episodewhile the canonical file is still on local disk, beforestorage.publish, so a bucket root never downloads its own upload to learn its content id; the recorded URI is the published object (a bucket prefix recipient has no local paths); andcontent_episode_idstays the single hashing implementation, reused rather than forked. The source cache is not hashed; the receipt covers what was delivered, not what it was made from.The truncation fixture from the issue lands as
test_manifest_content_id_detects_a_truncated_episode: after truncation, both the size and the content id disagree with the manifest, which is the detection the receipt exists for.Cost: one linear sha256 read of each canonical file while it is already on local disk, 4.6 ms at fixture scale, about 0.2 s for a 100 MB episode.
Gate: ruff, format, ty clean; 1471 passed / 6 skipped, the single failure (
utc-stats-test-date-collision) reproduced on the base commit and unrelated, fix already in flight on its own branch.Refs #379, builds on #377's published URI list.