feat(lerobot): resume multi-episode imports at episode boundaries (#303)

[VARUN3WARE]

Summary

• Reuse matching landing/*.mcap episodes after a mid-batch import failure instead of rebuilding them.
• Stamp import identity (source commit/episode, camera keys, converter version, GOP) and check it before reuse; refuse partial/unreadable/mismatched files.
• Bump prepared-manifest.json to schema 3 with per-episode uri, content_episode_id, and size_bytes (covers the delivery-receipt half of prepared-manifest.json records how many episodes were converted but not which ones, so a corrupt or missing episode cannot be detected from the delivery #379 without a verify command).
• Document the retry contract in docs/how-to/import-lerobot-v3.md.

Closes #303

Test plan

• uv run ruff check
• uv run ruff format --check
• uv run ty check
• uv run pytest tests/test_lerobot_converter.py -q (46 passed)
• uv run pytest -q (1471 passed, 6 skipped; 1 unrelated UTC date-substring flake in test_preview_stats_render_timestamps_in_utc_on_a_non_utc_host, upstream test(server): stop asserting UTC rendering with a substring that matches the date #385)

Notes

• Uses existing content_episode_id(); hashes before publish so bucket roots do not re-download for the receipt.
• Optional: comment on Make multi-episode LeRobot imports resumable after interruption #303 that you are taking it, and that schema-3 episode entries intentionally overlap prepared-manifest.json records how many episodes were converted but not which ones, so a corrupt or missing episode cannot be detected from the delivery #379 so that issue can follow or close as superseded for the entries half.

[kstonekuan]

Holding this rather than merging, and most of the reason is timing rather than your code. The resume design is right: stamping identity and checking it beats "the file exists", and refusing a mismatched or unreadable landing file instead of trusting the name is the part that makes reuse safe rather than fast-and-wrong.

One real defect, and it is the kind that only shows up later. Adding camera_keys and gop_seconds to the episode/v1 record changes the canonical bytes, and content_episode_id hashes the whole canonical file. CONVERTER_VERSION stays at v5, so the same episode imported before and after this PR gets two different content ids while claiming the same converter version. Same fixture, same input, one commit apart:

main    CONVERTER_VERSION=lerobot-converter-v5  content_episode_id=157868a95a35c547
pr-390  CONVERTER_VERSION=lerobot-converter-v5  content_episode_id=f6084f8403d1670d

That is the collision #383's v4-to-v5 move existed to prevent: the catalog dedupes on the content id, so a correct old episode and a correct new episode become two rows that disagree about what they are. Bump CONVERTER_VERSION to v6. The stamping is load-bearing for reuse so it cannot come out, which means the version has to move with it. TRANSFORM_BEHAVIOR_VERSION stays put; the transform's bytes are unchanged.

Two of your three halves landed under you in the last two hours, and I am sorry about the wasted effort. #389 merged the schema-3 per-episode entries for #379, and #385 merged a different fix for the UTC date-substring failure. So:

• Drop the manifest and receipt changes. What landed uses content_id rather than content_episode_id as the key name, so yours will conflict rather than rebase.
• Drop packages/hflow-server/tests/test_server_curation_preview.py entirely.
• Rebase onto current main.

You flagged the #379 overlap yourself and offered to let that issue follow or be superseded, which was the right instinct. It went the other way only because #379 already had a named direction and #389 was implementing it, not because your version was worse.

One thing from your entries is a genuine improvement over what landed: source_episode_index. The merged entries carry uri, content_id, and size_bytes, so which source episode produced which file is inferable from the filename but never stated. Worth adding, but as its own small PR after this one rather than folded in, so this stays about resume.

Two smaller things, both worth doing whichever way the rest goes:

except Exception: return False in _episode_identity_matches will swallow a genuine bug in open_reader as cheerfully as it swallows a truncated file, and turn it into a silent full reconversion. Narrow it to what a damaged or partial MCAP actually raises.

episodes_converted now counts reused episodes as converted. After a resume that reuses everything, the manifest reports work that did not happen. Either the field means delivered and the name is wrong, or it means converted and reuse should not increment it. It is pinned by a whole-dict assertion on main now, so whichever you pick, a test will hold it.

Gate on your branch as it stands is otherwise clean, and #303 is unassigned so nothing is competing with you for it. Say the word if you would rather I take the version bump myself and land the reshape; there is no need for this to sit.

[VARUN3WARE]

Thanks, reshaped on current main:

• Bumped CONVERTER_VERSION to v6 with the camera_keys/gop_seconds stamps
• Dropped our manifest/receipt changes (kept feat(import): per-episode receipts in prepared-manifest.json #389’s content_id shape)
• Dropped the UTC test fix (test(server): stop asserting UTC rendering with a substring that matches the date #385)
• Narrowed the identity-match except to OSError/McapError/ValueError
• episodes_converted now counts only conversions this run (reuse still listed under episodes)

source_episode_index in the manifest left for a follow-up as suggested.

[kstonekuan]

LGTM, merging. Every point addressed, and the v6 comment says why the version moved rather than just that it did, which is what makes it useful in a year.

I pushed one fixup, 6141cf4. Five of the six identity comparisons carried no weight. test_import_does_not_reuse_identity_mismatched_landing_episode builds its landing file from DatasetSource("other/repo", "deadbeef", ...), so both repo_id and revision differ at once and either one alone still catches it; the other four fields were never exercised. Dropping any single comparison left the suite green:

source_revision not compared          50 passed, nothing noticed
source_episode_index not compared     50 passed, nothing noticed
camera selection not compared         50 passed, nothing noticed
converter_version not compared        50 passed, nothing noticed
gop_seconds not compared              50 passed, nothing noticed

That mattered more here than it usually would. Reuse is the direction where trusting too much is dangerous: a landing file from another revision or another camera selection handed back as completed work is wrong data delivered silently, with no error anywhere. Your _write_identity_matching_landing_mcap helper made this cheap to close, it just needed override hooks so one field can break while the rest match. Each of the eleven cases is now the only failure when its own comparison goes, and there is a control asserting the unmodified fixture is reused, so the parametrization cannot pass by never matching in the first place.

Both < 1 size checks are dead code. Worth knowing rather than acting on: removing them together leaves everything green, because an empty file is not a readable MCAP and the reader refusal already covers it.

caller's storage.file_size guard removed   63 passed, nothing noticed
both zero-size guards removed              63 passed, nothing noticed

The one in _try_reuse_completed_episode still earns its place, since it short-circuits before storage.fetch and so avoids downloading a zero-byte object to learn it is empty. That reason is invisible to a local-root test, so it deserves a comment saying so. The one inside _episode_identity_matches runs after the fetch and has no such excuse; I would drop it. A guard that looks load-bearing and is not costs the next reader more than it saves.

Narrowing to (OSError, McapError, ValueError) with the comment about reader bugs still surfacing is the right shape, and importing McapError inside the function keeps it off the module's import path.

Gate on the merged result: ruff check, ruff format --check, ty check clean, 1508 passed / 6 skipped.

Closes #303. source_episode_index in the manifest entries is still worth doing and is yours if you want it, as its own PR. Beyond that, the higher-leverage thing given how much of the importer you have now been through: point HFlow at a real corpus (Egocentric-10K or Egocentric-100K on Hugging Face) and report what breaks or drags. #287 came from exactly that.