fix(import): report the collector's success label instead of inventing one - #403
Conversation
…g one The importer hardcoded success=true into every episode/v1 record while its episodes query read seven columns and never touched an outcome field. FORMAT.md defines success as the collector-labeled outcome and states the record is copied/merged from the source; the v3 format carries that label as a next.success feature with per-episode stats/next.success/min|max aggregates (lerobot/pusht declares it for all 206 episodes, uniformly False). The importer now reads stats/next.success/max as an eighth column when the source declares it (column discovery first: a corpus without the outcome feature is normal, not malformed), derives the episode outcome as MAX over the episode's frames, and stamps success together with a success_derivation naming the derivation so the methodology travels with the data. When the source declares no outcome feature the key is omitted entirely: FORMAT.md makes every episode/v1 key optional, and omission is the only encoding that cannot be read as a collector's judgment. The catalog promotion already renders an omitted key as SQL NULL; that behavior is pinned, not changed. CONVERTER_VERSION moves to v7 in the same commit: the label changes episode/v1 bytes that content_episode_id hashes. The fixture chain from Hebbian-Robotics#395 lands as tests through the real import, Episode reader, Catalog.append_episode, and SQL: max over a False,True transition reports true, an all-false source reports false, an unlabeled source omits the key and catalogs as NULL. Refs Hebbian-Robotics#395
…ty aggregate The PR carried 34 files under .zcode/, including generated MCAPs, MP4s and catalog parquets. Removed, and .zcode/ now sits beside .claude/ and .agents/ in .gitignore so it cannot recur for anyone using that tool. An outcome column present but empty for an episode stamped success: "false", since any([]) is False. That is the same invention the hardcoded "true" was, one value over, and nothing caught it; it now has its own case. Also lifts the column name and derivation string into constants, replacing a chr(34) workaround for f-string quote nesting on 3.11.
kstonekuan
left a comment
There was a problem hiding this comment.
Merging with fixups in a6bc98b. The change is right and follows the direction closely: max when the source declares the feature, omission when it does not, the derivation stamped so the methodology travels, and v7 in the same commit.
The PR carried 34 files under .zcode/, including generated MCAPs, MP4s, and catalog parquets. Removed. .zcode/ now sits beside .claude/ and .agents/ in .gitignore, so it cannot recur for you or anyone else using that tool. That is the durable version of what your .gitignore edit on #389 was reaching for; worth adding it to your global gitignore too, since this repo is not the only one that would collect it. The checklist line about not adding generated media is the one to watch: git status before git add -A catches this in a second.
One real gap, closed in the fixup. An outcome column that exists but is empty for an episode stamped success: "false", because any([]) is False. That is the same invention the hardcoded "true" was, one value over: a declared feature with nothing recorded is not a judgment. Nothing caught it, so it has its own case now.
empty aggregate treated as a label 1 failed <- was: 74 passed, nothing noticed
Everything else is properly held, and the is not None test rather than a truthy one is the detail that makes the False case work at all:
success hardcoded true again 2 failures
key stamped when the source has no label 1 failure
absent collapsed with False 1 failure
success_derivation not recorded 2 failures
outcome column never discovered 2 failures
CONVERTER_VERSION left at v6 1 failure
That last one is new and welcome. Three PRs in a row have needed a version bump and this is the first to pin it, so an omitted bump now fails rather than being caught in review.
Two derivation mutations go unnoticed, and that is correct rather than a gap. Flipping any to all, or to the last frame, changes nothing:
derivation flipped to MIN 75 passed, nothing noticed
derivation flipped to last frame 75 passed, nothing noticed
stats/next.success/max is already the per-episode maximum, so it arrives as a one-element list, exactly as your issue showed with ['False']. The any() reduces a single value. That means the reduction is defensive against a shape that does not occur, and the honest reading of success_derivation is "the source's own max", not a max we computed. Worth a sentence in the code saying so, since a future reader will otherwise assume the reduction is load-bearing and try to test it. I left the code alone; it is right, just easy to misread.
Small cleanup in the same fixup: the column name and derivation string are constants now, which removed the chr(34) workaround for f-string quote nesting on 3.11.
Gate on the merged result: ruff check, ruff format --check, ty check clean, 1531 passed / 6 skipped.
Closes #395. The trust thread reads well as a set now: #376 removed a stamp asserting an unmeasured value, #389 gave the delivery a checkable receipt, and this one stops the record speaking for the collector. The distinction you drew, that this one is about the data rather than the pipeline, is the right note to end on.
|
Both fixups fair. |
Closes #395.
The importer hardcoded
success: "true"into everyepisode/v1record while its episodes query read seven columns and never touched an outcome field. FORMAT.md definessuccessas the collector-labeled outcome and says the record is copied/merged from the source; the v3 format carries that label as anext.successfeature with per-episodestats/next.success/min|maxaggregates, andlerobot/pushtdeclares it for all 206 episodes, uniformly False. So the shipped record contradicted the source it read, 100 percent of the time, on the ecosystem's reference dataset.The fix, per the issue's open call: the importer reads
stats/next.success/maxas an eighth column when the source declares it, derives the episode outcome as MAX over the collector's frames (any success frame makes the episode a success), and stampssuccesstogether withsuccess_derivation: "max(stats/next.success)"so the methodology travels with the data. When the source declares no outcome feature, the key is omitted entirely: FORMAT.md makes everyepisode/v1key optional, and omission is the only encoding that cannot be mistaken for a collector's judgment. Column discovery runs first, because naming a missing column would fail the read, and a corpus without the feature is normal, not malformed.The catalog promotion already renders an omitted key as SQL NULL; that behavior is pinned now, not changed. The transform's copy path is untouched.
CONVERTER_VERSIONmoves to v7 in the same commit: the label changes episode/v1 bytes thatcontent_episode_idhashes.The #395 fixture chain lands as tests through the real import, the real
Episodereader, realCatalog.append_episode, and real SQL: a False,True transition reportstrueby MAX, an all-false source reportsfalse, an unlabeled source omits the key and catalogs as NULL.Gate: ruff, format, ty clean; 1530 passed / 6 skipped.
Refs #395. Third brick of the trust thread with #376 and #389.