Build Rerun Query-to-Train quality gate - #19
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements a revision-locked “Query-to-Train” quality gate pipeline for reBot data, using the Rerun Query API as the authority for segment-safe alignment, metric computation, deterministic thresholding/verdicts, and producing a checksummed training selection + locally validated derivative dataset.
Changes:
- Adds a staged competition workflow CLI (
inventory → materialize → audit → evaluate → prepare) with resumable caching and machine-readable outputs. - Hardens Rerun Query usage to be segment-safe (no cross-segment row mixing) and adds alignment/metrics/threshold calibration + evaluation plumbing with extensive tests.
- Introduces deterministic artifact writing (Parquet/JSON/HTML/MD + checksums), updates docs/specs, and adds a checked-in locked config for four HF sources.
Reviewed changes
Copilot reviewed 40 out of 42 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| requirements.txt | Pins Rerun SDK + adds HF Hub + PyArrow for the new pipeline dependencies. |
| p5_rerun_port/tests/test_rerun_hardening.py | Updates Query API smoke test to use a fixed recording segment identity. |
| p5_rerun_port/tests/challenge/test_report.py | Adds report rendering contract tests (self-contained, no scripts/styles). |
| p5_rerun_port/tests/challenge/test_query_selection.py | Adds tests for segment-safe selection + CLI path filtering behavior. |
| p5_rerun_port/tests/challenge/test_quality.py | Adds tests for label-blind calibration, deterministic digests, and scoring. |
| p5_rerun_port/tests/challenge/test_metrics.py | Adds deterministic metric calculation tests over aligned episodes. |
| p5_rerun_port/tests/challenge/test_inventory.py | Adds HF-locked inventory tests and source-lock writing validation. |
| p5_rerun_port/tests/challenge/test_curate.py | Adds derivative build + fresh-load validation tests and tamper resistance. |
| p5_rerun_port/tests/challenge/test_config.py | Adds config contract tests (pinned SHAs, robot profile digests, invariants). |
| p5_rerun_port/tests/challenge/test_challenge_cli.py | Adds tests for the new staged workflow CLI and “no upload flag” contract. |
| p5_rerun_port/tests/challenge/test_artifacts.py | Adds tests for deterministic manifest/run artifact writing + checksum enforcement. |
| p5_rerun_port/tests/challenge/test_alignment.py | Adds integration tests for segment-bound extraction + latest-at alignment behavior. |
| p5_rerun_port/rerun_query.py | Implements segment-safe dataframe alignment and episode/segment identity binding. |
| p5_rerun_port/query_challenge_cli.py | Introduces the competition workflow CLI with lazy stage imports. |
| p5_rerun_port/query_api_cli.py | Updates the compatibility Query API CLI to use selected RRD paths and identity verification. |
| p5_rerun_port/challenge/workflow.py | Adds concrete staged orchestration with caching and reproducible digests. |
| p5_rerun_port/challenge/quality.py | Adds outcome-blind threshold calibration and deterministic scoring/digesting. |
| p5_rerun_port/challenge/models.py | Adds typed immutable shared models for the curation pipeline. |
| p5_rerun_port/challenge/hub.py | Adds revision-pinned HF dataset reader adapter and protocol boundary. |
| p5_rerun_port/challenge/evaluate.py | Adds post-verdict label loading and held-out evaluation logic/contracts. |
| p5_rerun_port/challenge/config.py | Adds strict validation for the checked-in source lock + robot profile contract. |
| p5_rerun_port/challenge/canonical.py | Adds canonical RRD materialization for success episodes with verification gates. |
| p5_rerun_port/challenge/artifacts.py | Adds deterministic run artifact generation + selection manifest + verification. |
| p5_rerun_port/challenge/artifacts_types.py | Adds an alternate RunPayloads type module (currently redundant). |
| p5_rerun_port/challenge/alignment.py | Adds segment-bound Query extraction + integer-ns latest-at alignment. |
| p5_rerun_port/challenge/init.py | Exposes key challenge types in the package public surface. |
| docs/superpowers/specs/2026-07-19-rerun-query-quality-gate-design.md | Adds the approved design spec for the quality gate. |
| docs/Rerun_bounty_progress.md | Updates progress tracking to reflect the new quality gate workflow. |
| docs/p5_rerun_port/QUERY_API.md | Rewrites docs to reflect the competition workflow and outputs. |
| docs/HACKATHON_SUBMISSION_DEMO.md | Updates demo script to match the staged quality-gate workflow and truthfulness constraints. |
| config/rerun_query_challenge.yaml | Adds the checked-in challenge source lock (four pinned HF revisions + robot profile digests). |
| .superpowers/sdd/task-6-report.md | Adds Task 6 report with RED/GREEN evidence and alignment/metrics audit notes. |
| .gitignore | Ignores generated artifacts/ outputs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+57
to
+61
| if not path.exists(): | ||
| return expected | ||
| with path.open("rb") as stream: | ||
| value = pickle.load(stream) # local cache under the caller-selected artifacts root | ||
| if any(value.get(key) != expected_value for key, expected_value in expected.items()): |
Comment on lines
+28
to
+33
| cli = root.parent / "query_challenge_cli.py" | ||
| if cli.is_file(): | ||
| digest.update(cli.name.encode("utf-8")) | ||
| digest.update(b"\0") | ||
| digest.update(cli.read_bytes()) | ||
| return digest.hexdigest() |
Comment on lines
+1
to
+18
| """Shared payload type kept separate to avoid report/artifact import cycles.""" | ||
| from dataclasses import dataclass | ||
| from pathlib import Path | ||
| from typing import Any | ||
| from .evaluate import EvaluationReport | ||
| from .metrics import EpisodeMetrics | ||
| from .models import InventoryRow | ||
| from .quality import EpisodeVerdict, ThresholdSnapshot | ||
|
|
||
| @dataclass(frozen=True) | ||
| class RunPayloads: | ||
| source_lock: dict | ||
| inventory: tuple[InventoryRow, ...] | ||
| metrics: tuple[EpisodeMetrics, ...] | ||
| thresholds: ThresholdSnapshot | ||
| verdicts: tuple[EpisodeVerdict, ...] | ||
| evaluation: EvaluationReport | ||
| selection_manifest: dict |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Verification
Known runtime gate
The complete video-heavy 102-item materialize/evaluate/derivative run and Hugging Face publication were not performed because the machine has only 13 GiB free. No publication is claimed; upload remains gated on a measured local fresh-load and explicit user confirmation.