Harden hackathon Rerun, Viewer, and SmolVLA workflow

.gitignore

@@ -1,4 +1,5 @@
 .venv/
+.venv-lerobot/
 .venv-zed/
 pyzed-*.whl
 PyOpenGL*.whl
@@ -16,6 +17,7 @@ config/.env
 
 # Runtime / calibration artifacts
 runs/
+outputs/
 data/calibration/*.npy
 data/calibration/*.json
 data/calibration/*.png

SMOLVLA_TRAINING.md

@@ -0,0 +1,140 @@
+# SmolVLA — local training (Track B)
+
+How to fine-tune a **SmolVLA** policy locally on this machine's GPU, using the
+vendored LeRobot. This is the SmolVLA half of Track B; MolmoAct is trained
+separately on Modal.
+
+Verified working on an Apple Silicon Mac (MPS) on 2026-07-18 using the
+repository's standard LeRobot environment,
+`rebot_setup/vendor/rebot_lerobot/.venv`.
+
+**Fine-tune, don't train from scratch.** We warm-start from the pretrained
+`lerobot/smolvla_base` checkpoint — a SmolVLA that already knows how to move an
+arm — and adapt it to our data. This matches the official hackathon recipe
+("warm-start from the SO-101 base"). Training from scratch (`--policy.type=smolvla`)
+will not learn a usable policy on ~150 episodes, so we always pass
+`--policy.path=lerobot/smolvla_base` instead.
+
+---
+
+## One-time setup
+
+The system default Python (3.13) is too new for the ML stack. Build the
+repository's pinned **Python 3.11** environment, then add the SmolVLA policy
+dependencies to that same environment:
+
+```bash
+# from the repo root
+./rebot_setup/setup.sh
+uv pip install \
+  --python rebot_setup/vendor/rebot_lerobot/.venv/bin/python \
+  'transformers>=4.57.1,<5.0.0' \
+  'num2words>=0.5.14,<0.6.0' \
+  'accelerate>=1.7.0,<2.0.0' \
+  'safetensors>=0.4.3,<1.0.0'
+```
+
+The `uv pip` form is intentional: the environment built by `setup.sh` does not
+install the `pip` module. Also do not install the full vendored
+`lerobot[smolvla]` extra into this shared environment: that package metadata
+pins `rerun-sdk<0.27` and would downgrade the Rerun 0.34 runtime required by the
+Query API integration. `setup.sh` has already installed vendored LeRobot; the
+four packages above are the only SmolVLA additions it needs.
+
+Confirm it worked:
+
+```bash
+rebot_setup/vendor/rebot_lerobot/.venv/bin/lerobot-train --help
+rebot_setup/vendor/rebot_lerobot/.venv/bin/python -c \
+  "import torch; print('mps:', torch.backends.mps.is_available())"
+```
+
+## Important: the `pyav` flag is mandatory on this Mac
+
+The default video reader (`torchcodec`) is broken here — it can't link the system
+FFmpeg libraries and crashes the moment training reads a recording. **Every**
+training command must therefore include:
+
+```
+--dataset.video_backend=pyav
+```
+
+`pyav` is already installed and bundles its own FFmpeg, so no system install is
+needed. Leave this flag out and training fails at the first batch.
+
+## Important: camera names must match the base model
+
+`smolvla_base` was pretrained expecting **three** cameras named
+`observation.images.camera1`, `camera2`, `camera3`. Our dataset's cameras are
+named differently (`front`, `side`) and there are only two of them, so training
+stops with a "Feature mismatch" error unless you:
+
+1. **Rename** your camera keys to `camera1`/`camera2` with `--rename_map`, and
+2. **Pad** the missing third camera with `--policy.empty_cameras=1`.
+
+For our reBot dataset (`front` = overhead, `side` = 45°) that is:
+
+```
+--rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.side": "observation.images.camera2"}'
+--policy.empty_cameras=1
+```
+
+(Confirm the exact camera key names against the real dataset's `meta/info.json`
+before the first real run — adjust the map if they differ.)
+
+## Train
+
+```bash
+HF_HUB_ENABLE_HF_TRANSFER=1 rebot_setup/vendor/rebot_lerobot/.venv/bin/lerobot-train \
+  --policy.path=lerobot/smolvla_base \
+  --dataset.repo_id=<DATASET_REPO_ID> \
+  --dataset.video_backend=pyav \
+  --rename_map='{"observation.images.front": "observation.images.camera1", "observation.images.side": "observation.images.camera2"}' \
+  --policy.empty_cameras=1 \
+  --policy.device=mps \
+  --policy.push_to_hub=false \
+  --batch_size=8 \
+  --steps=5000 \
+  --save_freq=1000 \
+  --output_dir=outputs/smolvla/<run_name> \
+  --wandb.enable=false
+```
+
+- Replace `<DATASET_REPO_ID>` with the recorded dataset (e.g. the LeRobot repo id
+  the data team uses; a locally recorded dataset lives at
+  `~/.cache/huggingface/lerobot/<repo_id>`).
+- Checkpoints are written under `outputs/smolvla/<run_name>/checkpoints/`. The
+  `pretrained_model/` folder inside each checkpoint is what the inference/harness
+  side loads.
+- `outputs/` is git-ignored — sync checkpoints off-machine (Drive / HF) rather
+  than committing them.
+
+## Smoke test (no real data needed)
+
+To prove the machine can train before real recordings exist, run a few steps on a
+public SmolVLA example dataset:
+
+```bash
+HF_HUB_ENABLE_HF_TRANSFER=1 rebot_setup/vendor/rebot_lerobot/.venv/bin/lerobot-train \
+  --policy.path=lerobot/smolvla_base \
+  --dataset.repo_id=lerobot/svla_so101_pickplace \
+  --dataset.episodes='[0, 1]' \
+  --dataset.video_backend=pyav \
+  --rename_map='{"observation.images.side": "observation.images.camera1", "observation.images.up": "observation.images.camera2"}' \
+  --policy.empty_cameras=1 \
+  --policy.device=mps \
+  --policy.push_to_hub=false \
+  --batch_size=2 --steps=20 --log_freq=1 --save_freq=20 --eval_freq=0 \
+  --num_workers=0 \
+  --output_dir=outputs/smolvla/smoke_test \
+  --wandb.enable=false
+```
+
+(This public dataset's cameras are `side`/`up`, hence the different rename map
+than our real reBot data.) Success = all 20 steps report finite loss and
+`checkpoints/000020/pretrained_model/model.safetensors` exists. Do not require a
+monotonic loss curve from only 20 shuffled mini-batches. This exact command path
+was verified end-to-end on MPS on 2026-07-18: it loaded 2 public episodes / 569
+frames, trained 100M of 450M parameters, printed finite losses for every step,
+saved the step-20 checkpoint, and exited 0. It does not validate our reBot data
+or autonomous arm motion.

docs/HACKATHON_SUBMISSION_DEMO.md

@@ -0,0 +1,151 @@
+# Hackathon submission and judge demo
+
+Use this as the short presentation layer. Detailed setup and safety procedures
+remain in `docs/Rerun_bounty_progress.md`, `docs/p5_rerun_port/README.md`, and
+`p3_vlm_orchestrator/PERSON4_RUNBOOK.md`.
+
+## 60-second pitch
+
+> We ported the Rerun SO-101 learning loop to reBot, a different seven-joint
+> robot with Damiao motors, two named cameras, and an extra wrist-yaw degree of
+> freedom. One episode flows from joint, goal, camera, and URDF logging into a
+> local Rerun catalog; the Query API compares commanded and observed motion so
+> we can reject bad takes; selected episodes export in a reBot LeRobot v3
+> schema; and the same trajectory can be replayed or handed to our guarded
+> learned-policy runner. The important part is the loop, not a URDF screenshot:
+> collect, inspect, curate, export, and close the loop. Our checked-in evidence
+> proves that entire path with synthetic hardware. [Only after a successful
+> rehearsal: We also ran the same path on the physical reBot.] The result is a
+> reproducible non-SO-101 integration with auditable safety and data contracts.
+
+Do not say the bracketed live sentence unless the live row in the evidence
+table below has been filled with a recording, command log, and outcome.
+
+## Prize-track mapping
+
+| Track | Submission position | Evidence and remaining gate |
+|---|---|---|
+| Rerun non-SO-101 end-to-end port ($1k) | **Primary** | `p5_rerun_port` covers log, record, catalog/query, LeRobot export, and replay for a 7-DOF reBot. Synthetic path is documented green; physical end-to-end remains a required venue proof. |
+| Rerun Query API ($2k) | **Secondary** | `query_api_cli` uses the Rerun server/reader path for schema, entity reads, and goal-versus-position comparison. The tracked example is `docs/p5_rerun_port/examples/cans_query_report.md`; current proof is synthetic. |
+| Interesting Rerun Viewer ($2k) | **Additional submission** | Interactive recordings activate a purpose-built Blueprint with synchronized front/side cameras, the 3D reBot URDF, goal-versus-position traces, and the time panel in one operator view. Rehearse the real Viewer before claiming live use. |
+
+## Evidence boundary
+
+| Capability | Synthetic evidence currently supported | Live evidence required before claiming it |
+|---|---|---|
+| reBot telemetry, cameras, goals, and URDF in Rerun | `log_rebot --fake --teleop`; tracked progress marks dry-run done | Saved live `.rrd` showing the physical seven-joint follower, both real cameras, and URDF |
+| Episode recording and catalog | `record_episode --fake`; dry-run `.rrd`, trajectory sidecar, frames, and catalog path documented | One clean physical can-to-zone episode with matching `.rrd`, metadata, trajectory, and camera frames |
+| Query API curation | Schema/entity/goal-vs-position commands and tracked example report | Run the same query against the physical episode and show the episode ID on screen |
+| Central Viewer workflow | The checked-in Blueprint and real synthetic `.rrd` put both cameras, the URDF, and tracking error in one layout | Open the physical episode with the Blueprint active and use it during collection/curation, not only as a final screenshot |
+| LeRobot export | Fallback export is documented green with reBot type, seven joints, `front`, then `side` | Load and validate an export made from the physical episode; record exact output path/revision |
+| Replay / close loop | `replay_episode --fake` is documented green | Physical replay only after a supervised rehearsal, clear workspace, e-stop operator, and conservative speed |
+| Learned autonomous pick | Guarded runner, offline/shadow/live gates, and tests exist | A real checkpoint must pass inspect, offline, shadow, and empty-workspace live gates before filming a can pick |
+| Training quality or success rate | Training/evaluation tooling exists | A real checkpoint plus held-out trial report; do not infer success from training loss or code tests |
+
+## Judge demo sequence
+
+### 1. State the evidence level (5 seconds)
+
+Say either “This is the reproducible synthetic pipeline” or “This is the live
+pipeline validated in rehearsal.” Never switch labels mid-demo.
+
+### 2. Show one complete synthetic episode (25 seconds)
+
+Run from the repository root. A timestamped dataset avoids reusing stale demo
+artifacts:
+
+```bash
+export DEMO_DATASET="hackathon-demo-$(date -u +%Y%m%dT%H%M%SZ)"
+
+python -m p5_rerun_port.record_episode \
+  --fake --dataset "$DEMO_DATASET" \
+  --task "Pick up one can and place it in the taped sorting zone" \
+  --tag "Good episode" --seconds 5 --no-viewer
+
+rerun "recordings/$DEMO_DATASET/episode_01.rrd"
+```
+
+In Rerun, point out `follower/position`, `follower/goal`, the seven-joint arm,
+and both camera streams. If any entity is absent, stop and use the last
+rehearsed artifact without calling the new run successful.
+
+### 3. Query and curate (15 seconds)
+
+```bash
+python -m p5_rerun_port.query_api_cli --dataset "$DEMO_DATASET" --schema
+python -m p5_rerun_port.query_api_cli \
+  --dataset "$DEMO_DATASET" --compare goal-vs-position
+python -m p5_rerun_port.query_dataset \
+  --dataset "$DEMO_DATASET" --tag "Good episode"
+```
+
+Explain that goal-versus-position error exposes lag, dropped samples, or bad
+takes before they enter training.
+
+### 4. Export and close the synthetic loop (10 seconds)
+
+```bash
+python -m p5_rerun_port.export_lerobot \
+  --dataset "$DEMO_DATASET" --tag "Good episode" --fallback
+python -m p5_rerun_port.replay_episode \
+  --dataset "$DEMO_DATASET" --episode episode_01 \
+  --fake --speed 0.5 --no-viewer
+```
+
+Call this a schema/export and fake-replay proof. Do not call it physical replay.
+
+### 5. Optional live closeout
+
+Only use this after the full preflight below and one successful private
+rehearsal. Keep a dedicated operator on the physical e-stop/power cut.
+
+```bash
+./rebot_operator_kit/01_check_hardware.command
+./rebot_operator_kit/02_dual_camera_check.command
+
+python -m p5_rerun_port.log_rebot --teleop --seconds 20
+python -m p5_rerun_port.record_episode \
+  --dataset cans-live-demo \
+  --task "Pick up one can and place it in the taped sorting zone" \
+  --tag "Good episode"
+python -m p5_rerun_port.query_api_cli \
+  --dataset cans-live-demo --compare goal-vs-position
+python -m p5_rerun_port.export_lerobot \
+  --dataset cans-live-demo --tag "Good episode"
+```
+
+Physical replay is a separate safety decision. Do not improvise it during the
+judge session. If it was approved and rehearsed, use the documented command:
+
+```bash
+python -m p5_rerun_port.replay_episode \
+  --dataset cans-live-demo --episode episode_01 --speed 0.5
+```
+
+For a learned checkpoint, follow all four gates in
+`p3_vlm_orchestrator/PERSON4_RUNBOOK.md`; never jump directly to a can pick.
+
+## Preflight checklist
+
+- [ ] The presenter can identify every artifact as synthetic or live.
+- [ ] `git status --short` has no unexplained source changes; runtime recordings remain ignored.
+- [ ] `python -m p5_rerun_port.record_episode --help` and `python -m p5_rerun_port.query_api_cli --help` open successfully.
+- [ ] The timestamped synthetic sequence above has been rehearsed from a fresh dataset name.
+- [ ] Rerun opens the saved `.rrd`; both cameras, `follower/position`, and `follower/goal` are visible.
+- [ ] Query API schema and goal-versus-position comparison return for the same episode.
+- [ ] Export contains seven ordered joints and the `front`, then `side` image contract.
+- [ ] Demo screen recording, terminal font size, and backup artifact are ready.
+- [ ] Live only: hardware discovery and simultaneous camera checks pass immediately before the demo.
+- [ ] Live only: fixed cameras, lighting, taped zone, cables, and arm bases have not moved.
+- [ ] Live only: workspace is clear, follower is supported, and one person owns the physical e-stop/power cut.
+- [ ] Live only: no other process owns follower or leader serial ports.
+- [ ] Live only: one private full-path rehearsal produced a saved `.rrd` and command log.
+- [ ] Learned policy only: checkpoint identity, processor files, calibration/profile digest, offline, shadow, and empty-workspace live gates all pass.
+- [ ] Off-site copy of the chosen `.rrd`, export, checkpoint (if used), and demo video exists.
+
+## Failure fallback
+
+If live hardware, cameras, Rerun, or policy gating fails, stop motion and show
+the last verified synthetic `.rrd` plus the tracked query report. State the
+failure plainly. A reproducible dry-run with a clear live gap is stronger than
+an unsafe or mislabeled “live” claim.

docs/Rerun_bounty_progress.md

@@ -21,6 +21,7 @@
 | Slice D2 — **Rerun Query API** | done (dry-run) | `query_api_cli`: Server + `reader()` + goal-vs-position; see `QUERY_API.md` |
 | Slice E — `export_lerobot` | done (dry-run) | `robot_type=seeed_b601_dm_follower`, 7 joints, front/side |
 | Slice F — `replay_episode` | done (dry-run) | Fake replay OK; live follower TODO on venue |
+| Viewer Blueprint | done (synthetic) | Interactive recordings center both cameras, 3D reBot URDF, goal/position traces, and the time panel |
 | Live arm e2e (no `--fake`) | TODO | Venue: log → record → export → replay |
 | Demo video + 60s pitch | TODO | |
 | Off-site sync (RRDs / checkpoints) | TODO | |
@@ -36,7 +37,7 @@
 |-------|--------|
 | $1k non-SO-101 port | Code + dry-run; live e2e TODO |
 | $2k Query API | Implemented: `query_api_cli` + docs (dry-run on fake `.rrd`) |
-| $2k interesting Viewer | Not implemented (no custom blueprints/views yet) |
+| $2k interesting Viewer | Purpose-built operator Blueprint implemented and synthetic `.rrd` verified; live Viewer rehearsal TODO |
 
 ---
 

docs/TEAM.md

@@ -60,7 +60,13 @@
 
 **Two model routes — start with the easy one:**
 
-- **SmolVLA (do first — ready now, no cloud):** uses the vendored LeRobot at `rebot_setup/vendor/rebot_lerobot/lerobot/`. Install once: `pip install -e ".[smolvla]"` from that folder. Train: `lerobot-train --policy.type=smolvla --dataset.repo_id=<repo_id> --output_dir=outputs/smolvla_test --steps=200`. Expect auth/format errors on the first run — fixing them early is the whole point of starting at episode 10.
+- **SmolVLA (do first — ready now, no cloud):** warm-start
+  `lerobot/smolvla_base` in the repository's vendored LeRobot Python 3.11
+  environment; do not train from scratch with `--policy.type=smolvla` on this
+  small dataset. Use the tested install, camera rename/padding, PyAV, and MPS
+  commands in [`SMOLVLA_TRAINING.md`](../SMOLVLA_TRAINING.md). Expect
+  auth/format errors on the first real-data run — fixing them early is the
+  point of starting at episode 10.
 - **MolmoAct (bigger, more setup):** `python -m p5_training.build_dataset_mixture --dataset-root <path>` → `python -m p5_training.modal_finetune --mixture ...`. Runs on **Modal** (needs `pip install modal` + `modal token new` + credits). ⚠️ Runs a **placeholder trainer** until the real `train_cmd` + base checkpoint are set in `p5_training/configs/molmoact2_single_arm.yaml` — those come from the organizers.
 
 **Retrain** checkpoints as data hits 50 / 100 / 150. Log which model looks better.

docs/p5_rerun_port/README.md

@@ -13,6 +13,11 @@ Leaves `p4_data_collection` (native LeRobot record) and `rebot_operator_kit` unt
 
 **Query API (post-record):** step-by-step in [`QUERY_API.md`](./QUERY_API.md) — terminal CLI `python -m p5_rerun_port.query_api_cli` (not inside the Rerun Viewer; Viewer is optional for watching `.rrd` files).
 
+**Viewer workflow:** interactive logging/recording activates a purpose-built
+Blueprint that keeps both synchronized cameras, the 3D reBot URDF,
+goal-versus-position traces, and the time panel visible together. `--no-viewer`
+remains fully headless and creates no Viewer or gRPC sink.
+
 ## How it connects (flowchart)
 
 ![reBot pipeline: shared setup, GUI track, Rerun bounty loop, training](../assets/rebot_rerun_pipeline_flowchart.png)
@@ -101,7 +106,8 @@ python -m p5_rerun_port.export_lerobot --dataset cans --tag "Good episode" --fal
 python -m p5_rerun_port.replay_episode --dataset cans --episode episode_01 --fake --speed 0.5 --no-viewer
 ```
 
-Open any `.rrd` in the Rerun viewer: `rerun recordings/cans/episode_01.rrd`
+Interactive record commands open the purpose-built operator Blueprint. To
+inspect a saved artifact later: `rerun recordings/cans/episode_01.rrd`.
 
 ## Venue (live arm)
 

docs/p5_rerun_port/examples/hackathon_smoke_query_report.md

@@ -0,0 +1,29 @@
+# Rerun Query API report — `hackathon_smoke`
+
+Generated by `python -m p5_rerun_port.query_api_cli`.
+
+> Synthetic evidence only: this report was generated from a `--fake` episode
+> to verify the Rerun Query API path. It is not a physical-arm evaluation or a
+> measured can-picking result.
+
+## Catalog episodes
+
+| dataset | episode | tag | frames | dur_s | task |
+|---|---|---|---:|---:|---|
+| hackathon_smoke | episode_01 | Good episode | 6 | 0.9 | Pick up one can and place it in the taped sorting zone |
+
+## Compare `follower/goal` vs `follower/position`
+
+```
+episode=episode_01  rows=6  rms=2.4474
+joint                 mean|err|   max|err|
+shoulder_pan             2.0088     2.2416
+shoulder_lift            2.1277     2.7888
+elbow_flex               1.6884     2.9247
+wrist_flex               0.4094     1.2612
+wrist_yaw                0.8170     1.2826
+wrist_roll               1.7764     2.6569
+gripper                  4.9601     5.9775
+```
+
+High mean/max error can mean teleop lag, dropped frames, or a bad take — use this to curate before `export_lerobot`.

p5_rerun_port/README.md

@@ -2,4 +2,7 @@
 
 Rerun non-SO-101 bounty port (log / record / Query API / export / replay).
 
-**Docs:** [docs/p5_rerun_port/](../docs/p5_rerun_port/) · Query API: [docs/p5_rerun_port/QUERY_API.md](../docs/p5_rerun_port/QUERY_API.md) · progress: [docs/Rerun_bounty_progress.md](../docs/Rerun_bounty_progress.md)
+**Docs:** [docs/p5_rerun_port/](../docs/p5_rerun_port/) · Query API: [docs/p5_rerun_port/QUERY_API.md](../docs/p5_rerun_port/QUERY_API.md) · demo/pitch: [docs/HACKATHON_SUBMISSION_DEMO.md](../docs/HACKATHON_SUBMISSION_DEMO.md) · progress: [docs/Rerun_bounty_progress.md](../docs/Rerun_bounty_progress.md)
+
+Interactive recordings activate a purpose-built Viewer Blueprint with both
+cameras, the 3D reBot URDF, goal-versus-position traces, and the time panel.