Harbor: strict harbor/terminal-bench format compliance + e2b sandbox support #331

Goal

Make ClawBench a first-class Harbor benchmark, strictly following the conventions used by harbor-framework/harbor and harbor-framework/terminal-bench, including running on Harbor's remote sandboxes (-e e2b) — not just local Docker.

Today clawbench-harbor-adapt (added in v0.7.0, #232) produces Harbor tasks that load fine on Harbor 0.22, but we are not yet aligned with the ecosystem end-to-end.

Requirements

1. Strict task format (terminal-bench parity)

• Canonical layout per task: task.toml ([task] name = "<org>/<name>", authors, keywords), instruction.md, environment/ (or [environment].docker_image), tests/test.sh → numeric /logs/verifier/reward.txt + reward.json, solution/solve.sh (documented no-op — live sites have no scripted oracle)
• dataset.toml manifest with per-task digests (harbor add --scan / Packager content hash)
• Every task loads with Harbor TaskConfig in CI; committed dataset regenerates deterministically (scripts/harbor/regenerate.sh, validate-harbor.yml — on branch feat/harbor-hub-registry)
• Numeric-only reward.json metrics (depends on fix(harbor): numeric verifier metrics, lenient + strict judge rubrics, and interception correctness #324 — should merge first)

2. Registry + Hub (like terminal-bench)

• Root registry.json so harbor run --repo TIGER-AI-Lab/ClawBench -d clawbench-v2 works
• Publish tiger-ai-lab/clawbench-v2 to the Harbor Hub (harbor publish --public), tags per release
• Prebuilt runtime image on GHCR (clawbench-harbor-runtime:<version>) so tasks ship no build context

3. e2b (and other remote sandbox) support

• harbor run ... -e e2b completes a smoke task end-to-end (also verify -e daytona/modal are not broken)
• No dependency on local podman/docker specifics: image pullable from a public registry, no bind mounts outside the task dir, healthcheck-driven readiness (Chromium + CDP + interceptor)
• Judge + PurelyMail credentials passed via --ve/--env-file only; document resource floor (1 CPU / 2 GB per trial)
• Prior note: docs/harbor-0.20-port-notes.md (branch feat/harbor-0.20) verified the E2B path needs no local disk — fold conclusions into docs/harbor.md

4. Upstream adapter

• adapters/clawbench/ package in the official harbor adapter init layout (README template sections, adapter_metadata.json, parity_experiment.json, run_clawbench.yaml, src/.../{adapter,main}.py with --output-dir --limit --overwrite --task-ids)
• Parity experiment: same agent both sides (native clawbench-batch vs harbor run), ≥3 runs/side, mean ± sample SEM, ranges overlap
• PRs: [WIP] Adapter: clawbench → harbor-datasets datasets/clawbench/ → harbor registry.json → [Ready for Review]

Starting point

Branch feat/harbor-hub-registry already implements most of §1–§2 and the adapter package skeleton (129 prebuilt-mode tasks, 7 MB, all pass Harbor 0.22 TaskConfig; digests match harbor add). §3 (e2b) and the parity run in §4 are the open work.

Review

@Perry2004 — you built the original Harbor compatibility layer in #232; please review the format decisions here (strict layout, prebuilt image, registry/Hub split) and the e2b plan.