[cli] feat: add benchmark CLI command surface

CONTRIBUTING.md

@@ -44,6 +44,16 @@ The table below lists all development commands. If you installed with **pip**, d
 
 Coverage configuration is in `pyproject.toml` under `[tool.coverage.*]`. CI enforces a minimum coverage threshold of 70%.
 
+When changing remote run commands, preserve the naming convention: bare
+verbs act on a group's primary noun. `train` and `eval` manage runs with
+top-level `submit`, `list`, `info`, `logs`, and `stop` because the run is
+their noun; `benchmark list|info` act on benchmarks themselves (catalog and
+benchmark page), with run lifecycle nested under `osmosis benchmark runs
+list|info|logs|stop|download`. Eval and benchmark downloads
+share the manifest transfer engine in `osmosis_ai/platform/cli/run_download.py`;
+add domain-specific routes and fixed path classifiers instead of copying the
+transfer loop.
+
 ## Linting & Formatting
 
 This project uses [Ruff](https://docs.astral.sh/ruff/) for both linting and code formatting. Configuration lives in `pyproject.toml` under `[tool.ruff]`.

docs/README.md

@@ -31,7 +31,7 @@ The package (`osmosis_ai/`) is organized into top-level domains. See [architectu
 - [datasets.md](./datasets.md) — the dataset row contract enforced by the SDK validator.
 - [troubleshooting.md](./troubleshooting.md) — engineering issues (rollout timeouts, event-loop blocking, concurrency tuning).
 - [cli.md](./cli.md) — CLI internals for contributors (command shells, lazy imports, JSON envelopes).
-- [run-downloads.md](./run-downloads.md) — eval download command, platform route contract, fixed local layout, resume, confirmation, and retry behavior.
+- [run-downloads.md](./run-downloads.md) — eval and benchmark download commands, platform route contracts, fixed local layouts, resume, confirmation, and retry behavior.
 
 ## See also
 

docs/cli.md

@@ -7,7 +7,7 @@
 The console script is `osmosis_ai.cli.main:main` (aliases: `osmosis`, `osmosis-ai`, `osmosis_ai`). [../osmosis_ai/cli/main.py](../osmosis_ai/cli/main.py):
 
 - `main()` calls `_register_commands()` once, then runs the Typer `app` with `standalone_mode=False` so it can map exceptions to exit codes itself.
-- `_register_commands()` imports each command group **lazily inside the function**. Groups attach via `app.add_typer(...)`; the standalone `doctor` / `upgrade` commands attach via `app.command(...)`. Two `rich_help_panel`s split the help: `Workflow Commands` (`dataset`, `train`, `model`, `eval`, `rollout`, `template`, `doctor`) and `Platform Commands` (`auth`, `secret`, `upgrade`).
+- `_register_commands()` imports each command group **lazily inside the function**. Groups attach via `app.add_typer(...)`; the standalone `doctor` / `upgrade` commands attach via `app.command(...)`. Two `rich_help_panel`s split the help: `Workflow Commands` (`dataset`, `train`, `model`, `eval`, `benchmark`, `rollout`, `template`, `doctor`) and `Platform Commands` (`auth`, `secret`, `upgrade`).
 - The root `_callback` resolves `--json` / `--plain`, builds an `OutputContext`, installs it on the Typer context, registers `verify_output_emitted` on close, and loads `.env` via `python-dotenv`. `hoist_format_selectors` lets the format flags appear anywhere on the line.
 
 ## Command shells delegate; they don't do work
@@ -19,6 +19,16 @@ Files in [../osmosis_ai/cli/commands/](../osmosis_ai/cli/commands/) are thin Typ
 
 Module-level imports in `commands/` are kept light: `typer`, `cli.console`, `cli.errors`, the lightweight `osmosis_ai.platform.constants` (pagination limits), and stdlib. Everything heavy (`rollout.*`, `platform.api.*`, `platform.cli.*`, `eval.*`) must be imported **inside the function** to keep CLI startup fast — see the lazy-loading section of [architecture.md](./architecture.md).
 
+`osmosis benchmark` puts the benchmark first, mirroring the platform's
+Benchmarks pages: top-level `list` and `info` act on benchmarks (the catalog
+and one benchmark's page - `info` shows its metadata, leaderboard, and runs),
+`submit` starts a run, and run lifecycle lives under the nested `benchmark
+runs list|info|logs|stop|download` namespace.
+
+Catalog output includes a shell-safe benchmark `Key`, such as
+`terminal-bench-2-1`. Pass that key to `osmosis benchmark info <key>`;
+exact display names and UUIDs remain supported for compatibility.
+
 ## Commands return results; they don't print
 
 The Typer app is created with `result_callback=render_command_result` ([../osmosis_ai/cli/main.py](../osmosis_ai/cli/main.py)). A command function **returns** a `CommandResult`; the callback renders it in the active format. Do not `print()` from a command — return a typed result instead.

docs/run-downloads.md

@@ -1,6 +1,6 @@
-# Evaluation run output download contract
+# Run output download contracts
 
-The eval download implementation lives in [`platform/cli/run_download.py`](../osmosis_ai/platform/cli/run_download.py). Its Typer shell remains thin and lives in [`cli/commands/eval.py`](../osmosis_ai/cli/commands/eval.py).
+The shared manifest-to-disk engine lives in [`platform/cli/run_download.py`](../osmosis_ai/platform/cli/run_download.py). Eval and benchmark handlers provide their route loader, fixed path classifier, output-root resolver, operation name, and resource key; they do not duplicate transfer behavior.
 
 ## Commands
 
@@ -11,9 +11,15 @@ osmosis eval download NAME_OR_ID
   -o, --output ROOT
   --overwrite
   -y, --yes
+
+osmosis benchmark runs download NAME_OR_ID
+  --type summary,results|artifacts|logs|all
+  -o, --output ROOT
+  --overwrite
+  -y, --yes
 ```
 
-`--type` replaces the default selection and defaults to `metrics,trajectories`. A row selection includes every run for each selected row.
+`--type` replaces the default selection. Eval defaults to `metrics,trajectories`; benchmark defaults to `summary,results`. Eval row selection includes every run for each selected row.
 
 ## Platform routes
 
@@ -24,6 +30,13 @@ GET  /api/cli/eval-runs/[id]/samples/manifest?types=&rows=
 POST /api/cli/eval-runs/[id]/samples/download-urls
 ```
 
+Benchmark downloads use the parallel output routes:
+
+```text
+GET  /api/cli/benchmark-runs/[id]/outputs/manifest?types=
+POST /api/cli/benchmark-runs/[id]/outputs/download-urls
+```
+
 The manifest returns `{files: [{token?, path, size}], totals}`. `path` is the final path relative to the local run root, and `token` is an opaque server handle (a rollout id or an export snapshot token). URL requests contain at most 500 `{token, path}` items. The platform derives full S3 keys server-side and returns 15-minute presigned GET URLs; the SDK never accepts raw object keys.
 
 ## Fixed local layout
@@ -36,10 +49,15 @@ The manifest returns `{files: [{token?, path, size}], totals}`. `path` is the fi
 │   ├── trajectories/row_3_run_0.json
 │   ├── artifacts/row_3_run_0/logs/agent.log
 │   └── logs.txt
+├── benchmarks/<run-name>/
+│   ├── summary.csv
+│   ├── results.csv
+│   ├── artifacts/<stable-result-identity>/<artifact-path>
+│   └── logs.txt
 └── metrics/                       # legacy eval exports; never deleted
 ```
 
-`--output` relocates the run root; filenames and subdirectories below it do not change. Rich-mode `eval info` uses the same resolver and writes the same run-scoped `metrics.json` path. Names that require filesystem sanitization gain a stable run-ID suffix so distinct runs never share a local directory. Training download and training metrics-path migration are intentionally out of scope until the platform routes are ready; `train info` keeps its existing export behavior for now.
+`--output` relocates the run root; filenames and subdirectories below it do not change. Rich-mode `eval info` uses the same resolver and writes the same run-scoped `metrics.json` path. Names that require filesystem sanitization gain a stable run-ID suffix so distinct runs never share a local directory. Each domain supplies a strict path classifier: an eval manifest cannot write benchmark filenames and a benchmark manifest cannot write eval filenames. Training download and training metrics-path migration are intentionally out of scope until the platform routes are ready; `train info` keeps its existing export behavior for now.
 
 ## Transfer behavior
 

osmosis_ai/cli/commands/benchmark.py

@@ -0,0 +1,179 @@
+"""Benchmark catalog and run management commands."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Any
+
+import typer
+
+from osmosis_ai.platform.constants import (
+    DEFAULT_PAGE_SIZE,
+    MAX_LOG_PAGE_SIZE,
+    MAX_PAGE_SIZE,
+)
+
+app: typer.Typer = typer.Typer(
+    help="Manage benchmarks and their runs.",
+    no_args_is_help=True,
+)
+runs_app: typer.Typer = typer.Typer(
+    help="Manage benchmark runs.",
+    no_args_is_help=True,
+)
+app.add_typer(runs_app, name="runs")
+
+
+@app.command("list")
+def benchmark_list(
+    limit: int = typer.Option(
+        DEFAULT_PAGE_SIZE,
+        "--limit",
+        min=1,
+        max=MAX_PAGE_SIZE,
+        help="Maximum number of benchmarks to show.",
+    ),
+    all_: bool = typer.Option(False, "--all", help="Show all benchmarks."),
+) -> Any:
+    """List benchmarks available in the current workspace."""
+    from osmosis_ai.platform.cli.benchmark import list_benchmarks as _list_benchmarks
+
+    return _list_benchmarks(limit=limit, all_=all_)
+
+
+@app.command("info")
+def benchmark_info(
+    name_or_id: str = typer.Argument(..., help="Benchmark key, name, or ID."),
+    limit: int = typer.Option(
+        DEFAULT_PAGE_SIZE,
+        "--limit",
+        min=1,
+        max=MAX_PAGE_SIZE,
+        help="Maximum number of runs to show in the runs section.",
+    ),
+    all_: bool = typer.Option(False, "--all", help="Show all of the benchmark's runs."),
+) -> Any:
+    """Show a benchmark: metadata, task options, leaderboard, and runs."""
+    from osmosis_ai.platform.cli.benchmark import benchmark_info as _info
+
+    return _info(name_or_id, limit=limit, all_=all_)
+
+
+@app.command("submit")
+def benchmark_submit(
+    config_path: Path = typer.Argument(
+        ...,
+        exists=False,
+        file_okay=True,
+        dir_okay=False,
+        readable=False,
+        resolve_path=False,
+        help="Path to benchmark config TOML file.",
+    ),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt."),
+) -> Any:
+    """Submit a benchmark run."""
+    from osmosis_ai.platform.cli.benchmark import submit as _submit
+
+    return _submit(config_path, yes=yes)
+
+
+@runs_app.command("list")
+def benchmark_runs_list(
+    limit: int = typer.Option(
+        DEFAULT_PAGE_SIZE,
+        "--limit",
+        min=1,
+        max=MAX_PAGE_SIZE,
+        help="Maximum number of benchmark runs to show.",
+    ),
+    all_: bool = typer.Option(False, "--all", help="Show all benchmark runs."),
+) -> Any:
+    """List benchmark runs for the current workspace directory."""
+    from osmosis_ai.platform.cli.benchmark import list_benchmark_runs as _list
+
+    return _list(limit=limit, all_=all_)
+
+
+@runs_app.command("info")
+def benchmark_runs_info(
+    name_or_id: str = typer.Argument(..., help="Benchmark run name or ID."),
+) -> Any:
+    """Show benchmark run details, progress, and results."""
+    from osmosis_ai.platform.cli.benchmark import run_info as _info
+
+    return _info(name_or_id)
+
+
+@runs_app.command("logs")
+def benchmark_runs_logs(
+    name_or_id: str = typer.Argument(..., help="Benchmark run name or ID."),
+    limit: int = typer.Option(
+        DEFAULT_PAGE_SIZE,
+        "--limit",
+        min=1,
+        max=MAX_LOG_PAGE_SIZE,
+        help="Maximum number of recent log entries to show.",
+    ),
+    cursor: str | None = typer.Option(
+        None,
+        "--cursor",
+        help="Page further back using the next_cursor value from a previous page.",
+    ),
+) -> Any:
+    """Show recent logs for a benchmark run, oldest first."""
+    from osmosis_ai.platform.cli.benchmark import logs as _logs
+
+    return _logs(name_or_id, limit=limit, cursor=cursor)
+
+
+@runs_app.command("stop")
+def benchmark_runs_stop(
+    name_or_id: str = typer.Argument(..., help="Benchmark run name or ID."),
+    yes: bool = typer.Option(False, "--yes", "-y", help="Skip confirmation prompt."),
+) -> Any:
+    """Stop a benchmark run."""
+    from osmosis_ai.platform.cli.benchmark import stop as _stop
+
+    return _stop(name_or_id, yes=yes)
+
+
+@runs_app.command("download")
+def benchmark_runs_download(
+    name_or_id: str = typer.Argument(..., help="Benchmark run name or ID."),
+    output: str | None = typer.Option(
+        None,
+        "--output",
+        "-o",
+        help="Run output root (default: .osmosis/benchmarks/<run-name>/).",
+    ),
+    types: str = typer.Option(
+        "summary,results",
+        "--type",
+        help=(
+            "Comma-separated selector: summary, results, artifacts, logs, all. "
+            "Replaces the default selection."
+        ),
+    ),
+    overwrite: bool = typer.Option(
+        False,
+        "--overwrite",
+        help="Re-download files that already exist locally.",
+    ),
+    yes: bool = typer.Option(
+        False,
+        "--yes",
+        "-y",
+        help="Skip size confirmation.",
+    ),
+) -> Any:
+    """Download benchmark run summary, results, artifacts, or logs."""
+    from osmosis_ai.platform.cli.benchmark import download as _download
+
+    return _download(
+        name_or_id,
+        output=output,
+        types=types,
+        overwrite=overwrite,
+        yes=yes,
+    )

osmosis_ai/cli/commands/eval.py

@@ -138,7 +138,7 @@ def eval_info(
         None,
         "--output",
         "-o",
-        help="Run output root (default in rich mode: .osmosis/evals/<name>/).",
+        help="Run output root (default in rich mode: .osmosis/evals/<run-name>/).",
     ),
 ) -> Any:
     """Show evaluation run details, results, and metrics."""
@@ -154,7 +154,7 @@ def eval_download(
         None,
         "--output",
         "-o",
-        help="Run output root (default: .osmosis/evals/<name>/).",
+        help="Run output root (default: .osmosis/evals/<run-name>/).",
     ),
     types: str = typer.Option(
         "metrics,trajectories",

osmosis_ai/cli/main.py

@@ -187,6 +187,7 @@ def _register_commands() -> None:
     get_completion_inspect_parameters()
     # -- Command groups --
     from osmosis_ai.cli.commands.auth import app as auth_app
+    from osmosis_ai.cli.commands.benchmark import app as benchmark_app
     from osmosis_ai.cli.commands.dataset import app as dataset_app
     from osmosis_ai.cli.commands.eval import app as eval_app
     from osmosis_ai.cli.commands.model import app as model_app
@@ -202,6 +203,7 @@ def _register_commands() -> None:
     app.add_typer(train_app, name="train", rich_help_panel=_WORKFLOW)
     app.add_typer(model_app, name="model", rich_help_panel=_WORKFLOW)
     app.add_typer(eval_app, name="eval", rich_help_panel=_WORKFLOW)
+    app.add_typer(benchmark_app, name="benchmark", rich_help_panel=_WORKFLOW)
     app.add_typer(rollout_app, name="rollout", rich_help_panel=_WORKFLOW)
     app.add_typer(template_app, name="template", rich_help_panel=_WORKFLOW)
 

osmosis_ai/cli/metrics_export.py

@@ -83,18 +83,18 @@ def resolve_default_metrics_output(
     return metrics_dir / default_metrics_filename(run_name, run_id)
 
 
-def resolve_eval_output_dir(
+def _resolve_run_output_dir(
     run_name: str | None,
     run_id: str,
     *,
+    scope: str,
     workspace_directory: Path,
     output: str | None = None,
     create: bool = True,
 ) -> Path:
-    """Resolve the fixed eval-scoped output root used by ``info`` and download.
+    """Resolve a fixed run-scoped output root under ``.osmosis/<scope>``.
 
     ``output`` relocates the run root itself; the layout below it is fixed.
-    Without an explicit root, evals live under ``.osmosis/evals/<name>``.
     Unnamed runs fall back to their full ID so separate runs never share the
     unnamed directory.
     """
@@ -115,7 +115,7 @@ def resolve_eval_output_dir(
             safe_run_id = safe_name(run_id).strip("_") or "run"
             suffix = f"--{safe_run_id}"
             safe_run_name = f"{safe_run_name[: 255 - len(suffix)]}{suffix}"
-        root = workspace_directory / ".osmosis" / "evals" / (safe_run_name or run_id)
+        root = workspace_directory / ".osmosis" / scope / (safe_run_name or run_id)
 
     try:
         if root.exists() and not root.is_dir():
@@ -127,6 +127,44 @@ def resolve_eval_output_dir(
     return root
 
 
+def resolve_eval_output_dir(
+    run_name: str | None,
+    run_id: str,
+    *,
+    workspace_directory: Path,
+    output: str | None = None,
+    create: bool = True,
+) -> Path:
+    """Resolve the fixed eval-scoped output root used by ``info`` and download."""
+    return _resolve_run_output_dir(
+        run_name,
+        run_id,
+        scope="evals",
+        workspace_directory=workspace_directory,
+        output=output,
+        create=create,
+    )
+
+
+def resolve_benchmark_output_dir(
+    run_name: str | None,
+    run_id: str,
+    *,
+    workspace_directory: Path,
+    output: str | None = None,
+    create: bool = True,
+) -> Path:
+    """Resolve the fixed benchmark-scoped output root used by download."""
+    return _resolve_run_output_dir(
+        run_name,
+        run_id,
+        scope="benchmarks",
+        workspace_directory=workspace_directory,
+        output=output,
+        create=create,
+    )
+
+
 def resolve_eval_metrics_output(
     run_name: str | None,
     run_id: str,