[cli] feat: add benchmark CLI command surface

[JoyboyBrian]

What

• Add the benchmark run lifecycle: submit, list, info, logs, stop, download, matching the train/eval command surface.
• Move benchmark-definition discovery to benchmark catalog list|info so top-level list/info unambiguously mean runs.
• Add osmosis_ai/platform/cli/benchmark_config.py: a strict Pydantic parser for the Osmosis-owned benchmark TOML contract ([experiment], [tasks], [[agents]], [execution], [env]), with per-agent secret/env collision checks and reserved _OSMOSIS_ rejection.
• Generalize osmosis_ai/platform/cli/run_download.py into a shared manifest transfer engine parameterized by output resolver, path classifier, operation and resource key; eval behavior is the default, so its call sites are unchanged.
• Downloads default to summary,results; artifacts, logs and all are opt-in. Fixed layout under .osmosis/benchmarks/<run-name>/.
• Surface required_secret_names and nullable per-task difficulty in catalog output.
• Register benchmark as a structured-error command group so JSON errors report the full command path (benchmark download, benchmark catalog info) instead of just benchmark.
• Two shared-config changes reach beyond benchmark: validation_issue_to_config_issue now builds a correct issue key when the section name is empty, and validate_env_var_keys takes a source_label so per-agent env errors name the right section. Both keep their previous behavior for existing callers.
• Add typed API models and client methods, Rich/JSON lifecycle output, and CLI, API-client, lifecycle, download and structured-error tests.

Why

Benchmarks were only manageable from the Platform UI, so submit left users with no CLI path to monitor, stop, or retrieve results. Runs now follow the same lifecycle mental model as train and eval.

Catalog commands are nested rather than top-level because a benchmark definition and a benchmark run are different objects; sharing list/info between them would make the command surface ambiguous as soon as run management existed.

The config is an Osmosis-owned contract, not a Harbor config: Harbor stays an internal execution detail, and the CLI never sees or transmits secret values — api_key_secret fields and required_secret_names carry Platform secret record names that the platform resolves server-side.

Secret/env collision checks are scoped per agent because the platform injects each agent's API key only into that agent's environment, so one agent's secret name may legitimately be another agent's literal env var.

difficulty is passed through verbatim and is null when the benchmark source does not provide one; it is never inferred.

Depends on the platform endpoints added in Osmosis-AI/osmosis-monolith#936.

How to Test

• uv run pytest
• uv run ruff check . and uv run ruff format --check .
• uv run pyright osmosis_ai/
• From a workspace repository: osmosis benchmark catalog info "HLE", osmosis benchmark submit configs/benchmark/default.toml, then list / info / logs / download

Checklist

• PR title follows [module] type: description format
• Appropriate labels added (e.g. enhancement, bug, breaking)
• ruff check . and ruff format --check . pass
• pyright osmosis_ai/ passes
• pytest passes (new tests added if applicable)
• Public API changes are documented
• No secrets or credentials included

[codecov]

Codecov Report

❌ Patch coverage is 94.28571% with 14 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
osmosis_ai/platform/api/models.py	93.90%	4 Missing and 8 partials ⚠️
osmosis_ai/platform/api/client.py	94.28%	1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

[cubic-dev-ai]

All reported issues were addressed across 11 files

<sub>Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.

Fix all with cubic | Re-trigger cubic</sub>

[JoyboyBrian]

@cubic review

[cubic-dev-ai]

@cubic review

@JoyboyBrian I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 13 files

_{Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.}

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

All reported issues were addressed across 8 files (changes from recent commits).

_{Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.}

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[JoyboyBrian]

@cubic review

[cubic-dev-ai]

@cubic review

@JoyboyBrian I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

1 issue found across 13 files

Confidence score: 2/5

• In osmosis_ai/platform/cli/benchmark_config.py, the header guard only blocks Authorization, so users can still commit or transmit secrets via X-API-Key or Proxy-Authorization; this creates a concrete credential-leak risk in config and CLI traffic—expand validation to reject credential-bearing headers (or require secret-backed references) before merge.

_{Heads up: you’re close to your included review allowance. Set a flex budget so reviews don’t pause.}

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="osmosis_ai/platform/cli/benchmark_config.py">

<violation number="1" location="osmosis_ai/platform/cli/benchmark_config.py:72">
P1: Endpoint credentials can still be committed and sent through the CLI as `X-API-Key` or `Proxy-Authorization`, because this guard blocks only `Authorization`. Reject credential-bearing headers (or support secret-backed header references) so `api_key_secret` remains the only authentication input.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>