feat: vLLM benchmark harness by binaryaaron · Pull Request #527 · NVIDIA-NeMo/Safe-Synthesizer

binaryaaron · 2026-05-27T15:07:18Z

Summary

Adds a developer vLLM benchmark harness that replays captured generation traces against preset or custom candidate configurations.
Runs each candidate in a fresh subprocess and records throughput, acceptance, TTFT, startup overlap, vLLM/runtime observability, skip records, and optional WandB sink metrics.
Adds preset matrices for backend, batching, prefix-cache, max-length, bracketed A/B, n_fanout, speculative n-gram, and fp8 comparisons.
Adds compare/analyze CLI commands, cluster-conditioned aggregates, Welch CI reporting, and developer-guide usage docs.

Test plan

mise run check
uv run --frozen --extra cu129 --extra engine --group dev pytest tests/generation/test_vllm_benchmark_cli.py tests/generation/test_vllm_benchmark.py tests/generation/test_vllm_benchmark_analysis.py
mise run docs:build
git diff --check
GPU smoke: ran one local tiny-model candidate on an A100, validated BenchmarkOutput, and inspected populated observability fields.

Notes

PR feat: expand vLLM observability/telemetry #526 is merged; this PR consumes its vLLM observability helpers through GenerationObservability.
Analyzer JSON and CLI names use candidate-run terminology: n_candidate_runs and --min-runs-per-condition.

coderabbitai · 2026-05-27T15:07:38Z

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9a37aa5f-0845-41ca-95ae-2052d845420e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Commit unit tests in branch binaryaaron/vllm-benchmark-harness-work

_{Comment @coderabbitai help to get the list of available commands.}

mckornfield · 2026-06-04T22:33:09Z

+    )
+    structured_generation_backend: str = Field(
+        default="xgrammar",
+        description="Structured-outputs backend. ``'xgrammar'`` is vLLM's current default; ``'outlines'`` and ``'guidance'`` are the alternatives.",


I remember there were a few alternatives... should we nix those at this point? maybe not for this pr though

mckornfield · 2026-06-04T22:36:15Z

+        if corpus.prompts:
+            base_sampling = corpus.prompts[0].original_sampling_params
+        else:
+            base_sampling = {}


Suggested change

if corpus.prompts:

base_sampling = corpus.prompts[0].original_sampling_params

else:

base_sampling = {}

base_sampling = {}

if corpus.prompts:

base_sampling = corpus.prompts[0].original_sampling_params

nitty but if there's any like that, that's my preferred assuming assignment to an empty guy is pretty cheap

mckornfield · 2026-06-04T22:39:24Z

+            lines.append(f"### `{agg.condition_label}` (n={agg.n_cells})")
+            lines.append("")
+            lines.append(
+                f"- **Pooled**: eff_tok_s={agg.pooled_mean_effective_tok_s:.1f} "


oo these lines are spicy

mckornfield · 2026-06-04T22:39:41Z

+
+
+class ConditionClusterAggregate(BaseModel):
+    """Per-(condition × cluster) aggregate: in-cluster condition stats."""


do we want to use these non ascii characters (I saw a delta above too)

mckornfield · 2026-06-04T22:53:55Z

+# ---------------------------------------------------------------------------
+
+
+def load_cells(output_dir: Path) -> list[CandidateMetrics]:


just +1-ing my confusion on the word cell from the other PR

mckornfield · 2026-06-04T22:54:57Z

+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Benchmark-side wandb integration — per-cell new-run mode.


does make me think this is benchmarking wandb, but it's just using wandb as a sink?

mckornfield · 2026-06-04T22:55:45Z

+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+r"""vllm-benchmark: drive the vLLM benchmark harness from the command line.


should this have a README on usage, or is this enough?

codecov · 2026-06-15T21:11:53Z

Codecov Report

❌ Patch coverage is 89.85623% with 127 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
tools/vllm_benchmark.py	69.44%	44 Missing ⚠️
...afe_synthesizer/generation/vllm_benchmark_wandb.py	27.27%	32 Missing ⚠️
...nemo_safe_synthesizer/generation/vllm_benchmark.py	92.30%	23 Missing ⚠️
..._synthesizer/generation/vllm_benchmark_analysis.py	92.14%	19 Missing ⚠️
...e_synthesizer/generation/vllm_benchmark_presets.py	93.10%	6 Missing ⚠️
tests/generation/test_vllm_benchmark.py	99.30%	2 Missing ⚠️
tests/generation/test_vllm_benchmark_cli.py	98.71%	1 Missing ⚠️

📢 Thoughts on this report? Let us know!

+        init_start = time.monotonic()
+        try:
+            result.llm = vLLM(**kwargs)
+        except BaseException as exc:  # noqa: BLE001 - surface via the result object


+from pydantic import ValidationError
+
+import nemo_safe_synthesizer.generation.processors as processors_mod
+import nemo_safe_synthesizer.generation.vllm_benchmark as benchmark_mod


Adds the pydantic schemas the benchmark harness uses for corpus loading, candidate description, and result serialization. Pure data-models commit; the runner + subprocess wrapper land in the next commit. ## Models - ``TraceHeader``, ``BenchmarkPrompt``, ``BenchmarkCorpus`` — the replayable input schema. ``BenchmarkCorpus.from_trace_jsonl(path)`` loads a header-then-records JSONL into a typed corpus. - ``BenchmarkEngineConfig`` — sparse pydantic model holding vLLM constructor kwargs the harness will forward (attention backend, prefix caching, scheduler caps, kv_cache_dtype, speculative_config, compilation_config, kv_cache_metrics, etc.). Every field is optional and falls through to vLLM defaults when unset. - ``BenchmarkCandidate`` — one configuration to benchmark: name + engine config + sparse sampling overrides + prompt-assembly / batch-dispatch modes. - ``CandidateMetrics`` — per-cell measurements. Cell-specific fields (throughput, acceptance, TTFT, finish reasons, startup) live directly on the model; observability primitives (peak VRAM, KV cache fraction, loadavg, engine runtime config) compose ``CellObservability`` from PR-A's ``vllm_observability`` module. - ``SkipRecord``, ``BenchmarkOutput`` — failure record + matrix result wrapper. ## Composition over duplication ``CandidateMetrics.observability: CellObservability`` is the key DRY mechanism. Instead of re-declaring ``peak_vram_gb`` / ``kv_cache_usage_perc`` / ``loadavg_pre`` / etc. on the benchmark schema (the harness-v2 shape), the benchmark schema *embeds* the production observability event. Downstream consumers can: - Read ``metrics.observability.peak_vram_gb`` for the GPU number. - Read ``metrics.observability.engine_runtime_config`` for the scheduler/cache state. - Call ``metrics.observability.to_wandb_payload()`` to get a flat namespaced dict for wandb logging. Adding a new observability primitive in PR-A automatically flows through to benchmark output without touching this module. ## Type aliases ``PromptAssemblyMode`` and ``BatchDispatchMode`` are exported as ``Literal[...]`` aliases so callers (presets, CLI) can reference them by name without re-deriving the allowed values. ## What's NOT in this commit - The runner (``run_benchmark``). It lands next. - The subprocess wrapper. Lands with the runner. - Presets (``vllm_benchmark_presets``). Separate commit. - Wandb integration (``vllm_benchmark_wandb``). Separate commit. - CLI driver (``tools/vllm_benchmark.py``). Separate commit. - Bracketed-A/B methodology (``condition_label``, ``bracket_position`` on the candidate/metrics schemas). Separate commit; the schema additions are small but the methodology change is conceptually distinct. - Cluster-conditioned analyzer + effect-size + CI. Separate commits. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Adds the core ``run_benchmark`` function + supporting helpers to ``vllm_benchmark.py``. The runner builds a fresh vLLM engine per candidate, replays the corpus prompts through one concurrent ``LLM.generate`` call, and returns a ``CandidateMetrics`` with the composed ``CellObservability`` populated. ## Helpers added - ``_merge_sampling_kwargs`` — composes vLLM ``SamplingParams`` kwargs from the corpus default + candidate overrides; strips capture-time metadata fields (``structured_outputs`` summary) that ``SamplingParams`` doesn't accept. - ``_percentile`` — linear-interpolation percentile (``0.0`` on empty input); used for TTFT p50/p99. - ``_extract_ttft_ms`` — reads per-request first-token latency off ``RequestOutput.metrics``. Tries ``first_token_latency`` (modern, seconds) first; falls back to ``first_token_time - arrival_time`` for older captures. Queue-inclusive under batched submission (the docstring on ``CandidateMetrics.ttft_p50_ms`` flags this explicitly). - ``_build_vllm_kwargs`` — composes ``vllm.LLM(...)`` kwargs from the corpus header (model, LoRA, engine_parameters at capture) + candidate overlay (sparse ``BenchmarkEngineConfig`` fields). Translates ``attention_backend`` → ``attention_config={"backend": ...}`` and ``structured_generation_backend`` → ``structured_outputs_config=StructuredOutputsConfig(backend=...)``. Drops ``None``-valued fields explicitly. - ``_build_engine_async`` — starts ``vllm.LLM(...)`` on a daemon thread, returns ``(thread, ready_event, result_dict)``. The runner can sleep for the simulated-training-overlap window before joining via ``ready_event.wait()``, measuring how much engine-init cost can be hidden behind concurrent training. ## ``run_benchmark`` shape Single function. Wraps the entire body in: - ``NvmlPeakSampler`` context (from PR-A) — peak device VRAM across the whole cell. - ``read_loadavg()`` pre + post (from PR-A) — bracketing host load capture. - ``probe_engine_runtime_config(llm)`` (from PR-A) — captured once after engine init, reused for the observability event. - ``flag_engagement_mismatches(intended, actual)`` (from PR-A) — cross-checks candidate's ``engine_config.model_dump(exclude_none=True)`` against the probe; sets ``flag_did_not_engage`` and logs a warning when mismatches fire. End-of-generation: - ``read_vllm_runtime_metrics(llm)`` (from PR-A) — KV cache, prefix cache, spec accept. - Build :class:`CellObservability` event with all measurements. - Return :class:`CandidateMetrics` with the event composed in. Cleanup runs in ``finally`` so the sampler shuts down + the partial observability event still builds if generation raised mid-batch. ## What's NOT in this commit - Subprocess isolation wrapper (``run_benchmark_in_subprocess``). Lands next so each candidate gets a fresh CUDA context. - The ``vllm_benchmark_single_run`` subprocess entry point. Lands with the wrapper. - ``n_fanout`` dispatch mode. The runner currently dispatches all prompts as independent requests (``n=1`` per request). n_fanout needs the SamplingParams ``n=N`` path; landing in a follow-up. - Streaming step-loop with mid-batch abort (the PR-3-style stopping conditions). Out of scope for now; the synchronous dispatch suffices for the standard benchmark-sliver pattern. - Presets (``vllm_benchmark_presets.py``). Separate commit. - Wandb integration. Separate commit. - CLI driver. Separate commit. ## Dependencies on PR-A This commit lifts the architectural smell I flagged earlier: observability primitives live in PR-A's ``vllm_observability``, the benchmark schema *composes* them rather than re-defining them. The runner imports five symbols from PR-A: - ``NvmlPeakSampler`` - ``CellObservability`` - ``read_loadavg`` - ``probe_engine_runtime_config`` - ``read_vllm_runtime_metrics`` - ``flag_engagement_mismatches`` No PR-1 dependency. The vLLM engine is constructed via direct kwargs (via ``_build_vllm_kwargs``), not via PR-1's ``build_vllm_engine`` factory — keeps PR-B mergeable independently of PR-1's review timing. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Each candidate runs in a fresh Python interpreter via ``python -m nemo_safe_synthesizer.generation.vllm_benchmark_single_run`` so vLLM's module-level CUDA + DRAM state is reclaimed by the OS between candidates. The in-process Python runtime can't clean up vLLM's globals reliably; subprocess isolation is the only way to run a multi-candidate matrix without state bleed. Adds to ``vllm_benchmark.py``: - ``SUBPROCESS_STDERR_LIMIT = 500`` — captured-stderr cap. - ``_truncate_stderr`` — trim with a tail-preserving sentinel. - ``_parse_error_class`` — best-effort exception class extraction from a Python traceback. - ``SubprocessRunResult`` — pydantic model carrying either ``metrics`` (on success) or ``error`` + ``error_class`` (on failure). - ``run_benchmark_in_subprocess(candidate, corpus_path, ...)`` — spawns the child, reads back the JSON-serialised ``CandidateMetrics``, returns ``SubprocessRunResult``. New module ``vllm_benchmark_single_run.py``: - argparse-based entry point: ``--candidate`` (JSON), ``--corpus`` (path), ``--result-out`` (path), ``--simulate-training-overlap-seconds`` (float). - Loads the candidate + corpus, calls ``run_benchmark``, writes the resulting ``CandidateMetrics`` JSON. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Adds ``vllm_benchmark_presets.py`` — the named preset matrices the CLI driver exposes via ``--candidates <preset_name>``. Each preset takes the corpus's default ``BenchmarkEngineConfig`` and returns one or more ``BenchmarkCandidate``s with the appropriate engine overrides applied. ## Methodology constants - ``DEFAULT_BENCHMARK_SEED = 42`` — every preset-built candidate pins ``SamplingParams.seed`` to this. Prior triages observed acceptance-rate CoV up to ~14% from per-request RNG drift; seed-pinning collapses that to in-cluster noise (~3% CoV; ~5% pooled on long-output workloads). - ``DEFAULT_MAX_MODEL_LEN = 4096`` — fallback context-window cap when the corpus header doesn't specify. Stops vLLM from over-provisioning the KV cache to the tokenizer's reported max (32768 on Mistral-7B). ## Sweep matrices - ``baseline`` — pass-through. - ``attention_backend_sweep`` — ``FLASHINFER``, ``FLASH_ATTN``, ``TRITON_ATTN``. - ``prefix_caching_sweep`` — single ``prefix_caching=on`` candidate (the implicit baseline is off). - ``batching_sweep`` — (max_num_seqs, max_num_batched_tokens) = (128, 4096), (256, 8192), (512, 16384). - ``structured_backend_sweep`` — ``xgrammar``, ``outlines``, ``guidance``. - ``max_model_len_sweep`` — 2048, 4096, 8192. - ``default_matrix`` — concatenation of all sweeps, deduplicated by (engine_config, sampling_overrides) — two candidates with identical config but different names are treated as duplicates because the engine + sampling combo is what the runner actually measures. ## What's NOT in this commit - ``bracketed_ab`` family (the methodology presets that interleave baseline + candidate cells with seed-pinned bracketing). Separate commit; depends on the schema additions for ``condition_label`` + ``bracket_position`` on ``BenchmarkCandidate`` / ``CandidateMetrics``. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

…-run) Adds ``vllm_benchmark_wandb.py`` — the benchmark-side wandb wiring. Each benchmark cell becomes one wandb run, grouped by sweep ID and tagged by condition_label + dataset + 'benchmark' job_type. ## Distinct from PR-A's production wandb pattern PR-A's ``log_cell_observability(event)`` logs to the *currently active* wandb run (production logs cells as a time-series within one ``safe-synthesizer run generate`` invocation). This module's ``init_cell_run`` opens a *new* wandb run per benchmark cell so each candidate is its own row in the wandb UI — appropriate for sweep-style isolation where you want per-condition aggregates. Both patterns coexist because they're the right shape for different use cases. They reuse PR-A's ``WandbSettings`` so env-var handling (``WANDB_MODE``, ``WANDB_PROJECT`` / ``NSS_WANDB_PROJECT``) is consistent. ## API - ``resolve_sweep_id() -> str`` — ``WANDB_RUN_GROUP`` env when set; auto-generated timestamp otherwise. - ``init_cell_run(*, candidate_name, candidate_idx, total, corpus_run_id, corpus_size, sweep_id, candidate_condition_label='', candidate_bracket_position=0)`` — opens a wandb run; returns None when disabled/unavailable. Precedence for label + position: candidate-carried > env var > positional fallback. - ``_flatten_metrics(metrics)`` — flattens ``CandidateMetrics`` for ``wandb.log``. Direct fields land top-level; ``finish_reason_distribution`` flattens to ``finish_reason/<key>`` scalars; ``observability`` is delegated to ``CellObservability.to_wandb_payload()`` with the ``vllm_cell`` prefix so production + benchmark agree on observability key namespacing. - ``log_and_finish(run, metrics, exit_code=0)`` — flattens, logs, finishes. No-op when ``run is None``. ## Auth Via ``$HOME/.netrc`` (set up by ``wandb login``). ``WANDB_API_KEY`` is NOT read — passing the key via env leaks it via ``ps auxe``. ## Soft dependency Missing wandb / netrc / init exception → warning + None return, harness continues. The benchmark JSON output is still written. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Adds ``tools/vllm_benchmark.py`` — the click-based CLI exposing ``list`` / ``run`` / ``compare`` subcommands. ## Subcommands - ``list`` — prints available preset names. - ``run CORPUS --output PATH [--candidates PRESET | --candidates-file PATH] [--simulate-training-overlap-seconds N]`` — loads the corpus, resolves candidates, runs each in subprocess isolation (``run_benchmark_in_subprocess``), logs to wandb (via the benchmark-side per-cell wiring), writes the BenchmarkOutput JSON. - ``compare PATH`` — renders a saved BenchmarkOutput as a candidate-by-metric table, sorted by effective_tok_s. Reads ``peak_vram_gb`` from ``metrics.observability`` (composed from PR-A's schema). ## Bug fix in this commit ``BenchmarkEngineConfig`` switched from ``extra='forbid'`` to ``extra='ignore'`` so the CLI can validate the corpus header's raw ``engine_parameters`` dict into the typed config without raising on capture-time-only fields the model doesn't expose. The typed fields get extracted; everything else flows through ``_build_vllm_kwargs``'s base layer regardless. ## Wandb wiring Each candidate gets its own wandb run via ``init_cell_run``, grouped by ``sweep_id`` (resolved from ``WANDB_RUN_GROUP`` env or auto-timestamp). ``log_and_finish`` flattens ``CandidateMetrics`` (including the composed ``CellObservability``) and closes the run with the right exit code. Skip records also finish the run (exit_code=1) so failed candidates appear in the wandb UI rather than silently disappearing. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Adds the bracketed-A/B methodology presets that interleave baselines with candidates for drift detection, plus the supporting schema fields on ``BenchmarkCandidate`` and ``CandidateMetrics``. ## Schema additions - ``BenchmarkCandidate.condition_label: str = ''`` — sweep-level condition (``'baseline'`` / ``'n_fanout'`` / ``'spec_ngram'`` / ``'fp8'``). Set by the preset. - ``BenchmarkCandidate.bracket_position: int = 0`` — sequence index within a bracketed_ab stream. - ``CandidateMetrics.condition_label`` + ``bracket_position`` — copied from the candidate by the runner so the analyzer can read them directly off CandidateMetrics without joining back to the BenchmarkCandidate by name. ## bracketed_ab function ``bracketed_ab(base, *, candidate_engine_overrides=..., candidate_sampling_overrides=..., candidate_batch_dispatch_mode=..., condition_label, n_samples_per_condition=DEFAULT_BRACKETED_AB_N)`` returns 2N cells: N baselines interleaved with N candidate cells. Both pin ``SamplingParams.seed=DEFAULT_BENCHMARK_SEED``. N=6 default. The 2026-05-26 seed-pin verification established that pinning collapses the per-request RNG portion of variance but ~5% pooled CoV remains as structural non-RNG variance — that's what the cluster-conditioned analyzer (next commits) handles. ## Matrix-condition wrappers Registered in PRESETS for CLI consumption: - ``bracketed_ab_baseline_pool`` — N baselines (the shared pool). - ``bracketed_ab_n_fanout`` — uses ``batch_dispatch_mode='n_fanout'``. - ``bracketed_ab_spec_ngram`` — uses ``speculative_config={ 'method':'ngram', 'num_speculative_tokens':4, 'prompt_lookup_max':4 }``. - ``bracketed_ab_fp8`` — uses ``kv_cache_dtype='fp8'``. CLI invocation: ``--candidates bracketed_ab_spec_ngram`` etc. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Adds ``vllm_benchmark_analysis.py`` — the post-hoc analyzer that partitions cells by cluster signal, computes per-condition + per-cluster aggregates, and emits Welch-CI effect sizes. Combined #7+#8 from the original brief because they're tightly coupled (effect size lives on the same condition aggregate as the cluster stats). ## Pipeline ``analyze(output_dir, cluster_signal='auto', min_cells_per_condition=6) -> AnalysisReport``: 1. Load all BenchmarkOutput JSONs in dir, flatten candidate cells. 2. Auto-pick cluster signal (or use the operator's choice). 3. Silhouette-scored ``k`` in range [2, 4]; ``random_state=42``. 4. Remap labels so cluster 0 has the lowest signal mean (stable ordering). 5. Per-condition: pooled mean/stddev/CoV on effective_tok_s + acceptance_rate. 6. Per-(condition × cluster): mean/stddev/CoV within cluster. 7. Effect size: Welch's-t Δ ± 95% CI vs the 'baseline' condition, both pooled and per-cluster. 8. Refuse aggregates for conditions with <6 cells (brief mandate). ## Schema Pydantic models: ``AnalysisReport``, ``ConditionAggregate``, ``ConditionClusterAggregate``, ``ClusterStats``, ``ClusterAssignment``, ``EffectSize``. Each ``extra='forbid'``. ``AnalysisReport.to_markdown_summary()`` renders a human-readable table-formatted summary. ## CLI ``tools/vllm_benchmark.py analyze <output_dir> [--cluster-signal auto|wall_seconds|acceptance_rate] [--min-cells-per-condition N] [--json-out PATH]`` — runs the pipeline + prints markdown + optionally writes the full report JSON. ## Welch CI math ``_welch_ttest_ci(cand, base, alpha=0.05) -> (mean_diff, ci_low, ci_high, welch_df) | None``. Welch–Satterthwaite degrees of freedom computed explicitly. Returns None for underdetermined (n<2 either side or both stddevs zero). Validated synthetically: clear-difference inputs (cand mean ~1708, base mean ~1502, n=6 each) → Δ=+205, CI=[+193, +216], df=7.3, CI excludes 0; identical inputs → Δ=0, CI brackets 0 symmetrically. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

37 tests across two files (~55s wall). Covers consumer-facing contracts; skips runner end-to-end (requires vLLM spin-up). ## test_vllm_benchmark.py (29 tests) - **Schema contracts** — ``CandidateMetrics`` composition of ``CellObservability``, JSON round-trip (nested observability survives), ``BenchmarkEngineConfig.extra='ignore'`` tolerates capture-time kwargs, ``BenchmarkCandidate.extra='forbid'`` blocks silent drift. - **Corpus loader** — header + records JSONL parses correctly; missing header raises with a clear message. - **Helpers** — ``_percentile`` linear-interp + empty-list edge, ``_merge_sampling_kwargs`` strips capture-time-only metadata fields, overrides win on conflict, ``_extract_ttft_ms`` parametrized matrix (modern ``first_token_latency``, legacy ``first_token_time - arrival_time`` fallback, None on missing data), ``_truncate_stderr`` tail-preserving, ``_parse_error_class`` exception-name extraction. - **``_build_vllm_kwargs``** — header + candidate config overlay, attention_backend translation, None-valued field exclusion, attention='auto' treated as unset. - **Presets** — ``baseline`` pins seed, ``default_matrix`` dedupes by (engine_config, sampling_overrides), ``attention_backend_sweep`` covers known backends. - **bracketed_ab** — 2N interleaved cells with correct positions and labels, ``bracketed_ab_spec_ngram`` injects ``speculative_config`` on candidates only, all cells seed-pinned. - **SubprocessRunResult** — success + failure shapes. ## test_vllm_benchmark_analysis.py (8 tests) - **Welch CI math** — clear-difference CI excludes 0; identical-means CI brackets 0 symmetrically; underdetermined inputs (n<2 either side or both stddevs zero) return None. - **Full analyze pipeline** — synthetic 6+6 sweep (baselines around ~1500 tok/s, spec_ngram around ~1700) produces: - Correct n_cells + per-condition aggregates with pooled mean within tolerance. - Effect size emitted for spec_ngram, NOT for baseline (no self-comparison). - Refusal entries (not aggregates) for conditions with N<6. - AnalysisReport JSON round-trips losslessly. - Empty output dir raises with a clear message. ## DRY mechanisms - Module-level ``_metric`` helper builds CandidateMetrics with the necessary scaffolding; reused across all analyzer tests. - ``_write_output_dir`` writes a single BenchmarkOutput JSON to a tmp dir; used by the synthetic-sweep fixture. - Parametrized matrices for ``_extract_ttft_ms`` (4 cases) and ``_welch_ttest_ci`` underdetermined inputs (3 cases). - Shared fixtures: ``header`` (TraceHeader), ``empty_base`` (default BenchmarkEngineConfig), and ``synthetic_sweep_dir`` (12-cell baseline+spec_ngram fixture). Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

- ``make format`` output across 8 files (mechanical: line breaks, import sort, blank lines). - Runtime fix: ``processor.process(text)`` was wrong — the actual API is ``Processor.__call__(prompt_number, text) -> ParsedResponse`` (per ``processors.py``). Fixed to ``processor(prompt_idx, text)`` with the correct property access on ``parsed.valid_records`` / ``parsed.invalid_records``. Both surfaced by ``make check`` (format-check + typecheck). Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Apply the base observability branch's elegance pass to the benchmark-specific modules: - type PRESETS as dict[str, PresetFn] and _resolve_candidates(base: BenchmarkEngineConfig); drop the ty:ignore + isinstance(list) guard - replace the engine-init thread's dict[str, Any] result channel with a typed _EngineInitResult dataclass; raise InternalError if the thread finishes without an LLM or exception (narrows init_result.llm to LLM) - convert manual NvmlPeakSampler __enter__/__exit__ to a `with` block - drop the redundant try/except around read_vllm_runtime_metrics (the hardened primitive never raises and returns a stable VllmRuntimeMetrics) - replace getattr(candidate, ...) defensive reads with direct attribute access; swap the analyze cluster_signal ty:ignore for a localized cast(ClusterSignal, ...) at the click boundary No behavior change; benchmark + analyzer contract tests still pass. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Follow the base branch rename of the shared observability primitive (CellObservability -> GenerationObservability, log_cell_observability -> log_generation_observability, vllm_cell -> vllm_gen wandb prefix). Update the benchmark consumers: CandidateMetrics.observability field/default, the _flatten_metrics docstring, and the data-model tests. The benchmark's own grid "cell" vocabulary (cells, n_cells, init_cell_run, per-cell aggregation) is intentionally unchanged. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

binaryaaron force-pushed the binaryaaron/vllm-benchmark-harness-work branch 2 times, most recently from 8b62395 to 738dff5 Compare May 27, 2026 15:38

binaryaaron force-pushed the binaryaaron/vllm-observability-improvements branch from f7f177e to aa91b42 Compare June 3, 2026 21:44

binaryaaron force-pushed the binaryaaron/vllm-benchmark-harness-work branch from 738dff5 to 3d2a2c2 Compare June 4, 2026 17:40

mckornfield reviewed Jun 4, 2026

View reviewed changes

binaryaaron force-pushed the binaryaaron/vllm-observability-improvements branch from 5fc70a0 to 23c69c5 Compare June 5, 2026 23:32

binaryaaron force-pushed the binaryaaron/vllm-benchmark-harness-work branch from 3d2a2c2 to 4473005 Compare June 9, 2026 00:17

github-actions Bot added area:dev-ex Affects build or dev experience area:generation area:tests labels Jun 9, 2026

Base automatically changed from binaryaaron/vllm-observability-improvements to main June 9, 2026 18:08

binaryaaron force-pushed the binaryaaron/vllm-benchmark-harness-work branch from 4473005 to 027eecf Compare June 15, 2026 21:06

github-actions Bot added area:observability area:docs labels Jun 15, 2026

github-code-quality Bot found potential problems Jun 15, 2026

View reviewed changes

Comment thread src/nemo_safe_synthesizer/generation/vllm_benchmark.py Fixed

Comment thread tests/generation/test_vllm_benchmark.py Fixed

github-code-quality Bot found potential problems Jun 15, 2026

View reviewed changes

Comment thread src/nemo_safe_synthesizer/generation/vllm_benchmark.py Fixed

github-actions Bot added the area:config label Jun 16, 2026

github-code-quality Bot found potential problems Jun 16, 2026

View reviewed changes

binaryaaron added 10 commits July 10, 2026 21:46

binaryaaron added 7 commits July 10, 2026 21:46

Fix vLLM benchmark PR review issues

617fbc1

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Refine vLLM benchmark harness

0ab073b

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Share benchmark preset definitions

241bc93

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Move vLLM benchmark subprocess runner to tool

6b6d528

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

Fix vLLM benchmark replay fidelity

9707c60

Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>

binaryaaron force-pushed the binaryaaron/vllm-benchmark-harness-work branch from 295e25f to 9707c60 Compare July 10, 2026 22:45

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: vLLM benchmark harness - #527

feat: vLLM benchmark harness#527
binaryaaron wants to merge 17 commits into
mainfrom
binaryaaron/vllm-benchmark-harness-work

binaryaaron commented May 27, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 27, 2026 •

edited

Loading

Review skipped

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

mckornfield Jun 4, 2026

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 15, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants



		class ConditionClusterAggregate(BaseModel):
		"""Per-(condition × cluster) aggregate: in-cluster condition stats."""

		# ---------------------------------------------------------------------------


		def load_cells(output_dir: Path) -> list[CandidateMetrics]:

Uh oh!

Conversation

binaryaaron commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Notes

Uh oh!

coderabbitai Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

codecov Bot commented Jun 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

binaryaaron commented May 27, 2026 •

edited

Loading

coderabbitai Bot commented May 27, 2026 •

edited

Loading

codecov Bot commented Jun 15, 2026 •

edited

Loading