feat(generation): remote vLLM/NIM generation backend (GPU-free)#575
Draft
binaryaaron wants to merge 16 commits into
Draft
feat(generation): remote vLLM/NIM generation backend (GPU-free)#575binaryaaron wants to merge 16 commits into
binaryaaron wants to merge 16 commits into
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
github-actions
Bot
added
area:dev-ex
binaryaaron
force-pushed
the
agonzales/remote-generation-backend
branch
from
135a18b to
3d14d00
Compare
Base automatically changed from
binaryaaron/vllm-observability-improvements
to
main
June 9, 2026 18:08
7 tasks
binaryaaron
force-pushed
the
agonzales/remote-generation-backend
branch
from
3d14d00 to
5a6a324
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
| read_vllm_runtime_metrics, | ||
| ) | ||
| from ..llm.metadata import ModelMetadata | ||
| from ..llm.model_host import ModelHost |
| return random.uniform(0.0, capped) | ||
|
|
||
|
|
||
| def _coerce_token_count(value: object) -> int: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
Add RemoteBackend, a GPU-free generation backend that calls an external vLLM OpenAI-compatible /v1/completions endpoint instead of loading a local engine. Intended for generation-only setups where the fine-tuned LoRA adapter is already served by a running endpoint. - Lift the shared batch-generation loop from VllmBackend up into GeneratorBackend as a concrete template method. VllmBackend and RemoteBackend now implement only the engine-specific pieces (_generate_batch, _get_prompt_token_count, prepare_params, initialize, teardown). TimeseriesBackend keeps its own generate() override. - Add RemoteParameters (endpoint_url, model, api_key_env, timeout_seconds, max_concurrency) nested under GenerateParameters; wire backend selection in SafeSynthesizer.generate(). Remote + time-series is rejected at config validation time. - Structured generation maps to vLLM guided_regex / guided_json; the XGrammar structural_tag method has no OpenAI-API equivalent and is rejected with a clear error. - Uses httpx (already a dependency); one request per record, issued concurrently up to max_concurrency, with per-completion token counts taken from the server's usage report. Verified end-to-end against a live vLLM server (guided_json) plus 24 unit tests; ruff and ty clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Testing against a second vLLM server (0.20+, serving Nemotron-3-Super)
showed it silently ignores the legacy top-level guided_json / guided_regex
request fields, producing unconstrained output. Switch RemoteBackend to the
modern `structured_outputs: {json|regex: ...}` field, which both old and new
vLLM OpenAI servers honor and which mirrors the offline StructuredOutputsParams
the local backend already builds.
Verified end-to-end against the new server with the regex method (status
complete, valid JSONL records parsed by TabularDataProcessor). Note: the
json_schema method works at the transport level but the server pretty-prints
the JSON across lines, which the JSONL processor rejects; the regex method
enforces the compact single-line shape the pipeline expects.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
structural_tag was originally rejected on the assumption it had no HTTP
equivalent -- true for the legacy guided_* extensions, but stale now that
the backend uses the modern `structured_outputs` field. Verified against a
vLLM 0.20+ server that `structured_outputs: {structural_tag: <json-string>}`
is accepted and yields multi-record JSONL.
Map all three schema methods to structured_outputs (regex/json/structural_tag),
so the `auto` default (which resolves to structural_tag on xgrammar/auto
backends) now works out-of-the-box for remote generation. End-to-end run with
auto produced valid, realistic records via TabularDataProcessor.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
A PEP 723 uv tool to quickly spin up a vLLM model and poke it -- the debugging
companion to VllmBackend / RemoteBackend. Three subcommands:
- serve: launch the OpenAI-compatible server from the project venv, optionally
attaching a LoRA adapter (--adapter / --max-lora-rank). Bakes in this box's
A100 workarounds (offline HF cache, FLASH_ATTN, --enforce-eager). --dry-run
prints the command without touching the GPU.
- call: fire one or more chat/text completions, with thinking toggle, structured
outputs (--json-schema / --regex / --structural-tag -> structured_outputs),
token usage, latency, and --json output.
- models: list served models / health check.
Distilled from dev/evals/route-eval/{serve_vllm.sh,vllm_probe.py}. Validated
against a live server (models + chat + json-schema-constrained text call) and
serve via --dry-run.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Durable session note: RemoteBackend design and verified facts (the structured_outputs vs legacy guided_* version split, structural_tag now supported, httpx base-path join), the dev flow used, and a roadmap for targeting NIM / Triton / Dynamo / TensorRT-LLM (incl. a structured-output dialect seam if a target only speaks response_format). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…e arc Horizon note capturing the multi-month direction: self-managing, backend-like components with slowly-exposed knobs, coordinated by an async orchestrator that treats each component (and each component treats itself) as a state machine over a single shared state system, with config first-class throughout. Records what the codebase already embodies, the target contracts, engine-vs-transform archetypes, tensions to watch, and an incremental adoption path (backends -> evaluation -> assembly -> PII -> orchestrator). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
The --enforce-eager workaround existed only because the venv's FlashInfer was broken (vLLM's torch.compile path imports flashinfer.comm unconditionally). With FlashInfer repaired, default to CUDA graphs and make --eager an opt-in escape hatch for broken-FlashInfer environments. Verified end-to-end: same financial-transactions LoRA workload dropped from ~119s (eager) to ~15s (CUDA graphs) for 10 records, with identical, valid output. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
mise run format (copyright_fixer) added SPDX headers to the chronicle notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…servers
NIM (TensorRT-LLM) rejects several vLLM sampling extensions with a 400
(repetition_penalty, top_k, skip_special_tokens, include_stop_str_in_output),
so RemoteBackend's vLLM-shaped request fails against it out of the box.
Add RemoteParameters.dialect ("vllm" default | "openai"). prepare_params emits
the vLLM extensions only for the "vllm" dialect; "openai" sends just the
universal OpenAI fields (temperature/top_p/max_tokens/n) for strict servers
like NIM and Triton. Resolved sampling values are identical across dialects;
only the wire fields differ. Verified the dialect finding live against a NIM
serving a Safe Synthesizer LoRA.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…ronicle Records the NIM + vLLM real-adapter runtime validation, the NIM sampling-field dialect finding (and the shipped RemoteParameters.dialect seam), the face-validity-only quality caveat, the one clean perf comparison (eager->CUDA graphs ~7.8x), and the FlashInfer corrupted-install root cause + fix. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…icle - AGENTS.md module map: note RemoteBackend and that GeneratorBackend now owns the concrete generate() template method (was stale after the refactor). - chronicle: record the measured results — real Evaluator scores (SQS 8.0 / Data Privacy 8.8 on 250 remote-generated records) and the controlled single-stream vLLM-vs-NIM throughput (74.7 vs 66.3 tok/s), replacing the "not yet done" placeholder. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…rsing Add RemoteParameters.max_retries (default 4) and retry transient failures (connection drops, timeouts, HTTP 408/409/425/429/5xx) with full-jitter exponential backoff that honors a Retry-After header; non-retryable statuses fail fast. Split _complete_one into _post_completion / _raise_for_status / _parse_completion, coerce the advisory usage token count defensively, and treat only a missing choices array as fatal. Size the prompt-length clamp from an opportunistic local tokenizer (None on the offline-remote path) and compact pretty-printed json_schema completions to single-line JSONL (disable_any_whitespace at the source on the vllm dialect, client-side compaction as a portable net). Bake the constant prompt into the request body once and submit per-request futures so each completion retries independently. Expand unit coverage for retries, parsing, and helpers. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Validate the at-most-one constraint rule before touching the filesystem, and read schema/tag files through helpers that exit via bad_input (code 125) on a missing file or malformed JSON instead of raising a traceback. Guard parse_call_response against a response with no choices so do_call reports it as a failed CallResult rather than crashing on a KeyError/IndexError. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Add a CPU smoke test that drives the full RemoteBackend path -- real httpx client, ThreadPoolExecutor concurrency, the shared generate() loop, the real tabular processor, retry/backoff, and JSON compaction -- against an in-process loopback HTTP stub (no GPU, no network, no model). Add an opt-in live smoke test against build.nvidia.com gated by NSS_REMOTE_SMOKE_SEND=1 + NVIDIA_API_KEY that validates the transport and skips when a model rejects /v1/completions. Share tokenizer-free metadata and minimal-config builders via conftest and document both layers in the smoke README. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
…er KB Move the remote-generation-backend and modular-stateful-architecture-arc chronicle notes out of the product repo into the repo's knowledge base (/root/kb/safe-synthesizer/raw), where the working-memory chronicle belongs. The KB graph was refreshed to reflect the promoted content. Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
Signed-off-by: Aaron Gonzales <aagonzales@nvidia.com>
binaryaaron
force-pushed
the
agonzales/remote-generation-backend
branch
from
5a6a324 to
c226c79
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/v1/completionsendpoints, including retries, dialect-aware request fields, and structured-output handling.GeneratorBackendwhile keeping engine-specific calls in the local and remote backends.ModelHost[ModelT, TokenizerT]lifecycle boundary for locally owned models;VllmBackendimplements it, and feat: add option to use local classification #615 can use the same shape for its Hugging Face classifier.mainand adopts the nested structured-generation configuration introduced there.Test plan
mise run check