Expose optimized Gemma serving controls

[mohammadaaftabv]

What

Expose optimized Gemma/vLLM serving controls through the text-processing pipeline:

• forward speculative-decoding settings
• support independent temperature and top_p values per text stage
• expose cache-aware Dynamo routing
• add one shared Dynamo admission gateway that converts vLLM queue overload into HTTP 429 and applies a model-wide AIMD concurrency window
• automatically assign and discover a distinct Prometheus metrics endpoint for each vLLM worker
• allow prebuilt Dynamo environments to reuse their installed runtime
• retain existing defaults; every new behavior is opt-in

Why

The shared Dynamo path already represented model placement, tensor parallelism, context length, KV-cache dtype, sequence and token limits, request timeout, and per-client concurrency. It did not expose the remaining executable serving controls or provide one overload signal and adaptive concurrency state shared by all remote text-stage actors.

Queue control is implemented once at the server boundary. Remote stage clients do not poll Prometheus independently, and there is no manual metrics-URL fallback or selectable queue-control policy. The user-facing admission inputs are limited to the queue threshold and shared concurrency ceiling.

Compatibility

Existing behavior is unchanged unless the new options are supplied. Setting --inference_queue_max_waiting_requests enables the shared admission gateway. Worker metrics endpoints are discovered from Dynamo placements. Missing or stale queue metrics fail open.

Validation

Focused tests and static checks

• 77 focused Dynamo, admission, vLLM-infrastructure, and stage-sampling tests passed
• Python compilation passed for the modified runtime files
• focused Ruff checks and formatting checks passed for the modified Dynamo implementation and tests
• git diff --check passed

Matched Slurm performance experiment

Configuration:

• 2,048 English rows, four shards, and 10,240 LLM requests per profile
• PnC, TN, LanguageID, CodeSwitching, and SpeechQA with remote/fused execution
• RedHatAI/gemma-4-31B-it-FP8-dynamic
• one node, four GPUs, two TP=2 replicas
• two interleaved repetitions per profile to reduce run-order bias

Repetition	Baseline stage	Full stage	Stage throughput change	Baseline wall	Full wall	Wall throughput change
1	352.02 s	283.48 s	+24.18%	578.51 s	495.85 s	+16.67%
2	352.81 s	284.63 s	+23.95%	550.96 s	497.35 s	+10.78%
Paired median	—	—	+24.07%	—	—	+13.72%

Incremental feature attribution, reported as paired median throughput change:

Incremental change	Stage	End-to-end wall
8k to 16k max batched tokens	-0.16%	+0.59%
KV router after 16k batching	+1.37%	+2.48%
Admission/AIMD after KV router	-0.96%	-2.08%
Speculative decoding after KV router	+22.26%	+9.91%
Full configuration versus speculative profile	+0.27%	+0.35%

Speculative decoding accounts for most of the measured improvement. Draft-token acceptance was approximately 87.4%. The measured result is about 1.24x at pipeline-stage level; it does not reproduce a 2.5x improvement on this short-output text-processing workload.

Admission/AIMD is an overload-resilience feature rather than an unsaturated throughput optimization. The normal matrix never developed a vLLM waiting queue, so admission correctly remained inactive there.

Completeness and correctness parity

Across all 12 profiles:

• 24,576/24,576 expected rows and 48/48 .done markers were written
• all source-row identifiers were unique within each profile
• no JSON parse errors occurred
• no enabled output field was missing or empty
• every pipeline process exited successfully

Absolute prompt-contract checks were consistent between baseline and full configurations:

• PnC lexical preservation: 2,040/2,048 passed (99.61%); the same eight source rows failed in both configurations and both repetitions
• LanguageID output format: 2,047/2,048 passed (99.95%); the same source row failed everywhere
• SpeechQA output format: baseline 2,047/2,048 (99.95%), full 2,048/2,048 (100%)

Pooled full-versus-baseline exact-match rates over 4,096 paired rows:

Field	Exact match
llm_language_prediction	100.00%
pnc_text	99.58%
tn_raw	99.51%
code_switched_text	99.32%
speech_qa_text	97.07%

For comparison, the two baseline repetitions matched each other at 100.00% for LanguageID, 99.76% for PnC, 99.66% for TN, 99.51% for CodeSwitching, and 97.17% for SpeechQA. Full-versus-baseline text differences are therefore approximately the same scale as ordinary batched-inference repeat variability. No output-contract regression was detected.

These checks establish completeness, format/constraint compliance, and output parity. They do not establish semantic accuracy for TN, CodeSwitching, or SpeechQA, and this English workload does not validate the intended Indic languages.

Triggered-overload stress experiment

A separate run deliberately saturated one TP=2 worker:

• all 2,048 rows and four .done markers completed with no missing output fields or parse failures
• maximum gateway in-flight requests: 64
• maximum vLLM running/waiting requests: 16/48
• gateway queue rejections and successful client rate-limit events: 59/59
• AIMD multiplicative decreases/additive increases: 11/2
• adaptive limit moved from a maximum of 64 to 4 by completion
• upstream vLLM 429s: 0; vLLM preemptions: 0
• no overload-specific output corruption was detected

Experiment-design caveat

Baseline and full profiles used the same frozen complete source, with the opt-in runtime controls disabled or enabled. This cleanly isolates the executable controls without changing code between profiles. Prebuilt-environment reuse was common to every profile and was not separately ablated.