[ibrahimnd2000]

route_orchestrated_disaggregated_request sends the decode request with aiohttp.ClientTimeout(total=600). Any response still generating after ten minutes is therefore cut mid-stream:

• the router drops the connection;
• the decode engine logs Aborted request;
• the client gets a truncated body. Behind an OpenAI-compatible gateway this typically surfaces as a generic server error after the stream has started.

Long reasoning answers hit it routinely. At ~100 tokens/s per stream, anything above ~60k tokens fails, which is well within max_tokens for reasoning models.

Seen in production on a GLM-5.3 P/D deployment (vLLM 0.31) during an AIME eval sent with max_tokens: 131072:

• three streams ended at 600.7–601.4 s after 62–66k tokens;
• the decode engine was healthy and aborted each request at the same second the router hung up;
• no stream lasted longer than 602 s.

Fix: no total limit on the decode request. A streaming response now fails only after DECODE_STREAM_IDLE_TIMEOUT_S (600 s) with no data, so a stalled decode is still cut but a long stream that keeps producing tokens isn't. A non-streaming response only arrives after generation, so it gets no read limit either. The prefill request keeps its 300 s limit.

Verified on our deployment: a streamed 120,000-token generation ran 834 s and completed (finish_reason=length). Before the fix it would have been cut at 600 s.

Tests: src/tests/test_disaggregated_orchestrated_decode_timeout.py runs the orchestrated flow with a fake client and checks the timeout each request is sent with:

• streaming decode: no total, 600 s read;
• non-streaming decode: no total, no read limit;
• prefill: 300.

Both cases fail on main (assert 600 is None). The rest of src/tests passes (262; test_multipart_proxy.py couldn't be collected locally because httpx isn't installed). pre-commit passes.

Related orchestrated-P/D fixes: #1126 (prefill-request token limits), #1117, #1122, #1123.

[gemini-code-assist]

Code Review

This pull request removes the 10-minute total timeout limit for orchestrated decode requests to prevent long-running streaming responses from being cut off mid-stream. Instead, it introduces an idle timeout of 600 seconds that only triggers if no data is received during streaming, and adds a corresponding test suite. Feedback suggests adding explicit connection timeouts (connect and sock_connect) to the client timeout configuration to prevent requests from hanging indefinitely if the backend decode engine is unreachable.

[ibrahimnd2000]

@ApostaC @YuhanLiu11 @Shaoting-Feng @ruizhang0101 could one of you review when you have a moment? Three small fixes to the orchestrated P/D path, each with tests:

• [Bugfix][Router] Orchestrated P/D: no ten-minute total limit on the decode request #1127 (this): decode used ClientTimeout(total=600), so any response still generating after 10 minutes was cut mid-stream. Now a streaming decode only times out after 600 s without data.
• [Bugfix][Router] Orchestrated P/D: prefill request generates one token (min_tokens, /v1/responses) #1126: the prefill request drops min_tokens (vLLM 400s on min_tokens > max_tokens=1) and sets max_output_tokens=1 on /v1/responses (gibberish otherwise). Independently verified in the review above.
• [Bugfix][Router] Match P/D model labels exactly instead of by substring #1117: all checks green.

The k8s-discovery-e2e-test failure here is the CI cluster's vLLM pods not becoming ready within 600 s; it also fails on unrelated PRs this week. All three fixes have been running in our production deployment since 9 Oct.