Skip to content

[Bugfix][Router] Return 503 when failover ends on a backend client error - #1164

Open
Mazyod wants to merge 1 commit into
vllm-project:mainfrom
Mazyod:fix/unreachable-backend-503
Open

Mazyod wants to merge 1 commit into
vllm-project:mainfrom
Mazyod:fix/unreachable-backend-503

Conversation

@Mazyod

@Mazyod Mazyod commented Oct 20, 2026

Copy link
Copy Markdown

FIX #1163

Today, when failover ends before response headers and the last error is an aiohttp.ClientError (connection refused, DNS failure, and similar), route_general_request re-raises it. Clients get a plain-text 500, and uvicorn logs Exception in ASGI application with a traceback.

This PR returns the same 503 status and JSON body as proxy_multipart_request, {"error": "Failed to connect to backend: ..."}, and adds X-Request-Id to the response. Other exceptions are re-raised as before, mid-stream failures are unchanged, and the existing failover tests are untouched.

Testing:

  • New unit test test_unreachable_backends_return_503_after_all_attempts in src/tests/test_instance_failover.py. It fails on main and passes with this change.
  • Against a closed port, the router returned 503 application/json with X-Request-Id and logged no ASGI traceback. Main returns 500 text/plain and logs a traceback.
  • All 273 router tests pass on Python 3.12, and pre-commit and ruff check pass.

  • Make sure the code changes pass the pre-commit checks.
  • Sign-off your commit by using -s when doing git commit
  • Try to classify PRs for easy understanding of the type of changes, such as [Bugfix], [Feat], and [CI].

When failover ends before response headers and the last error is an
aiohttp.ClientError (for example, connection refused or DNS failure),
route_general_request re-raises it. Uvicorn then logs "Exception in ASGI
application" with a traceback, and the client receives a plain-text 500
Internal Server Error.

Return the 503 status and JSON body that proxy_multipart_request already
uses for this failure, {"error": "Failed to connect to backend: ..."},
and include the X-Request-Id header. Other exceptions are re-raised
unchanged, and mid-stream failures are unaffected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Mazyad Alabduljaleel <mazjaleel@gmail.com>

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the route_general_request function in src/vllm_router/services/request_service/request.py to handle aiohttp.ClientError when backends are unreachable, returning a 503 JSONResponse with the request ID and a descriptive error message. Additionally, a new unit test test_unreachable_backends_return_503_after_all_attempts has been added to src/tests/test_instance_failover.py to verify this error handling behavior. There are no review comments to address, and I have no further feedback to provide.

Mazyod pushed a commit to Mazyod/production-stack that referenced this pull request Oct 20, 2026
…laybook

Point the single-label hostname patch at upstream PR vllm-project#1154, which
superseded the now-closed vllm-project#737. Record that the DNS resolver change is
proposed upstream for the shared client only (vllm-project#1161/vllm-project#1162), and that
upstream vllm-project#1164's 503 must not replace the fork's documented 502 envelope
on a refresh. Add the evidence-based playbook for getting fixes merged
upstream. Docs only; does not trigger an image release. Standard
pre-commit hooks pass on the changed files.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A5cnah4yC2sS7LEZ8mdrTd
Signed-off-by: ci-sim <ci@sim>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: unreachable backends return a plain-text 500 with an ASGI traceback instead of the multipart path's 503

1 participant