Gateway: async healthcheck handler and explicit readiness probe timeout

[lorenzo-norcini-scale]

What

• healthcheck (serving /healthcheck, /healthz, /readyz) is now async def. As a sync handler it was dispatched to the shared anyio threadpool, so under load the readiness probe queued behind blocked requests and timed out while the app container was healthy.
• The gateway readinessProbe now sets an explicit timeoutSeconds (default 5, configurable via gateway.readinessProbeTimeoutSeconds). Previously unset, which means the Kubernetes default of 1s.

Why

During the 2026-08-13 gateway incident, saturated but healthy pods failed the 1s probe and were ejected from the Service, concentrating load on the remaining pods (11 of 60 Ready at the worst point while ~40 app containers were healthy). The two changes remove that ejection mechanism: the probe no longer competes with request handling for threadpool slots, and the timeout tolerates transient event-loop delay.

Notes

• No livenessProbe is added. Restarting a saturated pod discards its in-flight work and worsens overload; readiness gating plus the existing container restart policy remain the failure handling.
• Rendered chart verified: probe emits timeoutSeconds: 5 with default values.

🤖 Generated with Claude Code

Greptile Summary

The PR moves the gateway health endpoints onto the async request path and gives Kubernetes readiness probes a configurable five-second timeout.

• Converts the shared /healthcheck, /healthz, and /readyz handler to async def.
• Adds gateway.readinessProbeTimeoutSeconds to the Helm values.
• Renders the configured timeout into the gateway Deployment readiness probe.

Confidence Score: 5/5

The PR appears safe to merge, with the async handler and Helm timeout remaining aligned with the existing gateway readiness contract.

The registered health routes are awaited by the ASGI execution path, remain exempt from the application concurrency limiter, and the chart renders a valid five-second readiness timeout from its default values.

Important Files Changed

Filename	Overview
charts/model-engine/templates/gateway_deployment.yaml	Adds the configured readiness timeout to the gateway probe without changing its path, cadence, or failure threshold.
charts/model-engine/values.yaml	Introduces a documented five-second default for the gateway readiness timeout.
model-engine/model_engine_server/api/app.py	Makes the minimal health handler asynchronous so FastAPI executes it on the event loop rather than the shared worker threadpool.

_{Reviews (1): Last reviewed commit: "fix(gateway): async healthcheck handler ..." | Re-trigger Greptile}

Context used:

• Knowledge Base — Model-engine API and HTTP surface
• Knowledge Base — Model-engine deployment and startup

[lorenzo-norcini-scale]

Superseded by the consolidated incident-hardening PR: see the PR referencing this one.

[lorenzo-norcini-scale]

Superseded by #868.