Skip to content

Retry transient Responses failures across request paths - #3

Open
lawrencecchen wants to merge 41 commits into
upstream-main-20260814from
feat/persistent-capacity-retries
Open

Retry transient Responses failures across request paths#3
lawrencecchen wants to merge 41 commits into
upstream-main-20260814from
feat/persistent-capacity-retries

Conversation

@lawrencecchen

@lawrencecchen lawrencecchen commented Aug 14, 2026

Copy link
Copy Markdown

Retries transient request failures across safe replayable paths: sampling, legacy and streaming remote compaction, model requests after tool results, realtime calls, MCP operations, OAuth refresh, Guardian sampling, and Codex Apps startup reconnect. The exact provider message Selected model is at capacity. Please try a different model. is classified as capacity and retried.

The shared classifier separates transient transport, network, overload, rate-limit, cancellation, and websocket failures from permanent authentication, authorization, invalid-request, quota, policy, TLS, and malformed-response failures. Capacity uses jittered exponential backoff capped at 60 seconds with a 100-attempt budget. Other transient failures have finite operation budgets. Every retry emits a UI-only status with operation, delay, disposition, and attempt. The status is excluded from model history and request input. Completed tool output is preserved across a model-request retry. Subagents use the same turn-scoped notifier and retry loop.

Server cancellation remains retryable. Caller cancellation and deadlines remain terminal. WebSocket protocol and policy closes remain terminal even when their reason text contains capacity words. Requests with one-shot bodies or ambiguous side effects are not replayed when replay cannot be proven safe.

Behavior coverage includes the exact latest capacity body, nested overload envelopes, slow-down responses, 429 outer-budget exhaustion, network and websocket failures, remote compaction, server cancellation, caller cancellation, tool continuation, subagent status, permanent-error precedence, successful-response preservation, the Hyper operation was canceled branch, retry-marked permanent provider errors, and over-budget capacity counters.

Focused verification passed:

  • codex-client retry: 19/19
  • codex-api library: 186/186
  • codex-core response retry: 15/15 selected
  • codex-core retry-after: 25/25
  • codex-core realtime: 24/24
  • rmcp-client retry: 16/16; recovery: 13/13
  • codex-mcp reconnect: 8/8
  • Guardian: 1/1
  • HTTP transport: 7/7
  • model-provider-info: 26/26
  • image generation: 10/10
  • web search: 8/8
  • workspace manifest verification, fmt-check, and git diff checks

Red and green regression pairs include:

  • persistent capacity budget: a12f94d406 -> 55ffde6a65
  • outer 429 budget: 3563b36429 -> bb40bea9af
  • Apps reconnect: dc906bd19b -> 4dcba7b64
  • Hyper caller cancellation: b3a94675e7 -> 19e4126414
  • permanent and over-budget edge cases: 606036ffed -> 707673f916
  • local cancellation: d05955a677 -> 3527561eb8
  • server cancellation: 630303b7c0 -> ef7f16071d
  • websocket policy close: 473be5595d -> 891d3de60d

The optimized arm64 release build passed with Cargo's optimized profile, opt-level=3, thin LTO, four codegen units, and line-table debug info. Source head: 891d3de60d6cabafb37b052feae522ea3edb09fe. Installed binary: /Users/lawrence/.local/bin/codex, pointing to the commit-named artifact under /Users/lawrence/.local/libexec/codex-persistent-capacity-retries/. Release and installed SHA-256: 9b8f46f44ad8a9cfec23b2db8a9c9a66f1baad5313a054a135e831f5e670baf8.

Hosted checks that are not code failures remain: cargo-deny reports baseline RUSTSEC-2026-0258 for h2 0.4.13; SDK and Windows gnullvm jobs cannot find the required codex-runners group; Linux and macOS Bazel jobs reach the hosted timeout. Local Bazel lock update is unavailable because Bazel is not installed. Main build, format, cargo shear, spelling, blob-size, metadata, and socket checks pass.

Residual scope is deliberate: file blob upload bodies are one-shot, memory-summary status has no notifier at that API boundary, generic client sleep has no cancellation token, Guardian prewarm is fail-fast, and custom MCP event streams are not replayed. These paths remain bounded or terminal when replay is unsafe.

@coderabbitai

coderabbitai Bot commented Aug 14, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 584ee316-87e8-4eac-8de2-ae6222408ed3

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@lawrencecchen lawrencecchen changed the title Retry model capacity errors persistently Retry transient Responses failures across request paths Aug 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant