Skip to content

Flakes: egress repeat hop no longer redials past its connection's lagging return; a hook call refused at the cap as its instance is quarantined waits for its trial - #504

Merged
MattJackson merged 5 commits into
predevfrom
p6-flakes-3
Oct 7, 2026

Conversation

@MattJackson

Copy link
Copy Markdown
Collaborator

Two flakes, each fixed at the root, each with a forcing test that is RED without the fix.

1. plane_host::egress::tests::a_repeat_hop_reuses_the_pooled_connection_instead_of_redialing

The race. An HTTP/1.1 exchange whose body was not drained when its head arrived goes back to the pool only once hyper's dispatcher finishes it, on a spawned watcher task. The plane host's stream task reported the body's end to the plane as soon as it read it, before that watcher ran. A repeat hop opened right after the end could miss the pool, park, and start a fresh dial. The watcher then handed the returning connection to the parked hop, and the extra dial's connection parked idle with 0 requests. That is exactly #484's red: conn1 served 2 requests, conn2 served 0.

The fix. The engine puts ConnReturned on the response when the return waits on the exchange. It settles once the connection is back in its pool, or dropped as dead. The plane host's stream task awaits it, within the hop's deadline, before it reports the body's end. A hop opened after that end is lent the connection.

Forcing test (a_repeat_hop_reuses_the_connection_whose_return_lags_its_body): a test-only lag holds that authority's return back 300 ms. With the fix reverted it fails 10 of 10 (two connections). With the fix it passes 200 of 200 under load, and the original test passes 200 of 200 under load.

2. hook_door::tests::the_inflight_cap_saturates_and_fails_on_the_caller_deadline_through_the_axis

The race. The wedged calls sleep 1.5 s, past the 1 s Call class budget, so the watchdog faults the instance at about 1 s. In #491's run, the freed-slot call was refused at the cap (every unit still held), and the watchdog faulted the instance before the refusal was looked at. The cap logic only runs if Refused && !is_faulted(), so the refusal fell through as broken (hook ... answered Refused) at 1.00 s.

The fix (hook_door.rs submit): a refusal is never a crossing. A call refused while its instance has meanwhile been faulted was never made by it, so it waits for the trial window within its own budget. This is the same rule a call meeting the fault before submission already follows (R2). It never answers the refusal.

Forcing test (a_call_refused_at_the_cap_as_the_watchdog_faults_the_instance_waits_for_its_trial): a test-only hold keeps the refused call until the watchdog has faulted the instance, which reproduces the CI order every time. With the fix reverted it fails 6 of 6 with exactly #491's message. With the fix it passes 100 of 100 under load, and the original test passes 100 of 100 under load (3.5 s each).

All soaks ran on Latchkey large with 2x nproc busy loops. fmt and clippy -D warnings are clean on busbar-kernel and busbar-plugin-loader; the busbar-kernel egress suite passes 273 of 273.

…ion is back in the pool (ConnReturned), so a repeat hop is lent it instead of racing its return with a fresh dial; forcing test holds the return back 300 ms
…ime (a test lag on the authority), and the lag is cleared after
…ted before the refusal was looked at waits for its trial, never answering the refusal; forcing test holds the refused call until the fault
@MattJackson
MattJackson enabled auto-merge October 6, 2026 22:06
@MattJackson
MattJackson added this pull request to the merge queue Oct 6, 2026
@github-actions

github-actions Bot commented Oct 6, 2026

Copy link
Copy Markdown

promote into predev: BOARD @92ff4d590: 3 failing test row(s), 12 DENY row(s)

Failing tests (3)

crate test step first panic
`` nextest xtask::cli::selftest_runs_every_registered_gates_red_proof test:workspace
transport_dropped_in_serves a_dropped_in_transport_registers_through_the_one_fold_and_serves test:dropped-in-tcp-transport crates/busbar/tests/transport_dropped_in_serves.rs:168:13: busbar exited (ExitStatus(unix_wait_status(512))) before serving; log:
-p busbar --test transport_dropped_in_serves test target failed test:dropped-in-tcp-transport

DENY rows (12)

gate row detail
construction one-pick-site 3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8
kind-isolation kind-isolation:deps 4 finding(s), 78 shipped edge instance(s) over 28 class(es), 78 declaration(s); 62 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> plugin-tooling busbar-core-admin -> busbar-plugin-l
kind-isolation kind-isolation:test-deps 5 finding(s), 36 test edge instance(s) over 19 class(es), 36 declaration(s); 22 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge
kind-isolation-ship kind-isolation:deps 16 finding(s) over 78 shipped edge(s): ship-edge cleanliness -> plugin-tooling busbar-core-admin -> busbar-plugin-loader is 'not-allowed': the architecture grants no cleanliness -> plugin-tooling edge
kind-isolation-ship kind-isolation:test-deps 13 finding(s) over 36 test edge(s): ship-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is 'not-allowed': the architecture grants no cleanliness -> legacy edge, and the ship criterion is t
kind-isolation-ship kind-isolation:faces 4 finding(s) over 31 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Store' 2 time(s) in shipped source — the entry face of kind 'store'. A trait implementation is a claim
kind-isolation-ship kind-isolation:testkit 2 finding(s) over 12 crate(s): battery-ignored crates/busbar-plane-decisions busbar-plane-decisions carries a tests/conformance.rs whose every entry is '#[ignore]'d (or which has none). 'cargo test'
kind-isolation-ship kind-isolation:legacy-drain 5 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier)
kind-isolation-ship kind-isolation:control-path 74 finding(s) over 3 control surface(s): upstream crates/busbar-core-admin/src/admin_codec/meta.rs:84 busbar-core-admin names 'egress' — a control surface has no upstream to reach, so the vocabulary o
ship-ready ship-ready:ship-twin 'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is
instance-noun-neutrality instance-noun-neutrality:voice tracked known-debt census — 8: crates/busbar-plane-llm/src/codec/gemini/handler.rs×7 [cross-plugin] | crates/busbar-plane-llm/src/codec/gemini/tests/handler_tests.rs×2 [cross-plugin] | crates/busbar
structure-lint structure-lint:plane-dup:unledgered 24 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs mcp:crates/busbar-mcp/src/mcp/config.rs voice:crate

Judged against base 77eb3e484: 0 new red, 0 worse, 8 standing (excused).

tests passed: 24265, failed: 1. Run: https://github.com/GetBusbar/busbar/actions/runs/37538388696 . Artifact verdict-92ff4d5908a80cf7c2b8d8b229c054f6dad97dfd (failures.json, junit.xml, raw.log; 90 days).

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 7, 2026
@MattJackson
MattJackson added this pull request to the merge queue Oct 7, 2026
Merged via the queue into predev with commit ad3cae9 Oct 7, 2026
8 checks passed
@MattJackson
MattJackson deleted the p6-flakes-3 branch October 7, 2026 02:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant