Repository navigation
Flakes: egress repeat hop no longer redials past its connection's lagging return; a hook call refused at the cap as its instance is quarantined waits for its trial - #504
Merged
Conversation
…ion is back in the pool (ConnReturned), so a repeat hop is lent it instead of racing its return with a fresh dial; forcing test holds the return back 300 ms
…ime (a test lag on the authority), and the lag is cleared after
…ted before the refusal was looked at waits for its trial, never answering the refusal; forcing test holds the refused call until the fault
MattJackson
enabled auto-merge
October 6, 2026 22:06
promote into
|
| crate | test | step | first panic |
|---|---|---|---|
| `` | nextest xtask::cli::selftest_runs_every_registered_gates_red_proof |
test:workspace | |
transport_dropped_in_serves |
a_dropped_in_transport_registers_through_the_one_fold_and_serves |
test:dropped-in-tcp-transport | crates/busbar/tests/transport_dropped_in_serves.rs:168:13: busbar exited (ExitStatus(unix_wait_status(512))) before serving; log: |
-p busbar --test transport_dropped_in_serves |
test target failed |
test:dropped-in-tcp-transport |
DENY rows (12)
| gate | row | detail |
|---|---|---|
| construction | one-pick-site |
3 production call site(s) of 'pick_among(' (ceiling 2): crates/busbar-kernel-egress/src/walk.rs:298; crates/busbar-llm/src/engine/exhaustion/fallback.rs:120; crates/busbar-llm/src/engine/pipeline.rs:8 |
| kind-isolation | kind-isolation:deps |
4 finding(s), 78 shipped edge instance(s) over 28 class(es), 78 declaration(s); 62 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> plugin-tooling busbar-core-admin -> busbar-plugin-l |
| kind-isolation | kind-isolation:test-deps |
5 finding(s), 36 test edge instance(s) over 19 class(es), 36 declaration(s); 22 '[[dep]]' row(s), 0 question(s): unlisted-dep-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is a test edge |
| kind-isolation-ship | kind-isolation:deps |
16 finding(s) over 78 shipped edge(s): ship-edge cleanliness -> plugin-tooling busbar-core-admin -> busbar-plugin-loader is 'not-allowed': the architecture grants no cleanliness -> plugin-tooling edge |
| kind-isolation-ship | kind-isolation:test-deps |
13 finding(s) over 36 test edge(s): ship-edge cleanliness -> legacy busbar-core-admin -> busbar-a2a is 'not-allowed': the architecture grants no cleanliness -> legacy edge, and the ship criterion is t |
| kind-isolation-ship | kind-isolation:faces |
4 finding(s) over 31 crate(s): foreign-entry crates/busbar busbar is kind 'root' and implements 'Store' 2 time(s) in shipped source — the entry face of kind 'store'. A trait implementation is a claim |
| kind-isolation-ship | kind-isolation:testkit |
2 finding(s) over 12 crate(s): battery-ignored crates/busbar-plane-decisions busbar-plane-decisions carries a tests/conformance.rs whose every entry is '#[ignore]'d (or which has none). 'cargo test' |
| kind-isolation-ship | kind-isolation:legacy-drain |
5 finding(s): transitional-live qa/kind-isolation.toml 'busbar-a2a -> busbar-core-admin' (legacy drain: the retiring A2A engine's tests drive the admin surface that drained into the cleanliness tier) |
| kind-isolation-ship | kind-isolation:control-path |
74 finding(s) over 3 control surface(s): upstream crates/busbar-core-admin/src/admin_codec/meta.rs:84 busbar-core-admin names 'egress' — a control surface has no upstream to reach, so the vocabulary o |
| ship-ready | ship-ready:ship-twin |
'kind-isolation-ship' is not green: kind-isolation:deps (a dependency the architecture does not grant is still in the graph); kind-isolation:test-deps (a dependency the architecture does not grant is |
| instance-noun-neutrality | instance-noun-neutrality:voice |
tracked known-debt census — 8: crates/busbar-plane-llm/src/codec/gemini/handler.rs×7 [cross-plugin] | crates/busbar-plane-llm/src/codec/gemini/tests/handler_tests.rs×2 [cross-plugin] | crates/busbar |
| structure-lint | structure-lint:plane-dup:unledgered |
24 finding(s): PLANE-DUPLICATE (module): 'config.rs' — a2a:crates/busbar-a2a/src/a2a/config.rs decisions:crates/busbar-plane-decisions/src/config.rs mcp:crates/busbar-mcp/src/mcp/config.rs voice:crate |
Judged against base 77eb3e484: 0 new red, 0 worse, 8 standing (excused).
tests passed: 24265, failed: 1. Run: https://github.com/GetBusbar/busbar/actions/runs/37538388696 . Artifact verdict-92ff4d5908a80cf7c2b8d8b229c054f6dad97dfd (failures.json, junit.xml, raw.log; 90 days).
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two flakes, each fixed at the root, each with a forcing test that is RED without the fix.
1.
plane_host::egress::tests::a_repeat_hop_reuses_the_pooled_connection_instead_of_redialingThe race. An HTTP/1.1 exchange whose body was not drained when its head arrived goes back to the pool only once hyper's dispatcher finishes it, on a spawned watcher task. The plane host's stream task reported the body's end to the plane as soon as it read it, before that watcher ran. A repeat hop opened right after the end could miss the pool, park, and start a fresh dial. The watcher then handed the returning connection to the parked hop, and the extra dial's connection parked idle with 0 requests. That is exactly #484's red: conn1 served 2 requests, conn2 served 0.
The fix. The engine puts
ConnReturnedon the response when the return waits on the exchange. It settles once the connection is back in its pool, or dropped as dead. The plane host's stream task awaits it, within the hop's deadline, before it reports the body's end. A hop opened after that end is lent the connection.Forcing test (
a_repeat_hop_reuses_the_connection_whose_return_lags_its_body): a test-only lag holds that authority's return back 300 ms. With the fix reverted it fails 10 of 10 (two connections). With the fix it passes 200 of 200 under load, and the original test passes 200 of 200 under load.2.
hook_door::tests::the_inflight_cap_saturates_and_fails_on_the_caller_deadline_through_the_axisThe race. The wedged calls sleep 1.5 s, past the 1 s Call class budget, so the watchdog faults the instance at about 1 s. In #491's run, the freed-slot call was refused at the cap (every unit still held), and the watchdog faulted the instance before the refusal was looked at. The cap logic only runs
if Refused && !is_faulted(), so the refusal fell through asbroken (hook ... answered Refused)at 1.00 s.The fix (
hook_door.rssubmit): a refusal is never a crossing. A call refused while its instance has meanwhile been faulted was never made by it, so it waits for the trial window within its own budget. This is the same rule a call meeting the fault before submission already follows (R2). It never answers the refusal.Forcing test (
a_call_refused_at_the_cap_as_the_watchdog_faults_the_instance_waits_for_its_trial): a test-only hold keeps the refused call until the watchdog has faulted the instance, which reproduces the CI order every time. With the fix reverted it fails 6 of 6 with exactly #491's message. With the fix it passes 100 of 100 under load, and the original test passes 100 of 100 under load (3.5 s each).All soaks ran on Latchkey large with 2x nproc busy loops. fmt and clippy -D warnings are clean on busbar-kernel and busbar-plugin-loader; the busbar-kernel egress suite passes 273 of 273.