Pre-prod: WebSocket server-initiated close and TCP reset (os error 54) — different failure modes from ping timeout

[Al629176]

Summary

In addition to the ping timeout disconnects (same as staging, #5603), the pre-prod build shows two new socket failure modes: server-initiated connection close and TCP connection reset by peer (os error 54). These indicate the backend is actively terminating connections rather than just going silent.

Log evidence (pre-prod, 2026-08-20)

17:57:59 WRN [socket] Connection lost: Server closed connection

21:40:58 WRN [socket] Connection failed (attempt 1/5): WebSocket connect:
IO error: Connection reset by peer (os error 54)

22:13:33 WRN [backend_api] transient transport failure on GET /orchestration/v1/sessions:
error sending request for url (https://api.tinyhumans.ai/...) → client error (Connection reset by peer)

Distinction from #5603

• Staging: WebSocket drops on exact ~11min cycle — server-side session timeout misconfiguration #5603 (staging): client detects missing ping → passive timeout
• This issue: server actively closes the TCP connection (RST or FIN) — affects both WebSocket and REST API calls simultaneously (17:57 socket close, 22:13 both socket and REST reset)

Impact

When the server resets the connection, both the WebSocket channel and REST API calls fail simultaneously. Any in-flight agent turn is dropped with no recovery path until reconnect completes.

Fix direction

• Investigate the pre-prod backend for conditions that trigger active connection termination (memory pressure, request queue overflow, deploy/restart events).
• The simultaneous WebSocket + REST reset at 22:13 suggests a backend process restart rather than a network event.

[M3gA-Mind]

How to fix this

Re-scope first. The three quoted lines are three different events across a 4+ hour session, and all three are already-handled paths, not a defect:

• 17:57:59 Connection lost: Server closed connection is a WebSocket Close frame (src/openhuman/platform/socket/ws_loop.rs:508-510). It resets the failure streak and the backoff to 1s (ws_loop.rs:184-187), so recovery is ~1s.
• 21:40:58 Connection failed (attempt 1/5) — worth knowing: there is no 5-attempt cap. FAIL_ESCALATE_THRESHOLD = 5 (ws_loop.rs:51) is only the count at which one Sentry event is emitted (log_connection_failure, ws_loop.rs:321-351). The loop retries forever, backoff doubling to a 30s ceiling (ws_loop.rs:290). attempt 1/5 means it recovered on the next try. Please don't let anyone "repair" a give-up that doesn't exist.
• 22:13:33 REST reset is deliberately classified transient and downgraded out of Sentry (src/api/rest.rs:591-612, phrase list at src/core/observability.rs:47-54).

Two things argue against a backend cause. The pre-prod build was pointed at production (https://api.tinyhumans.ai/... in the excerpt above), and a production session on the current build held one unbroken socket for 49 minutes with zero close/reset/timeout events. And #5636 reports the third-party host ac.inference.sh/mcp dropping 14x in this same pre-prod session — a second unrelated remote failing far more often points at the reporting machine's network path.

So: the transport events are noise. What is real is the sentence in the Impact section — "any in-flight agent turn is dropped with no recovery path". That is a genuine client defect, it is not what the title describes, and it is worth keeping this issue open for. The rest of this comment is its fix.

Root cause

The outbound channel outlives the connection. emit_tx/emit_rx is created once per connect() (src/openhuman/platform/socket/manager.rs:264-265) and moved into ws_loop (manager.rs:274-276), which passes it by &mut into each run_connection. So a message queued while the socket is dead is not dropped — it is flushed onto the next socket, which has a different sid and whose roster the backend has already cleared.

The three hosted-brain handlers tokio::spawn their work and emit the result through that channel with no connection guard: orch:effect:send_dm emits at event_handlers.rs:101, orch:tool_call at event_handlers.rs:117, orch:effect:evict at event_handlers.rs:133. A device tool that is still running when the socket drops therefore emits its result into the next connection, where nothing is waiting for it.

There is a second half that makes it worse, and it is the part worth reading twice. orch:tool_call is at-least-once with server-side redelivery, guarded by a call_id latch claimed on entry (src/openhuman/hosted/orchestration/effect_executor.rs:434, 496-506). The claim is released only when dispatch_device_tool returns Err on a local-execution tool (effect_executor.rs:463-471). A call that succeeds stays latched forever. So when the result never reaches the wire, the backend's redelivery hits the latch and gets the synthetic frame {"accepted": true, "status": "running", "duplicate": true} (effect_executor.rs:441-449) instead of the real result — the brain waits on a tool_completion that can never arrive. The code already names this exact hazard for the dispatch-failure path (effect_executor.rs:456-462); it just doesn't cover the succeeded-but-undeliverable path.

Net effect: today the result is delivered orphaned; naively dropping it strands the brain permanently. The fix has to do both — suppress the orphaned emit and release the claim so redelivery genuinely re-runs.

Where

• src/openhuman/platform/socket/event_handlers.rs:93-103 (orch:effect:send_dm), :107-119 (orch:tool_call), :125-135 (orch:effect:evict)
• src/openhuman/platform/socket/medulla/workflows.rs:162 — visibility widening only
• src/openhuman/platform/socket/ws_loop.rs:165 — optional backstop

The change

1. Widen connection_generation() at medulla/workflows.rs:162 from pub(super) to pub(in crate::openhuman::platform::socket), matching begin_connection_generation / end_connection_generation on the two lines below it. Right now it is only visible inside socket::medulla, and event_handlers is a sibling.

2. In each of the three handlers, snapshot the generation before the spawn and gate the emit on it, releasing the claim when it cannot be sent:

"orch:tool_call" => {
    let tx = emit_tx.clone();
    let connection_cancel = super::medulla::workflows::connection_generation();
    tokio::spawn(async move {
        // Deliberately NOT a `select!` against the token: dropping the tool
        // future mid-flight would leave its call_id latched with no result,
        // which is the failure this is meant to prevent. Let it finish, then
        // gate the emit.
        if let Some((call_id, result)) = effect_executor::handle_tool_call(&data).await {
            if connection_cancel.is_cancelled() {
                effect_executor::release_call(&call_id);
                log::debug!(
                    "[socket] orch:tool_result undeliverable (socket closed); \
                     released claim call_id={call_id}"
                );
            } else {
                emit_via_channel(&tx, "orch:tool_result", result);
            }
        }
    });
}

Note this differs on purpose from the tokio::select! { biased; _ = connection_cancel.cancelled() => ... } shape used at medulla/workflows.rs:253-270 and :287-310. That work is a registration read with no latch, so cancelling it mid-flight costs nothing. A device tool call has a side effect and a claim, so it must be allowed to complete; only the emit is gated. Same treatment for send_dm and evict — per the comment at effect_executor.rs:485-487 both feed the same SEEN_CALL_IDS set.

3. Optional backstop at ws_loop.rs:165, next to the existing cancel_all(): drain whatever is still queued when a connection ends, so a message that slipped through between the emit and the cancellation isn't flushed onto the next socket either.

shared.ack_registry.cancel_all();
let dropped = drain_pending_emits(&mut emit_rx);   // while rx.try_recv().is_ok() { n += 1 }
if dropped > 0 {
    log::warn!("[socket] dropped {dropped} queued emit(s) on disconnect");
}

This also covers the one emit_with_ack caller (src/openhuman/security/devices/tunnel_client.rs:82), whose waiter cancel_all() already fails at ws_loop.rs:165 while its message is still queued for delivery — the caller sees Socket ack channel dropped (manager.rs:337-339) and the server does the work anyway.

How to prove it

• drain_pending_emits as a free function is directly unit-testable in ws_loop_tests.rs with the existing make_shared() harness: queue three messages, drain, assert the returned count is 3 and try_recv() is now empty. Reverting the drain fails it.
• For the handler guard, extract the gate into a small helper (emit_if_connected(&tx, &cancel, event, result, &call_id)) and test it twice — live token emits and does not release, cancelled token releases and emits nothing. Non-vacuous either way.
• Please do not write the test as "dispatch orch:tool_call with a cancelled generation and assert nothing lands on the channel". With the guard reverted, handle_tool_call in a unit-test process returns None for want of a real device tool anyway, so that test passes with and without the change — vacuous.
• End-to-end there is an existing signature to watch for: [socket] SIO ACK had no pending waiter ack_id=... (ws_loop.rs:700). It should stop appearing after a reconnect.

APP CHECK (for whoever runs the desktop app — one clone, one pair of ports, so this needs the manager, not a worker):

1. Start the app signed in, wait for [socket] Server ready — auth successful.
2. From the hosted brain, trigger a device tool call that takes >10s (any orch:tool_call route; run_local_agent is the side-effecting one and the most interesting).
3. While it is running, kill the socket — pull Wi-Fi for ~60s, long enough for the 50s ping deadline at ws_loop.rs:493 to fire — then restore it.
4. Confirms the defect: after Connection lost → EIO OPEN → Server ready, the log shows the orch:tool_result emit going out on the new connection, and/or [socket] SIO ACK had no pending waiter. The brain never completes the turn.
5. Refutes it: the result is emitted before the drop, or the turn completes normally after reconnect.

Risk / blast radius

connection_generation() is a process-global OnceLock<RwLock<..>> (medulla/workflows.rs:139-151) already consumed by medulla/workflows.rs:253, 287, 375 and medulla/mod.rs:514; widening its visibility changes no behaviour. The behavioural change is confined to the three orch:* handlers. The regression to watch for is over-releasing: release_call on a call whose result did go out would let a redelivery re-execute a side-effecting tool, so the release must sit strictly in the is_cancelled() branch. The ws_loop drain changes plain emit() from "delivered late on the next socket" to "dropped" — that is the intended semantic (the backend replaces a socket's whole entry on each registration, per medulla/workflows.rs:251-253), but it is a real behaviour change and deserves its own line in the PR description.

Not covered by this

• The three transport events in the issue body. They are transient network noise; nothing here changes how often they occur.
• The claim that the WebSocket and REST failed simultaneously at 22:13 — only the REST line is quoted, no socket line at 22:13 is shown. The "backend process restart" inference in the Fix direction rests entirely on that pairing and is unsupported as filed. Worth confirming against the raw pre-prod log before anyone acts on it.
• The 50s detection window itself (ws_loop.rs:493, pingInterval + pingTimeout + 5s). That value is correct for Engine.IO v4 — the server pings, so the client cannot probe faster without adding an application-level ack. Halving detection would be a separate change.
• "error sending request" in TRANSIENT_TRANSPORT_PHRASES (src/core/observability.rs:53) matches every reqwest send failure including DNS and connection-refused, so a sustained outage on the REST path produces zero Sentry events. Deliberate noise trade-off, but it means "no Sentry issues" can never be evidence this path is healthy.

Confidence: high on the mechanism and the code sites — every line above was read on e1c332bf0 and the latch behaviour is spelled out in the existing comments at effect_executor.rs:425-433 and :456-462. Medium on the verdict that the transport events themselves are noise, since I could not see the reporter's raw log and am reasoning from event rarity plus the #5636 correlation.

— fleet research, verified against main @ e1c332b

[M3gA-Mind]

Re-scoping this, and fixing the defect underneath it.

I re-verified the research verdict against main @ e1c332bf0 before acting on it, and it holds. The three transport events in the body are not a defect — each is an already-handled path:

• 17:57:59 Connection lost: Server closed connection is a WebSocket Close frame (src/openhuman/platform/socket/ws_loop.rs:508-510). It resets the failure streak and the backoff to 1s (ws_loop.rs:181-190), so recovery is ~1s.
• 21:40:58 Connection failed (attempt 1/5) — there is no 5-attempt cap. FAIL_ESCALATE_THRESHOLD = 5 is only the count at which one Sentry event is emitted; the loop retries forever with backoff doubling to a 30s ceiling. attempt 1/5 means it recovered on the next try. Worth stating plainly so nobody "repairs" a give-up that does not exist.
• 22:13:33 REST reset is deliberately classified transient and downgraded out of Sentry (src/api/rest.rs, phrase list in src/core/observability.rs).

What is real is the sentence in the Impact section — "any in-flight agent turn is dropped with no recovery path." That is a genuine client defect, it is not what the title describes, and it is worth keeping this issue open for. I have re-titled nothing and changed no labels; treat the scope as that sentence.

The defect. The outbound channel outlives the connection. emit_tx/emit_rx is created once per connect() (socket/manager.rs:264) and passed by &mut into every run_connection inside the reconnect loop (ws_loop.rs:154). So a message queued while the socket is down is not dropped — it is flushed onto the next socket, which has a different sid and whose roster the backend has already cleared.

The three hosted-brain handlers tokio::spawn their work and emit through that channel with no connection guard: orch:effect:send_dm (event_handlers.rs:93-103), orch:tool_call (:107-119), orch:effect:evict (:125-135). A device tool still running when the socket drops emits its result into the next connection, where nothing is waiting for it.

The second half is what makes it unrecoverable rather than merely lossy. orch:tool_call is at-least-once with server-side redelivery, guarded by a call_id latch claimed on entry (hosted/orchestration/effect_executor.rs:434, :496-503). The claim is released only when dispatch returns Err on a local-execution tool (:463-472). A call that succeeds stays latched forever — so when the result never reaches the wire, the backend's redelivery hits the latch and is answered with the synthetic {"accepted": true, "status": "running", "duplicate": true} frame (:441-449) instead of the real result. The brain then waits on a tool_completion that can never arrive. The code already names this exact hazard for the dispatch-failure path (:456-462); it just does not cover the succeeded-but-undeliverable path.

So suppressing the orphaned emit is only half a fix — on its own it would strand the brain permanently rather than delivering the result to the wrong place. Both halves are needed.

PR: #5719. It gates the emit in all three handlers on a snapshot of the connection generation taken before the spawn, releases the call_id claim when the result cannot be sent, and drains the emit channel when a connection ends as a backstop. Two notes on the shape:

• The tool future is deliberately not select!ed against the token, unlike the registration reads in medulla/workflows.rs:253-270. Dropping a device-tool future mid-flight would abandon a side-effecting call with its call_id still latched and no result — the exact failure being fixed. The work is allowed to finish; only the emit is gated.
• The release sits strictly inside the cancelled branch. Releasing a call whose result did go out would let a redelivery re-execute a side-effecting tool, which is the regression to watch for in review.

Behaviour change worth flagging: the drain turns a plain emit() queued during a disconnect from "delivered late on the next socket" into "dropped". That is the intended semantic — the backend replaces a socket's whole entry on each registration — but it is a real change and it is called out in the PR.

Still open, and not addressed by that PR:

• The 22:13 simultaneity claim. The body infers a backend process restart from the WebSocket and REST failing together, but only the REST line is quoted; no socket line at 22:13 appears. That inference is unsupported as filed and should be checked against the raw pre-prod log before anyone acts on it.
• Two things argue against a backend cause: the pre-prod build was pointed at production, and a production session on the current build held one unbroken socket for 49 minutes with zero close/reset/timeout events. Pre-prod: ac.inference.sh MCP server transport drops repeatedly (14x) — reconnects but unstable #5636 also reports the third-party host ac.inference.sh/mcp dropping 14x in this same session — a second unrelated remote failing far more often points at the reporting machine's network path.
• "error sending request" in TRANSIENT_TRANSPORT_PHRASES (src/core/observability.rs:53) matches every reqwest send failure including DNS and connection-refused, so a sustained REST outage produces zero Sentry events. A deliberate noise trade-off, but it means "no Sentry issues" can never be evidence this path is healthy.

APP CHECK — the one thing I could not do from a worker session (one clone, one pair of ports; this needs whoever runs the desktop app):

1. Start the app signed in; wait for [socket] Server ready — auth successful.
2. From the hosted brain, trigger a device tool call that takes >10s — any orch:tool_call route; run_local_agent is the side-effecting one and the most interesting.
3. While it is running, kill the socket — pull Wi-Fi for ~60s, long enough for the 50s ping deadline at ws_loop.rs to fire — then restore it.
4. Confirms the defect: after Connection lost → EIO OPEN → Server ready, the orch:tool_result emit goes out on the new connection, and/or [socket] SIO ACK had no pending waiter appears. The turn never completes.
5. Refutes it: the result is emitted before the drop, or the turn completes normally after reconnect.

With the PR applied the same run should instead log orch:tool_result undeliverable (socket closed before the result was ready); released claim call_id=… and the redelivery should re-run the tool.

— fleet research, verified against main @ e1c332b

[M3gA-Mind]

Follow-up: this issue is only half fixed by #5734, and the other half needs a decision I should not make alone.

Review on that PR (Codex, corroborated by CodeRabbit) caught a mistake in my original approach, and correcting it changes what the PR delivers — so recording it here rather than only in the PR.

What I had. Suppress the orphaned emit and release the call_id claim, so the backend's at-least-once redelivery would re-run the call and the turn would complete.

Why that was wrong. Every pre-existing release_call sits on the dispatch-failed path, where nothing happened. Releasing on the succeeded-but-undeliverable path re-executes work that already ran:

• handle_send_dm peer path → execute_send_dm → handle_tinyplace_signal_send_message(recipient, plaintext). No call_id reaches that transport, so a redelivery re-sends the Signal message and a human sees it twice.
• orch:tool_call on a local-execution tool → dispatch_device_tool spawns a sub-agent. A redelivery spawns a second one.
• Only persist_reply is safe to repeat — it is idempotent on message_id = "reply:{call_id}".

So the release is gone. Between a stalled turn and a duplicate message delivered to a person, the stall is the recoverable one.

What #5734 now delivers: the result is no longer emitted onto the next connection (a different sid, whose roster the backend has already cleared), the check and the send are atomic under the generation lock so a disconnect cannot slip between them, and the emit channel is drained on disconnect. The stall itself is now a warn! naming the exact stranded call_id instead of silence.

What is still open — the "no recovery path" in the Impact section. The claim stays latched, so the redelivery is answered with the synthetic {"accepted": true, "status": "running", "duplicate": true} frame and the brain's tool_completion never arrives. Unchanged from before, so no regression, but not fixed either. Two ways to close it, and neither belongs in a client-side patch:

1. Cache the result for replay on the next connection, once the backend can correlate a result arriving on a new sid against the original call_id. Needs a backend contract, not just client storage.
2. A server-side deadline on the tool call, so an unanswered call fails and the brain moves on rather than waiting forever. Cheaper, and arguably right regardless — a device can also simply be shut down mid-call, which no client-side fix can cover.

MAINTAINER CHECK: which of those the hosted brain should take, and whether a tool_call deadline already exists server-side. If one does, the stall may already self-resolve after a timeout, and the remaining work is just to confirm the window is tolerable. I could not determine this from the repo — the orchestration brain is not in this tree.

The APP CHECK in my earlier comment still stands as the way to observe the behaviour end to end; with #5734 applied the expected log line is now [socket] orch:tool_result undeliverable (the socket closed before the result was ready); dropped rather than sent on the next connection. The effect ran, so its claim stays latched and the turn will not complete: call_id=….

— fleet research, verified against main @ e1c332b