serve: close a connection that falls behind, and size its buffer in bytes

[jaredLunde]

Fixes serve audit defect #2 (HANG: a connection dropped from the fanout stays open and silent forever).

Problem

• OutFanout::broadcast dropped a sink on its first failed try_send, but the attach loop's reply_tx clone kept the channel open. The socket stayed up, pings kept arriving, and the client's commands still ran, but it never got another frame. It never reconnected, its session counted as attached and was never reaped, and approvals waited on it forever.
• Healthy clients were being dropped too. The per-connection buffer was a 1024-frame channel. The daemon runtime is single-threaded by default, so the session writer moves a whole model read into that channel before the socket writer runs. A 3000-delta burst measured 252 KB queued at once and dropped a client that never stopped reading (reproduced: stalled after ~1000 deltas).

Design decision

Buffer by bytes, then drop, and a drop closes the socket.

• ConnTx/ConnRx replace the bounded mpsc. The cap is 8 MiB of live, unwritten output per connection (CONN_BACKLOG_MAX_BYTES), about 33× the measured burst. The bound exists to cap memory, so it is counted in bytes.
• Not backpressure. Nothing upstream can wait. The event sink is synchronous. One writer task feeds every attached connection, so awaiting a slow one would stall the others. Approval, elicitation and app frames are broadcast from sync code under the fanout lock. A frame that doesn't fit can only be lost, and a lost frame is a silent hole in the stream.
• A refused frame cancels the connection's dropped token:
  • The send task stops writing at once, even mid-send to an unread socket, and sends Close 1013 Try Again Later (bounded).
  • The read loop stops and unpins, which starts the reaper clock.
  • The client reconnects and is caught up.
• Frames seeded on attach (catch-up, views, turn replay) are queued outside the cap, so a reconnect is never dropped for its own replay. As a side effect, replay into the old 1024 slots no longer truncates (overlaps audit pi #13).
• A dropped HTTP POST answers 503 instead of "session ended".
• The send task's cancel and close are now bounded, so teardown can no longer hang on a client that stopped reading.

Tests

• tests/serve_slow_client.rs (the audit repro made permanent; both tests failed before this change):
  • a_reading_client_gets_every_frame_of_a_burst: 3000 deltas, every byte arrives in order.
  • a_client_that_stops_reading_is_disconnected_and_can_reattach: a 24 MiB turn with a paused reader; the socket closes, and the reconnect is caught up and answered.
• Unit tests in serve.rs: the cap boundary (dropped exactly past it, no holes, token fires), a draining connection is never dropped, and seeded frames are exempt.
• cargo fmt, clippy --all-targets -D warnings, the full agent suite and dprint check all pass. ARCHITECTURE.md has a new "Why a slow connection is closed" section.

🤖 Generated with Claude Code

https://claude.ai/code/session_01JimHGjsfk2Ktm5GxyZJKKk