Skip to content

fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops - #6090

Open
mfethe1 wants to merge 1 commit into
block:mainfrom
mfethe1:fix/fanout-gap-signal
Open

fix(relay): emit BUZZ_SYNC_REQUIRED gap signal on fan-out backpressure drops#6090
mfethe1 wants to merge 1 commit into
block:mainfrom
mfethe1:fix/fanout-gap-signal

Conversation

@mfethe1

@mfethe1 mfethe1 commented Aug 17, 2026

Copy link
Copy Markdown

What

Relay-side fix for the sync-lag diagnosis finding that the first broken boundary in relay->client sync is silent fan-out frame drops under backpressure. When the priority ctrl channel's fan-out buffer is full, the relay now emits a gap signal to the affected connection instead of dropping frames invisibly, letting clients know their view is stale.

Contract (frozen)

  • Signal: ["BUZZ_SYNC_REQUIRED","backpressure"] on the priority ctrl channel.
  • Emitted only on fan-out drops (TrySendError::Full from the fan-out path). A closed ctrl channel cancels immediately; no signal into a closed channel.
  • sync_required(reason) is monomorphic over SyncRequiredReason::Backpressure — the relay cannot emit a reason the contract does not define.
  • Grace counter behavior preserved exactly as pre-existing.
  • buzz_fanout_dropped_frames_total counts every drop unsampled.
  • Disconnect counting is CAS-guarded (once_disconnect_counted) so concurrent disconnect paths count once per connection (8-thread regression test).
  • Protocol documentation lives in the protocol.rs module docs beside the RelayMessage definitions.
  • No client-side handling — relay-only by design; the client contract is frozen for a follow-up.

Diff

3 files / 736 ins / 27 del, all under crates/buzz-relay/ (handlers/event.rs, protocol.rs, state.rs). Single commit on current main (f956e6fe), no rebase.

Verification at exact head 0048e96

Two independent Windows (MSVC) hosts, separate gate stacks:

Gate Host A Host B (independent)
cargo test -p buzz-relay --lib --no-fail-fast 885 passed / 43 ignored / 3 failed 884 passed / 4 failed / 43 ignored
cargo fmt --all -- --check clean clean
cargo clippy -p buzz-relay --tests --all-features -- -D warnings clean clean

All failures are pre-existing host-shaped issues reproducing on the f956e6fe base:

  • api::git::policy::tests::bash_hmac_matches_rust_hmac and bash_hmac_single_ref — WSL bash resolution.
  • api::mesh_demo::tests::demo_join_forwarded_arm_round_trips_echo — redis-gated.
  • (Host B only, intermittent) telemetry::tests::trace_context_lookup_does_not_enable_callsites — passes in isolation (-- --exact); the test exists unchanged at base (telemetry.rs:475) and this diff touches zero telemetry code, so the flake is scheduler-shaped (global tracing callsite registration leaking between parallel tests), not diff-shaped. A Host A rerun reproduced the same intermittent flake.

Per the repo's ci.yml path-filter semantics, a crates/** diff triggers rust-lint / unit-tests / backend-integration on ubuntu — maintainer-approved CI runs on this PR are the supported-environment complementary evidence.

Attribution: commits carry Co-authored-by + Signed-off-by for the human operator; no AI attribution trailers.

Evidence (contract 2026-08-17, tier T2 protocol/relay)

Locked-head exception applies: paired-gate receipts are locked at this exact head SHA, so this evidence lives in the body rather than a docs/pr-evidence/6090/ commit (a new commit would move the head and invalidate the receipts).

Before — problem reproduced at base f956e6fe (read-only diagnosis, 2026-08-16):

  • Production-relay probe (NIP-42-authenticated raw WS subscriber, 2026-08-16 19:02–19:11 UTC): healthy-path pushes arrive in 254–457 ms and are queryable on first poll — proving the reported lag is a loss branch, not latency.
  • Static trace at f956e6f: try_send_ws_message (state.rs:592-620) on TrySendError::Full DROPS the frame with a server-side-only warn!; drop_count is never signaled to the client. Below the grace limit (default 15, config.rs:566-580) the socket stays "connected" with permanent holes in its event stream.
  • Runtime consequence: a briefly backgrounded consumer sees events "late until restart"; events were persisted all along and only reappear via reconnect-replay (desktop relayReconnectReplay.ts, mobile relay_session.dart:261-262).
  • Diagnosis record + probe captures: RESEARCH/SYNC_LAG_DIAGNOSIS_LANE3_HANDOFF_2026_08_16.md, .scratch/sync-lag-probe/probe_long.log, exp2.log (agent workspace).

After — fix verified at head 0048e96a0 (git rev-parse HEAD in the same shell as the runs):

  • Runtime harness (local relay 127.0.0.1:3000, BUZZ_SEND_BUFFER=1, scratch DB buzz_gaptest; raw tokio-tungstenite + NIP-42 auth, victim stalled 8s while publisher floods kind-1): victim received 5 buffered EVENTs, then ["BUZZ_SYNC_REQUIRED","backpressure"] as the ctrl channel saturated at capacity 8 — the exact path that used to drop silently now emits the contracted client-visible gap signal. Capture taken during development at pre-fix head 3cf682b50; the delta to this head is the metrics-inflation fix below + commit trailers (no mechanism change to the signal path).
  • Wire contract pinned by the exact-bytes test (protocol.rs) and the end-to-end test through send_fanout_frames (handlers/event.rs) — both run green in the two-host gate table above at this exact head.
  • Every drop counted unsampled (buzz_fanout_dropped_frames_total); the harness also caught drops-after-cancel re-incrementing buzz_ws_backpressure_disconnects_total (observed 46 for one connection) — fixed via is_cancelled() idempotence guards with regression test repeated_drops_after_cancel_do_not_inflate_disconnect_counter, in this head.

End-to-end: the first broken boundary in relay→client sync (silent fan-out drops under backpressure) is no longer silent: the relay emits a contracted gap signal on the priority ctrl channel, letting clients know their view is stale without tearing down the socket. Client-side replay on the signal is deliberately OUT of this PR (frozen client contract; follow-up lane).

Verified-by: paired Bingo + Winnie gate at 0048e96a0 (two independent hosts, table above); gate-call PASS event 77628f7d (Bingo/Winnie paired authority, Michael tiebreak, Ernie concurrence 2026-08-17T00:37Z). Contract: agent-workspace GUIDES/PR_EVIDENCE_CONTRACT.md.

…re drops

When a subscriber's outbound data channel is full, EVENT fan-out frames
are dropped silently: the socket stays up, the grace counter increments
server-side only, and the client's live view keeps a permanent hole
until the grace limit kills the connection (or a reconnect replay
accidentally heals it). Lane 3 of the sync-lag diagnosis proved this is
the first broken boundary behind "messages arrive late until restart".

Give the client a machine-readable gap signal instead:

- RelayMessage::sync_required() formats the extension frame
  ["BUZZ_SYNC_REQUIRED","backpressure"]. The constructor is deliberately
  monomorphic so the relay cannot emit a frame the wire contract does
  not define; protocol.rs module docs define the contract and delivery
  rules (unknown relay->client heads are non-fatal per NIP-01 client
  convention).
- The EVENT fan-out path — ConnectionManager::send_fanout_frame,
  renamed from send_to_text_bytes to make its role explicit — routes
  through try_send_fanout_frame:
  * every dropped frame (full data channel, closed data channel, or
    connection already gone) increments buzz_fanout_dropped_frames_total
    with no sampling, so a silently lost EVENT always leaves a
    telemetry trail;
  * on a Full data channel with a live socket, the gap signal is queued
    on the connection's priority control channel — never on the data
    channel it signals about, never as a human-readable NOTICE — and
    the existing consecutive-full grace counter decides cancellation
    exactly as before;
  * when the control channel is itself full or closed, in-band
    signaling is impossible: cancel at once and let reconnect replay
    recover the missed events;
  * cancellation is idempotent under concurrent drops: a
    compare-exchange on ConnEntry::backpressure_disconnect_counted
    picks exactly one of the parallel droppers to increment
    buzz_ws_backpressure_disconnects_total (a runtime flood harness
    showed the unguarded is_cancelled->cancel sequence inflated the
    disconnect counter 46x for one connection).
- Non-fan-out sends (CLOSED notices etc.) share the try_data_send
  success path but keep the old drop semantics — no gap signal, no
  dropped-frame counting.

Tests: 12 new tests plus a wire-shape pin (exact bytes) in the protocol
table test — full/closed/gone/ctrl-full branches, grace-counter
interaction, the signal never riding the data channel, disconnect
idempotence (including an 8-thread concurrent-drop case), and an
end-to-end case through the real send_fanout_frames path.

Gates on this Windows host: cargo fmt --check clean; cargo clippy
-p buzz-relay --tests --all-features -- -D warnings clean; cargo test
-p buzz-relay --lib = 885 passed / 43 ignored, failing only on 3
pre-existing host failures that reproduce on the base commit f956e6f
untouched by this diff (2 bash-HMAC tests resolve `bash` to WSL on
this host; the redis-gated mesh_demo test returns 504 with local
Redis up).

Co-authored-by: Michael Feth <michael@jira-flow.com>
Signed-off-by: Michael Feth <michael@jira-flow.com>
@mfethe1
mfethe1 requested a review from a team as a code owner August 17, 2026 00:17
@mfethe1

mfethe1 commented Aug 17, 2026

Copy link
Copy Markdown
Author

Hi @themiguelamador — could you approve the pending CI workflow runs on this PR at head 0048e96a0?

Context on why the CI run matters beyond the usual signal: both dev hosts available to us are Windows, where several workspace gates fail at base for platform reasons. We therefore verified with targeted crate gates at the exact head on two independent Windows hosts (full evidence table in the PR description): cargo test -p buzz-relay --lib (885/3 and 884/4, all failures pre-existing host-shaped and reproducing on the f956e6fe base, the one-test delta isolation-verified as a scheduler flake on both hosts), plus clean cargo fmt --check and cargo clippy -p buzz-relay --tests --all-features -- -D warnings.

Per the repo's ci.yml path-filter semantics, this crates/** diff triggers rust-lint / unit-tests / backend-integration on ubuntu — the supported-environment evidence we cannot produce locally. The diff is relay-only Rust (3 files under crates/buzz-relay/), so mobile/desktop jobs skip by the repo's own gate semantics.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant