perf(cluster): low-latency event channel for proof status#96
Open
tamirhemo wants to merge 6 commits into
Open
Conversation
tamirhemo
changed the title
tamirhemo
force-pushed
the
tamir/streaming
branch
from
d00bb5b to
476b2da
Compare
tamirhemo
changed the title
Replaces the coordinator's 500ms ProofRequestList polling with a
Postgres LISTEN/NOTIFY-driven push channel. Cluster-internal pickup
latency drops from 0–500ms to ~5ms. Sets up the gateway-side hub the
SDK long-poll RPC will subscribe to next.
Wiring (api -> coordinator + gateway):
- migrations/20260427... : AFTER INSERT/UPDATE trigger on
proof_requests emits pg_notify('proof_event', json_payload).
- bin/api: ClusterEventsService.SubscribeProofEvents server-streams
those events, fed by a single PgListener + tokio broadcast.
- bin/coordinator: claim_one(proof_id) on each Pending event using
a single-row ProofRequestGet (O(1) per event vs the old O(N)
full list). Safety-net loop drops to 1s and now only runs the
full list+reconcile path; event-driven path skips reconcile.
- bin/network-gateway: ProofEventsHub demuxes the stream into
per-proof_id broadcast::Senders. Empty today; Lever 3's long-poll
will subscribe per request_id.
Local proto only — cluster_events.proto lives in sp1-cluster, not
sp1-prover-types, so this ships without a sp1 release.
Verified live on the running cluster: INSERT -> coordinator claim in
~2ms, INSERT -> gateway sees Pending+Failed transitions in ~3.5ms.
Known follow-ups:
- Gateway hub never garbage-collects per-proof Senders. The forget()
hook is in place but unused until Lever 3 wires the long-poll.
- SDK still polls every 2s; that's Lever 3.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Holds the existing get_proof_request_status RPC open until the proof transitions out of its current state, or 25s elapses (well under the SDK's 60s tonic per-RPC deadline at crates/sdk/src/network/grpc.rs:9). Mainnet is unaffected — only the gateway's server-side handler changes; the SDK's polling loop, the proto, and request/response types are all untouched. Subscribes to ProofEventsHub before fetching the row to close the race where a transition fires between SELECT and recv. Recovers cleanly from broadcast lag by re-fetching. Combined with the cluster-internal NOTIFY chain already on this branch, the SDK now sees terminal status within ~5ms of the coordinator updating it, instead of 0–2s on the next polling tick. For sub-1s proofs this is the difference between a 200ms proof returning at 200ms and at ~2.2s. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanup paths so the in-process channel map can't grow unbounded: 1. Terminal-publish: when the api-fed event stream broadcasts a terminal cluster status (Completed/Failed/Cancelled), drop the per-proof entry immediately after the send. No further NOTIFY events will fire for that proof_id, so the channel is dead weight. Subscribers still attached drain any buffered values and observe Closed on the next recv. 2. Drop-on-idle: subscribe() now returns a Subscription RAII guard. On drop, it decrements receiver_count and asks the hub to GC the entry iff no receivers remain (DashMap::remove_if locks the bucket so a concurrent subscribe can't sneak a receiver in between the count check and the removal). Catches the case where someone subscribes to an already-terminal proof — terminal-publish can't fire a second time, so without this the entry would leak. Tests cover create/drop, multi-subscriber, terminal eviction with active receiver, non-terminal keep-alive, post-drop subscribe, and basic recv flow. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
tamirhemo
force-pushed
the
tamir/streaming
branch
from
3f55974 to
6eca21e
Compare
tamirhemo
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
End state: a self-hosted gateway proof finishes within milliseconds of the cluster completing it (vs. up to ~2s before). Replaces the cluster's polling chain with a Postgres
LISTEN/NOTIFYpush from the api → coordinator + gateway → SDK.Summary
proof_requestsemitspg_notify('proof_event', json).ClusterEventsService.SubscribeProofEventsserver-stream, fed by onePgListener+tokio::sync::broadcast. Local proto inside sp1-cluster — nosp1-prover-typeschange required.ProofRequestListpoll → event-drivenclaim_one(proof_id)(single-row fetch). Safety-net poll keeps a 1s reconcile path for missed NOTIFYs.ProofEventsHubdemuxes the stream into per-proof_idbroadcast::Senders. Bounded by terminal-publish eviction + RAII drop on idle.get_proof_request_statuslong-polls (up to 25s) on the hub before returning. No SDK change, no proto change — the SDK's existingwait_proofloop just observes "the call took longer to return," and mainnet's handler still returns immediately so polling clients are 100% unaffected.Verification
INSERT proof_requests→ coordinator dispatch in ~2ms, gateway sees Pending+Failed transitions in ~3.5ms (synthetic test on the running cluster).End-to-end: cluster pickup ~5ms (was 0–500ms), SDK observes terminal status ~5ms after the cluster commits it (was 0–2000ms).
🤖 Generated with Claude Code