fix(buzz-acp): gate reactions to kind-9 + raise relay rate limits - #5991
fix(buzz-acp): gate reactions to kind-9 + raise relay rate limits#5991BrennonTWilliams wants to merge 3 commits into
Conversation
Reactions (kind 7) and deletions (kind 5) were also getting 👀 reactions, so two co-located agents echoed 👀 at each other in an infinite loop (1.4M events, ~475/s). Gate the seen reaction to kind-9 chat messages only.
7 agents polling/queuing burst past the HTTP bridge 300/min default. Raise agent standard to 6000/min and human API to 60000/min.
Same bug class as the 👀 loop fixed in 4fc050300: the `reaction_ids` vec built at the top of run_prompt_task fed both `react_working` (adds 💬) and `clear_reactions` (removes 👀 + �) without filtering by event kind. When a batch contained kind-7 reactions or kind-5 deletions, those events got cosmetic 💬 reactions whose subsequent cleanup deletions became new batch events — closing the loop. 2026-08-15 incident in #ci-cd: ~50 events/sec of mixed kind-5 + kind-7 across Builder / QA / Watcher for ~12 minutes before subsiding. Gate `reaction_ids` to kind-9 only at construction (pool.rs:1559) so both the add and cleanup paths stop touching non-chat events. Pin the filter with two regression tests covering mixed-kind batches and a kind-9-absent batch (the latter exercises ReactionGuard::new dropping the rest client when ids are empty). Signed-off-by: Brennon Williams <88200+BrennonTWilliams@users.noreply.github.com>
themiguelamador
left a comment
themiguelamador
left a comment
There was a problem hiding this comment.
The kind-9 gates address the reaction echo loop, but two blocking issues remain. First, the new reaction_ids tests duplicate the filter instead of calling production code, so removing the production guard leaves both regressions green; the 👀 gate also has no regression test. Second, the deployment commit is unsafe and internally inconsistent: its prose says agent API 6,000/min while Compose sets 60,000/min, enforce_http_admission still uses human_api_calls_per_min for every caller (so the agent API knob does not fix the reported HTTP path), and the change raises the human quota from 300 to 60,000/min for every Compose deployment. The first two commits also currently fail DCO. I prepared a clean signed candidate ending at cf75da56c: it drops the quota commit, extracts production helpers used by both reaction paths and their tests, and repairs signoffs. Verified with focused regression tests, all 781 buzz-acp unit tests, nine lifecycle integration tests, cargo fmt --all --check, strict Clippy, and diff checks. A role-aware HTTP admission change should be handled separately with security/load evidence. Maintainer edits are enabled, but the head-pinned force-with-lease push to the contributor fork returned 403.
2 participants
Summary
Three fixes carried locally on our self-hosted relay, now upstreamed.
1. Gate the 👀 "seen" reaction to kind-9 chat messages only
The 👀 seen-reaction was emitted for every accepted event with no kind filter. Two agents co-located in a channel each reacted to the other's 👀, whose kind-5 cleanup deletions were themselves reacted to — a mutual echo loop that generated ~1.4M kind-7 + ~37k kind-5 events at ~475/s and deadlocked both agents.
Fix: only react to kind-9 chat messages (
if accepted && event_for_steer.kind.as_u16() == 9).2. Apply the same kind-9 gate to the 💬 "working" reaction
The 💬 "working" reaction (
react_working+clear_reactions) was a second path that fired on all events and re-introduced the same echo loop. The guard lives inrun_prompt_task—reaction_idsis now filtered to kind-9 only, which gates both the add (react_working) and the cleanup (clear_reactions). Two regression tests pin it.3. Raise relay rate limits for multi-agent load
HTTP-polling agents hit the hardcoded
human_api_calls_per_min(300/min) admission path and entered a 429 retry storm. Raised the relevant knobs so agent traffic isn't throttled by the human quota.All three are deployed and verified on our relay.