Skip to content

docs: field writeup — running a local multi-model debate lab on a self-hosted relay - #5496

Open
artemtrofymenko wants to merge 15 commits into
block:mainfrom
artemtrofymenko:docs/local-debate-lab
Open

docs: field writeup — running a local multi-model debate lab on a self-hosted relay#5496
artemtrofymenko wants to merge 15 commits into
block:mainfrom
artemtrofymenko:docs/local-debate-lab

Conversation

@artemtrofymenko

@artemtrofymenko artemtrofymenko commented Aug 10, 2026

Copy link
Copy Markdown

What this is

A field writeup from standing up Buzz as a local multi-model debate lab on Windows: a self-hosted relay in Docker, the packaged desktop, and four agents — Claude Code and Codex over OAuth, plus Qwen3.5 9B and Gemma4 12B locally through Ollama — arguing in one channel with turns driven by a Workflow.

Four documents under docs/local-debate-lab/, plus the workflow definition that ran the sessions.

The docs are written in Ukrainian. I am opening this as offered rather than assumed — if a non-English doc set is not something the project wants to carry, say so and I will close it, or translate it if that would make it useful.

Why it might be worth carrying

Most of what cost time here is already known to this tracker. What the writeup adds is the sequence: which symptoms lead to which cause, and what a self-hoster sees before they know the vocabulary to search for.

What this does not touch

No code. Documentation only.

@artemtrofymenko
artemtrofymenko requested a review from a team as a code owner August 10, 2026 17:22
artemtrofymenko and others added 8 commits August 10, 2026 19:33
@artemtrofymenko @claude
docs: add local debate lab runbook and findings
40d4087 
Runbook for standing up a multi-model debate on a self-hosted relay:
several agents in one channel, one on Claude Code over OAuth, the rest
on local models via Ollama.

Alongside it, two documents that exist because the setup was not
straightforward:

- DEBATE_SETUP.md records the measurement behind every config value.
  Notably the served Ollama context window (8192) versus the one the
  models advertise (262144), and the output budget these reasoning
  models need before any visible text appears.

- FINDINGS.md records product behaviour that cost time to diagnose.
  Two are candidate bug reports: a self-hosted relay configured with
  `localhost` in RELAY_URL silently breaks every agent, because the
  desktop normalizes loopback to 127.0.0.1 before handing the URL to
  the harness while the relay resolves the community from Host; and
  `claude auth status`, which the desktop uses as its auth probe,
  consults a different credential store than the ACP adapter and was
  observed disagreeing with runtime reality in both directions on the
  same machine in one day.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: drop the bad MAX_HISTORY_BYTES value and record the startup floors
fb945b7 
Setting BUZZ_AGENT_MAX_HISTORY_BYTES to 48000 looked reasonable next to a
32k-token window, but the crate validates it against MAX_PROMPT_BYTES and
refuses to start below 1 MiB — all ten harness processes exit immediately
with "all 10 agents failed to start". It is a byte cap on the transcript,
not a competitor to the token window.

Drop the variable from the recommended config and document the three
values the crate validates at startup, including that the real default is
16 MiB rather than the 1 MiB the crate README states.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: record tool-call reliability as the metric that picks a local m…
a9f6926 
…odel

A local model's fitness for this lab turns out to hinge on whether it holds
the shape of the publish command, not on how well it argues. Those are
different abilities and the second is invisible in general benchmarks:
qwen3.5 wrote sound rebuttals and then passed the prose itself as the shell
command's argument, so nothing ever reached the channel.

Five runs per configuration, validating that the tool call really is a
`buzz messages send --channel <uuid> --content "…"` invocation rather than
merely containing those words:

  gemma4-debate   15/15 across three prompt variants
  qwen3.5-debate  4/5 short prompt, 2/5 once extra rules were added
  ornith:9b       3/4
  lfm2.5:8b       0/4

Two lessons went into the guide. Extra rules in the system prompt hurt the
weaker model rather than helping it, so roles moved out of the prompt and
into the round message, leaving the prompt to answer only "how do I
publish". And the Activity panel — not the harness log — is where a silent
agent is diagnosed, because the log at info level never shows tool calls at
all and so looks identical whether the call was malformed or never
attempted.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: workflows cannot drive owned agents on the release build
900b26c 
A Workflow paces turns perfectly — delays honoured to the second, mentions
resolved — and the agents still never answer, because the engine signs its
messages with the relay key while every managed agent is stuck on an
owner-only gate that admits only the owner and their own verified agents.

The gate being stuck is block#2501, open since 0.4.23 and confirmed
through 0.5.8, with its fix (block#2505) unmerged. Worth recording here because
it derailed the diagnosis: managed-agents.json keeps two records per agent,
the UI writes the choice onto the definition record, and the harness gates
on the instance record. The raw file therefore shows both "allowlist" and
"owner-only" at once, and reading the wrong one leads straight to a false
conclusion — as it did for me before the upstream thread set it right.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: the respond-to workaround is verified, not speculative
42081bf 
Editing respond_to and respond_to_allowlist directly on the instance
records, with the app closed, survives startup: the harness came up with
respond_to=allowlist(1) and an agent answered a workflow-generated mention
52 seconds later, where every mention before the restart was dropped.

Nobody in block#2501 has posted a working workaround, so record the
fragility too: the app rewrites instance records during its own operations,
which means editing the agent in the UI can silently restore owner-only.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: what the debates actually showed, and the workflow that ran them
40ed4bd 
Three sessions in, the interesting difference between the models is not
what they know — all four understand the subject — but whether they can
run an argument: spot a specific flaw, concede their own, and narrow a
claim under criticism. Codex caught Claude proposing nftables owner-match
for a macOS example; Claude took the correction and listed its own error
among the session's unsupported claims. Neither local model did anything
of the kind, and qwen3.5 twice answered a direct request to concede by
asserting its objection had survived.

Also records the structural mistake that nearly produced the wrong
conclusion: the first run put the topic only in the opening step, so the
local models went looking for it in a channel already full of previous
sessions and picked up the wrong one. With the topic injected into every
step they were on topic immediately — so the failure was the harness, not
the model size.

Ships the workflow that ran the clean session, plus the two rules it
encodes: topic in every step, and str_starts_with as the trigger so a step
may quote the topic without re-triggering the run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: a workflow that runs but cannot be seen or stopped
2a80b45 
The Workflows screen said "No workflows yet" while the workflow was
triggering and posting. Upstream already has the cause — the HTTP bridge
narrows a multi-value #h filter to the lexicographically smallest channel
UUID (block#5053, block#4659, block#4804) — and our channel list matches it
exactly: the debate channel holds the largest UUID of four, so the query
resolved to a different channel entirely.

Recorded because the cost is larger than a missing list. With no row in
the UI there is no way to disable or delete, and three runs ended up
overlapping, each posting round instructions into the same channel with no
way to stop them from the app. The screen also reports the empty result as
"nothing created" rather than as a query that returned nothing, which is
what kept us looking in the wrong place.

Includes the SQL workaround, since creation still works and only the
listing is broken.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: give Gemma4 a realistic turn budget, note two engine limits
57276cb 
Gemma4 answered at 110s in one session against a 110s budget — it finished
exactly as the engine fired the next mention. Its latency varies about
twofold across runs (66s to 110s observed), so the budget after its steps
goes to 150s. The cloud agents stay at 60s against measured 19-25s.

Also records two engine limits found in the tracker rather than the hard
way: a single delay cannot exceed 270 seconds, which the documented "5m"
and "1h" examples both violate (block#3021), and the YAML `enabled`
field is inert because the scheduler reads a separate database column
(block#4639) — so disabling by editing YAML does nothing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
artemtrofymenko and others added 7 commits August 10, 2026 19:41
@artemtrofymenko @claude
docs: the recommended prompt workaround is worse here, and why
c7839e7 
block#2698 recommends a one-shot exemplar — a fabricated prior turn
showing the correct tool call — as the most reliable way to get a local
model to publish. Measured against a plain instruction on our two models it
was worse: qwen3.5 dropped from 5/5 to 3/5 correct commands, and the
failures show it transcribing the channel UUID out of the exemplar and
corrupting it. The exemplar states the UUID twice instead of once, which
helps a model that fails to choose the tool and hurts one that chooses
correctly but copies badly.

The competing-tool failure from the same issue did not reproduce: with a
second plausible tool present in every run, both models picked the right
one 20/20.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: the isolated benchmark overstates what a local model does in th…
783a9cd 
…e harness

Two consecutive workflow sessions, same prompts and config, different
topics: the cloud agents answered 7/7 both times, the local ones 5/5 and
then 1/5. In the failing session qwen3.5 was not idle — the harness log
shows two LLM calls seconds after the mention, then a clean end of turn
with nothing published, which is block#2698.

That bounds the reliability table recorded earlier. Those runs went
straight to Ollama with one system prompt and an empty history; the
harness adds buzz-agent's own prompt, the channel history and the full
MCP tool set, and the same models degrade there — unpredictably between
sessions. The isolated number is a ceiling on capability, not a measure of
reliability, and the document now says so.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko @claude
docs: prove the sibling gate instead of quoting the code comment
088d78f 
Finding 6 said owner-only admits sibling agents, sourced from a comment in
access_policy.rs — an intent, not an observation. Upstream then reported the
opposite, and the counter-example I reached for first was contaminated: the
kickoff message had @-mentioned every participant, so nothing distinguished
an agent waking an agent from a human waking both.

Redone so only an agent-authored mention could have woken the target — the
human mentioned Claude alone, "Codex" stayed plain text with no p tag — and
Codex answered Claude 23 seconds later. Codex carries an allowlist of one
entry that is not Claude, so `allowlist.contains(author)` was false and
`is_owner_or_sibling` is the only branch that could have admitted it.

Also records where to look when it fails: that function fails closed on a
transient REST error and caches the negative for the process lifetime
(block#5450), which reads as selective deafness rather than a fault.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko
docs: the measurements are from 0.5.8, not the version we wrote down
de7f2e4 
Three files claimed Buzz desktop 0.5.5 while FINDINGS.md's own
0.5.8-only section said otherwise — the app updated at the start of
the session and the header never caught up. A reader weighs this
whole report by that number, so a stale one is not a typo.

Also state plainly which versions were never tested: 0.5.11 and
0.5.14 are already being reported in the tracker, and nothing here
was re-run on them.

Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko
docs: re-check the findings on 0.5.14, and fix the advice that misled me
9df7d2d 
The desktop auto-updated 0.5.8 -> 0.5.14 mid-session, so every finding
got a second look. None of them fell over: the loopback host check, the
auth probe, the README's key claim, the model filter, the author gate
and the liveness probe all behave exactly as recorded. Findings 8 and
10 are marked not re-tested rather than quietly assumed.

Two things the update itself taught us. The managed-agents.json
workaround survived it — the harnesses came back up with
respond_to=allowlist(1), so the compiled owner-only clamp is not
enabled in this build. And the harness log is appended to across runs,
not recreated, so "read the first line" sent me to a week-old startup
line and briefly convinced me the update had broken the gate. It is
the last one that describes the running process.

Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko
docs: close out findings 8 and 10, and correct what fail-closed means
1e8f286 
Finding 8 is now reproduced rather than recalled: a standalone
buzz-acp pointed at an unmapped host logs the 404 as a terminal error
and exits without a single retry attempt — while the same binary,
seven minutes earlier, rode out a relay restart with "autonomous
reconnect attempt 3/5 … succeeded". The retry machinery works; 404
just never reaches it. That makes the finding about error
classification, which is a smaller and more fixable claim.

Finding 10 gets its actual culprit: extract_channel_id_from_filter in
the relay returns the first parseable #h value and scopes the SQL to
it, so the post-filter can only narrow what that one channel returned.
Two functions share the name and disagree — the one in api/bridge.rs
returns None for multi-value filters, the deciding one in
handlers/req.rs does not.

Finding 1 needed a correction I earned the hard way. The relay is
fail-closed only for hosts it does not know; for its own configured
host it creates a fresh empty community instead. Breaking the host map
to force a 404 produced a second community and a working connection
into an empty workspace, which is a far more confusing failure than
the 404 it replaced.

Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
@artemtrofymenko
docs: catch finding 10 in the act, at the SQL level
16a2748 
Turned on statement logging, opened the Workflows screen, watched it
say "No workflows yet", and read back the query the relay issued. The
desktop asked for four channels; the SQL carries `channel_id = $2`
with a single value next to the four-value IN list, and $2 is the
smallest UUID — the one channel that has no workflow in it.

The chain now runs from the click to the row filter with nothing
inferred in between, which is the difference between confirming
someone else's diagnosis and having one.

Signed-off-by: Artem Trofymenko <99894081+artemtrofymenko@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@artemtrofymenko