You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
with GPS_CONSENSUS_CHECK_PROMPT asking for exactly YES or NO.
Two consequences:
Debate is all-or-nothing. One disagreement on a minor point triggers a full round in which every model re-argues everything, including the 90% they already agreed on. With 5 responders that is 5 full-context calls to settle one contested claim.
The signal is thrown away. The judge has just read every opinion and identified where they diverge, and we discard all of that to keep one bit.
There is also a robustness edge: any judge response that doesn't start with YES counts as disagreement, so a model that replies "Yes, they agree" is fine but one that replies "They agree" triggers a debate round.
Proposed direction
Have the consensus judge return structured output — agreed claims and contested claims — and use it to scope the debate:
Only the contested claims go into the debate prompt, so models argue the actual disagreement instead of restating their position.
The agreed claims can be passed to synthesis directly.
The UI could show what the models disagree about, which is arguably the most interesting thing llmgps knows and currently never surfaces.
Open questions to settle in this issue first
Output format and how to parse it robustly. Not every supported provider does reliable JSON mode, and generateSearchQueries already shows the regex-extract-then-JSON.parse fallback pattern used elsewhere in this file.
Fallback behaviour when parsing fails. Today a thrown consensus check is conservatively treated as "no consensus"; the structured version needs an equivalent that doesn't silently skip debate.
Whether this adds a new stream event (for example consensusDetail) or extends progress. A new event type means updating both src/lib/gps.ts and src/components/llmgps-shell.tsx together — see AGENTS.md invariant 4.
Problem
Consensus in llmgps is a single boolean derived from a string match.
checkConsensusinsrc/lib/gps.ts:with
GPS_CONSENSUS_CHECK_PROMPTasking for exactlyYESorNO.Two consequences:
There is also a robustness edge: any judge response that doesn't start with
YEScounts as disagreement, so a model that replies "Yes, they agree" is fine but one that replies "They agree" triggers a debate round.Proposed direction
Have the consensus judge return structured output — agreed claims and contested claims — and use it to scope the debate:
Open questions to settle in this issue first
generateSearchQueriesalready shows the regex-extract-then-JSON.parsefallback pattern used elsewhere in this file.consensusDetail) or extendsprogress. A new event type means updating bothsrc/lib/gps.tsandsrc/components/llmgps-shell.tsxtogether — see AGENTS.md invariant 4.Relevant files
src/lib/gps.ts(checkConsensus, debate loop,GpsStreamEvent)src/lib/llm.ts(GPS_CONSENSUS_CHECK_PROMPT)src/components/llmgps-shell.tsx(stream consumption, UI)See
CONTRIBUTING.mdfor setup and PR expectations.