Skip to content

Structured claim-level consensus instead of a single YES/NO #7

Description

@leodenglovescode

Design discussion wanted before implementation — please comment before opening a PR.

Problem

Consensus in llmgps is a single boolean derived from a string match. checkConsensus in src/lib/gps.ts:

return result.trim().toUpperCase().startsWith("YES");

with GPS_CONSENSUS_CHECK_PROMPT asking for exactly YES or NO.

Two consequences:

  1. Debate is all-or-nothing. One disagreement on a minor point triggers a full round in which every model re-argues everything, including the 90% they already agreed on. With 5 responders that is 5 full-context calls to settle one contested claim.
  2. The signal is thrown away. The judge has just read every opinion and identified where they diverge, and we discard all of that to keep one bit.

There is also a robustness edge: any judge response that doesn't start with YES counts as disagreement, so a model that replies "Yes, they agree" is fine but one that replies "They agree" triggers a debate round.

Proposed direction

Have the consensus judge return structured output — agreed claims and contested claims — and use it to scope the debate:

  • Only the contested claims go into the debate prompt, so models argue the actual disagreement instead of restating their position.
  • The agreed claims can be passed to synthesis directly.
  • The UI could show what the models disagree about, which is arguably the most interesting thing llmgps knows and currently never surfaces.

Open questions to settle in this issue first

  • Output format and how to parse it robustly. Not every supported provider does reliable JSON mode, and generateSearchQueries already shows the regex-extract-then-JSON.parse fallback pattern used elsewhere in this file.
  • Fallback behaviour when parsing fails. Today a thrown consensus check is conservatively treated as "no consensus"; the structured version needs an equivalent that doesn't silently skip debate.
  • Whether this adds a new stream event (for example consensusDetail) or extends progress. A new event type means updating both src/lib/gps.ts and src/components/llmgps-shell.tsx together — see AGENTS.md invariant 4.
  • Interaction with Blind model identities during consensus checks and debate rounds #6 (blinded identities): claims should be attributed to handles, not model names.

Relevant files

  • src/lib/gps.ts (checkConsensus, debate loop, GpsStreamEvent)
  • src/lib/llm.ts (GPS_CONSENSUS_CHECK_PROMPT)
  • src/components/llmgps-shell.tsx (stream consumption, UI)

See CONTRIBUTING.md for setup and PR expectations.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions