llmgps

中文文档

llmgps is a self-hosted multi-model reasoning workspace. It gives several LLMs the same question and the same retrieved evidence, collects their independent answers, optionally lets them debate the points they disagree on, and synthesizes one final response.

It started from a practical annoyance: Claude, ChatGPT, DeepSeek and friends each have real strengths, but using them in separate tabs means duplicated context, inconsistent research, and manually reconciling three different answers yourself. llmgps does that reconciliation in one place, on your own machine, with your own keys.

Why this is not just "chat with multiple models"

• Shared evidence. Web retrieval happens once, centrally (Brave or Tavily), and the same evidence block is handed to every responder — so models disagree about reasoning, not about which search results they happened to get.
• Independent opinions first. Responders never see each other's answers in the first round. Cross-contamination is deliberately delayed until the debate phase.
• Deliberation is gated, not unconditional. A synthesizer model is asked whether the responders already broadly agree. Debate rounds only run when they don't.
• Failures degrade instead of aborting. A responder that errors is recorded as a failure and the run continues on the survivors. Search failures, compression failures and memory-relevance failures are all non-fatal.
• Expensive work is not thrown away. If only the final synthesis fails, you get the raw model opinions back and can retry synthesis alone — without re-running the fan-out or the debate.
• Local-first configuration. Provider keys, settings and chat history live in one AES-256-GCM-encrypted row in a local SQLite file. Nothing is sent to a llmgps-operated service, because there isn't one.

llmgps does not claim to have invented multi-agent debate, LLM councils, or mixture-of-agents. It is a practical, self-hostable implementation of those ideas that you can run and modify.

---

Looking for contributors

This is a small project with a lot of surface area, and it is actively looking for collaborators. Concretely, these are the areas where help would matter most right now:

Area	What it looks like in this repo
LLM orchestration	The debate loop, consensus gating and round handling in src/lib/gps.ts
Provider integrations	New providers and request shapes in src/lib/llm.ts + src/lib/gps.ts
Inference cost reduction	Prompt caching, avoiding duplicate calls, cheaper judge/consensus paths
Retrieval / evidence grounding	Query generation, result ranking, per-round evidence refresh
Evaluation & benchmarking	There is currently no harness for comparing Single vs GPS vs Debate
Debate & consensus algorithms	Today consensus is a single YES/NO judgement — claim-level structure would be better
Frontend UX	src/components/llmgps-shell.tsx is one large client component that needs decomposition
Local-model support	Ollama path, model discovery, timeouts, proxy bypass behaviour
Reliability & testing	There is no test framework configured yet — that is an open, high-value slot

Start with CONTRIBUTING.md for setup, architecture orientation and the review expectations, and docs/ARCHITECTURE.md if you want the deeper technical picture first. If you use a coding agent, point it at AGENTS.md.

Small, focused pull requests are very welcome. So are issues that just describe a problem you hit.

---

The three modes

Mode	Models used	What happens
Single	1 responder	One model answers directly. No synthesis step. This is ordinary chat.
GPS	Up to 5 responders + 1 synthesizer	All responders answer the same prompt in parallel and independently. The synthesizer merges their answers into one response, noting where they disagreed.
Debate	Up to 5 responders + 1 synthesizer	GPS, plus a consensus check. If the responders do not broadly agree, they each read the others' answers and argue point-by-point. Up to 2 rounds, then synthesis.

Debate mode is the expensive one: with 5 responders and 2 rounds you are paying for up to 15 model calls plus consensus checks and synthesis. GPS mode is the common case.

Debate loop in detail

1. Fan-out. All responders receive the same prompt (plus shared evidence, if search is on) and answer independently, in parallel.
2. Consensus check. The synthesizer model is shown all opinions and answers a single YES/NO: do these broadly agree on the core answer? If the check itself fails, llmgps conservatively assumes no consensus.
3. Debate round. On NO, every model is shown the other models' answers and asked to address specific disagreements point-by-point.
4. Evidence refresh. Before round 2 — and only if web search is enabled and configured — retrieval runs again so the second round argues over fresher evidence. A failed refresh is non-fatal; the round continues with the previous evidence.
5. Synthesis. After consensus is reached or 2 rounds are exhausted, the synthesizer produces the final answer. Optionally, the debate is compressed into a structured summary first to keep the synthesis prompt small.

Note on wording: llmgps produces a web-grounded, multi-model synthesized answer. Retrieved search snippets are evidence supplied to the models, not verification — the final answer is not fact-checked by the system.

Shared retrieval architecture

When web search is enabled, retrieval happens once per round and is shared by every responder:

user question
  → search-query model generates up to 5 focused queries   (optional; falls back to the raw question)
  → Brave or Tavily is queried per query
  → results deduplicated by URL, formatted into one dated evidence block
  → the block is prepended to the conversation as a user/assistant pair
  → every responder model receives the identical evidence
  → consensus check → debate rounds (with optional evidence refresh)
  → synthesis

---

Screenshots

---

Features

• Three reasoning modes: Single, GPS and Debate, switchable per message.
• 14 provider integrations: OpenAI, Anthropic, Gemini, OpenRouter, DeepSeek, Kimi, Qwen, Mistral, Zhipu, xAI, any OpenAI-compatible custom endpoint, Ollama, plus Claude Code and Codex via your existing subscription — behind five request shapes (openai-compatible, anthropic, gemini, ollama, agent-service).
• Subscription providers, no API key: use a Claude Pro/Max or ChatGPT Plus plan you already pay for. See Subscription providers.
• You choose the models: model IDs are entered per provider in Settings rather than picked from a hard-coded list, so new model releases work without waiting for a llmgps update.
• Live streaming: token-level output and reasoning/"thinking" traces stream from every responder at once (OpenAI-compatible, Anthropic and Ollama stream incrementally; Gemini is buffered).
• Ollama integration: local models as first-class responders or synthesizers, with model discovery and an optional proxy bypass.
• Web search: Brave or Tavily, with an optional dedicated model for query generation.
• Context compression & rolling context: compress a long debate into a structured summary, and optionally carry that summary forward as the context for the next turn.
• User memories: up to 20 short notes, injected only when a relevance check says they are useful for the current question.
• Extended thinking: opt-in Anthropic extended thinking with a configurable token budget.
• Encrypted local storage: settings, provider keys and chat history in an AES-256-GCM encrypted payload inside SQLite.
• Proxy support: HTTP or SOCKS5 for provider and search traffic, with an Ollama bypass.
• Bilingual UI: English and Simplified Chinese.

---

Deployment & Setup

Docker (Recommended)

Pull and run:

docker pull ghcr.io/leodenglovescode/llmgps:main

docker run -d \
  -p 3000:3000 \
  -e LLMGPS_DATA_KEY=your-secret-key \
  -v llmgps-data:/data \
  --name llmgps \
  ghcr.io/leodenglovescode/llmgps:main

Access: http://localhost:3000

Set LLMGPS_DATA_KEY to any strong secret string. The /data volume persists your history and settings. Containerized runs require LLMGPS_DATA_KEY or LLMGPS_DATA_KEY_FILE — the app refuses to generate and persist a key next to the database in Docker.

Local Development

Requires Node.js 22 (the app uses the built-in node:sqlite module; CI and the Docker image both run Node 22).

1. Install: npm install
2. Run: npm run dev
3. Access: http://localhost:3000

On first run you create a single owner account (username + password) in the browser, then add provider API keys and model IDs in Settings.

Other scripts: npm run build, npm run start, npm run lint. Type-check with npx tsc --noEmit. There is no test script yet — see CONTRIBUTING.md.

Environment variables

See .env.example. All are optional in local development:

Variable	Purpose
LLMGPS_DATA_FILE	Path to the SQLite database. Defaults to .llmgps-data.sqlite in the project root, or /data/llmgps-data.sqlite in the container.
LLMGPS_DATA_KEY	Encryption secret, supplied directly.
LLMGPS_DATA_KEY_FILE	Path to a file containing the encryption secret (Docker secrets, mounted files).
LLMGPS_LOG_DIR	Directory for server logs. Defaults to ./logs, retained for 1 day.
NEXT_PUBLIC_APP_NAME	Display name in the UI.
NEXT_DEV_ALLOWED_ORIGINS	Comma-separated dev origins for Next.js. Defaults to localhost plus detected private IPv4 addresses.

Data & Security

llmgps stores all sensitive configuration in an encrypted payload inside SQLite.

• Encryption: AES-256-GCM, key derived with scrypt from LLMGPS_DATA_KEY.
• Key handling: in local development a .llmgps-data.key file is generated automatically (mode 0600). In Docker you must supply the key yourself.
• Persistence: mount /data to keep your history and settings across container restarts.
• Single owner: one account per deployment, session cookie signed with an HMAC secret that itself lives in the encrypted payload.

This protects data at rest against someone reading the database file without the key. It is not a substitute for not exposing the app to the public internet.

Subscription providers (no API key)

If you already pay for Claude Pro/Max or ChatGPT Plus, llmgps can use those plans as models — no API key, no per-token billing. It works by talking to the claude and codex CLIs you have already logged in with.

Those CLIs are coding agents that can run shell commands, and llmgps puts untrusted text (web search results) into every prompt. So they do not run inside llmgps. They run in a separate container that holds no database, no API keys and no encryption key — Claude Code additionally runs with every tool denied (verified to actually block command execution); Codex has no equivalent lockdown and should be assumed able to act, contained only by the fact that the container has nothing in it worth acting on besides its own subscription tokens. agent-service/README.md explains the threat model, why the two CLIs are not equally restricted, and how to verify the lockdown yourself.

# 1. Log in on the host, once
claude          # complete login, then exit
codex           # same

# 2. Start both containers
export LLMGPS_DATA_KEY=your-strong-secret
export LLMGPS_AGENT_SECRET=$(openssl rand -hex 32)
docker compose up -d

Then in Settings → Providers, enable Claude Code or Codex, set the service URL to http://llmgps-agents:8080, paste the same LLMGPS_AGENT_SECRET, and press Test connection. Add model ids (sonnet, opus, haiku, gpt-5.6-terra, …) like any other provider.

Worth knowing before you enable it:

• Claude Code and Codex are not equally locked down. Claude Code's tool lockdown was verified to actually stop it from running commands. Codex's could not be made equally strong — it has no single "remove all tools" switch — so treat it as able to run commands inside its own container, and rely on that container holding nothing worth reaching. See agent-service/README.md for what was tested and why.
• The agent container holds your subscription tokens in plaintext — that is what it is for. A break-in there costs subscription access, but not your API keys, chat history or database.
• Expect noticeably higher latency than a plain API call: each request starts a CLI process. They are poor choices for the cheap meta-roles (consensus checks, query generation).
• Check your plan's terms. These are interactive coding tools, and driving them from another application may not be what your subscription is sold for.

Ollama Support

1. Start Ollama.
2. In llmgps Settings, enable Ollama and set the base URL (e.g. http://127.0.0.1:11434).
3. Add the local model IDs you want and select them as responders or as the synthesizer.

If you route provider traffic through a proxy, enable bypass proxy for Ollama so local requests go direct.

---

Debate Mode Flow Chart

---

Documentation

• CONTRIBUTING.md — setup, architecture orientation, checklists, PR expectations
• docs/ARCHITECTURE.md — how the system actually works, file by file
• AGENTS.md — short entry point for coding agents

---

Credits & License

• Logic & Idea: @leodenglovescode
• Assistance: GitHub Copilot

Licensed under the MIT License.