circumvention-corpus

A structured, LLM-callable corpus of censorship-circumvention research.

This is a research index, not a paper hosting site. Each entry is a YAML record with the paper's core metadata, controlled-vocabulary tags (censor / detection technique / defense), team notes, and (optionally) extracted findings. The whole thing is exposed through an MCP server so an LLM can answer questions like "what did anyone find about active probing in 2024?" or "find every paper that evaluates a defense against the GFW's fully-encrypted-traffic detector" without anyone re-reading the field.

Why this exists

The field already has wonderful resources: net4people/bbs is the discussion forum, gfw.report publishes original research, CensorBib is a maintained bibliography, OONI runs network measurement. None of them are LLM-callable, none of them have a consistent structured metadata schema, and none of them let a Claude/GPT-shaped agent compose a corpus query with an internal-data query in the same conversation.

This corpus adds that one missing layer: a controlled-vocabulary metadata schema over the field's existing literature, with a tiny MCP server that lets any AI assistant query it.

How it's organized

schema/              JSON Schemas + the YAML taxonomy. The durable artifact.
  paper.schema.json
  finding.schema.json
  taxonomy.yaml      Controlled vocabulary: censors, techniques, defenses.
corpus/
  papers/            One YAML per paper.
  findings/          (Optional) extracted findings, one YAML per paper.
  pdfs/              (Optional) local PDF cache.
cmd/corpus-mcp/      Local stdio MCP server (Go) — offline / privacy fallback.
cmd/corpus-bundle/   Bundles YAMLs into JSON for the Worker to inline.
cmd/corpus-site/     Static site generator (Go html/template).
functions/mcp/       Hosted MCP server (Cloudflare Pages Function, TS).

Visibility model

Every paper has a visibility field. The schema has four levels:

Level	What it means	Where it can be served
public	Published openly, fine to share with anyone	Public MCP endpoint + local
community	Shared in confidence among circumvention-tool developers	Local only
internal	Lantern team only — partner draft, NDA'd doc, leaked material	Local only
embargoed	Will become public on a known date (embargo_until required)	Local only until the date

The repo split:

• circumvention-corpus (this repo) — public. Holds only visibility: public papers.
• circumvention-corpus-private (separate, private) — Lantern team only. Holds the community, internal, and embargoed papers, with the same schema.

The MCP server takes one or more corpus directories. The local Lantern-team Claude Code instance points at both; the public-facing endpoint runs with --public-only and reads only this repo. That's a hard data-path separation, not a runtime filter — an authorization bug in the public endpoint can't accidentally serve a private record because the private records aren't on disk.

When a private paper uses an LLM-derived finding, the redistribution constraints are propagated: the redistribution_terms field is included in get_paper results so the calling agent knows what it can and can't do with the citation.

Browsable site

A static site rendered from the same YAMLs lives at corpus.lantern.io, auto-deployed from main via the workflow in .github/workflows/deploy.yaml. Build it locally:

make site            # renders to ./dist/
python3 -m http.server -d dist 8080

The site uses cmd/corpus-site/, which reuses the same YAML loading the MCP server uses. There's no JS framework, no node_modules, just Go's html/template. About 0.1 seconds to render the whole corpus.

Using the MCP server

Hosted (recommended)

The corpus runs as a Cloudflare Pages Function at https://corpus.lantern.io/mcp. Zero install, no toolchain, no version drift — auto-deploys on every push to main so connected clients always see the latest committed state.

claude mcp add --transport http -s user circumvention-corpus \
  https://corpus.lantern.io/mcp

Or in any MCP client's config:

{
  "mcpServers": {
    "circumvention-corpus": {
      "url": "https://corpus.lantern.io/mcp",
      "transport": "http"
    }
  }
}

Self-hosted (offline / privacy)

For users behind censorship that blocks Cloudflare, or anyone who'd rather not send queries off-machine. The Go stdio server in cmd/corpus-mcp/ reads YAMLs directly from a local clone:

go install github.com/getlantern/circumvention-corpus/cmd/corpus-mcp@latest
git clone https://github.com/getlantern/circumvention-corpus ~/code/circumvention-corpus

claude mcp add -s user circumvention-corpus \
  $(go env GOPATH)/bin/corpus-mcp -- --corpus $HOME/code/circumvention-corpus

For a local team instance that also reads the private companion repo, point at a parent directory containing both, or run two server instances and let the agent compose results. Use --public-only to mirror the visibility clamp the hosted endpoint applies.

Architecture

The hosted MCP is a Cloudflare Pages Function (functions/mcp/index.ts) implementing MCP's Streamable HTTP transport directly — POST JSON-RPC, get JSON back. The corpus is bundled into the Worker at deploy time by cmd/corpus-bundle/ (Go), which reads every YAML and emits functions/_data/corpus.json; esbuild inlines that JSON into the Worker bundle, so there's no R2 read or runtime fetch.

Push to main → CI runs the bundler → Wrangler deploys the static site

• the Function together → live in ~60 s for everyone.

Tools the MCP exposes

• search_papers — keyword + tag-filter search. Returns ranked records.
• get_paper — fetch a single record by id.
• list_taxonomy — return the controlled vocabulary so the agent knows what tag IDs to filter on.
• find_related — papers that share censor / technique / defense tags.

Adding a paper

1. Pick a stable id: YYYY-firstauthor-shortslug (lowercase, dashes).
2. Create corpus/papers/<id>.yaml from the schema.
3. Tag against the controlled vocabulary in schema/taxonomy.yaml. If you need a tag that doesn't exist, add it to the taxonomy in the same PR.
4. Set visibility honestly. If unsure, default to non-public; promoting later is easy, recalling a leak isn't.
5. Write the notes field. This is where the team's actual judgment lives — the abstract is what the authors said, the notes are what we think about it.

For private papers: open the PR against circumvention-corpus-private instead. Same schema, same review process.

Curation philosophy

Curation beats volume. A hand-selected 200-paper corpus the team trusts is more useful than a 50,000-paper auto-crawl. We do run crawlers (arXiv, USENIX, NDSS, FOCI proceedings, net4people watcher), but they propose additions to a triage queue rather than auto-ingesting. A human always decides what enters the corpus.

The bar is roughly: would I expect a Lantern protocol designer to be slowed down if they hadn't read this? If yes, it's a candidate. "Comprehensive coverage of the field" is a non-goal.

Contributing

Most welcome:

• New papers (especially recent measurement / threat-model work)
• Better tags on existing papers
• Findings extracted from papers (one-to-three-sentence claims with paper id + section reference)
• Taxonomy additions when the existing terms don't fit

Issues are open. PRs are open. The maintainers (Lantern team for now) will get involved as needed; ideally the schema is uncontroversial enough that most PRs are routine merges.

License

Schema, taxonomy, and corpus metadata: CC0 / public domain. Paper PDFs (when stored locally): NOT redistributed. Each paper YAML points at the canonical URL; downloads are the user's responsibility.

Discovery coverage and historical backfills

The default --source all crawl includes Crossref title searches for encrypted traffic classification, application identification, VPN fingerprinting, and TLS traffic analysis. This adds publisher metadata (including IEEE papers) beyond the conference pages and arXiv cs.CR / cs.NI feeds. Candidates still pass the keyword filter, deduplication, classifier, and existing PR ingestion policy. Sources are interleaved before the classification budget is applied, and curated sources (net4people, gfw-report, FOCI, …) fill that budget first: Crossref matches hundreds of thousands of works per query, and a curated candidate pushed past --max-classify is neither ingested nor cached as a rejection, so next week's narrower window loses it for good.

The window is applied as Crossref's indexing date, not publication date. Crossref reads a date-parts of [[2026,9]] as 2026-09-01 and [[2026]] as 2026-01-01, so a publication-date floor a few days back silently excludes every month- or year-granularity record — most conference proceedings, i.e. exactly the IEEE/ACM metadata this source exists to reach. Indexing date asks "what did Crossref learn about during the window" instead, at the cost of re-seeing already-indexed older work each run; dedup and the rejection cache absorb that.

Crossref requests carry a mailto: in the User-Agent so they run in Crossref's polite pool, are paced, and retry on HTTP 429 honoring Retry-After. A query that stays throttled is logged and skipped — the remaining queries still run, because each reaches a different slice of the literature.

A recent-paper crawl cannot recover all foundational work. The installed weekly job uses a 10-day window. Run a separate bounded historical discovery pass:

go run ./cmd/corpus-crawl run --source crossref --window-days 3650 \
  --dry-run --max 400 --max-classify 400

Review the candidate list before running without --dry-run; the normal run uses the existing PR/auto-merge policy. For example, --max 10 --max-classify 80 bounds a real batch. Dry runs do not write papers or call the classifier. Crossref returns relevance-ranked, partial matches. Each query is limited to 200 results over two pages, with truncation logged. Most Crossref works carry no abstract, so those candidates are classified from the title and venue alone; the classifier is told to skip rather than guess when a title will not support a taxonomy tag. This is targeted discovery, not an exhaustive archive scan; widening the time window alone does not remove that limit. Source HTTP failures and classifier-budget truncation are also logged and must not be interpreted as an empty research field.

Keep a historical discovery pass alongside recent crawls when adding a research area. Regression fixtures cover both FlowPic titles, publisher pagination and metadata, rate-limit retry and skip-to-next-query behavior, and competition between sources for the classifier budget. Passing those tests protects these specific discovery paths, not completeness of the literature.