decisions-v1: add document records and rerun the coverage critique

[MaggieAppleton]

Note

Holdout check passed (2026-10-19), including the 41 document records. Merge after #669.

Since review: documents/validate.py passes its allowed values to the shared checks instead of patching them, and reuses the shared leak-field list.

What

This PR is stacked on #669. It folds an earlier cloud session's work into the decisions-v1 corpus.

• documents/decisions.jsonl: 41 decision records taken from complete files.
  • Sources: TC39 notes (15); ADRs, KEPs and Node TSC minutes (16); PEPs (6); Rust RFC and lang-team texts (4).
  • These are the v0 records with fidelity.method == "git-file", excluding chopin-v2. Webfetch and mixed records were dropped.
  • They're converted to the Add decisions-v1 corpus of real GitHub decisions #669 record format:
    • caseId: null, collector: "cloud-git".
    • Flows lose their F1 -style prefixes.
    • Deciders map onto the enum, with the original wording kept in notes.
    • reactionsTotal: null and decidedOffPlatform are added.
    • Each gap becomes cluster: detail.
• documents/validate.py. It reuses the main validator's checks, allows only those values, and runs a verbatim-leak check against the 72 refetched source documents. Two v0 phrases leaked and are reworded in convert_v0.py.
• analysis/.
  • The cloud session's two critiques, unchanged apart from a provenance note.
  • coverage-critique.md, which reruns H1–H12 and re-ranks the gaps for all 173 records, threads only (132) and the small-team quota (11).

flowchart LR
  V0[cloud corpus v0<br/>82 records] -->|git-file, not chopin-v2| C[convert_v0.py<br/>41 records]
  C --> D[documents/decisions.jsonl]
  D --> F[fetch_docs.py<br/>72 source docs → .cache]
  F --> DV[documents/validate.py<br/>schema + 8-word leak check]
  T[decisions.jsonl<br/>132 thread records] --> TG[thread gaps coded into<br/>v0 tag vocabulary]
  TG --> R[blind recode of 60 doc gaps<br/>κ = 0.68]
  D --> A[analyse_corpus.py]
  TG --> A
  A --> M[corpus-metrics.json]
  M --> W[coverage-critique.md]

Loading

Results

Verdicts:

• Supported: H2, H5, H6, H7, H9, H10.
• Refuted: H1, narrowly (mixed or none 9% [6–14]), and H3 on options.
• Inconclusive: H8 in substance, and H11.
• Untestable: H4.
• Partly supported: H12. The votes half holds; the single-decider half is supported only through the small-team proxy (82%).

Top gaps (share of all records):

Gap	Share
Composite answer	61%
Decided elsewhere	53%
Follow-ups	50%
Coupled or blocking	38%
Supersede or revert	32%

In the small-team stratum the order changes: composite answer 73%, then follow-ups 46% and supersede or revert 46%, then parked 36%. Delegation, consensus mechanics and votes are 0% there.

Measurement notes:

• Thread gaps were coded into v0's tag vocabulary so both sources count alike. A blind recode of document gaps gives κ = 0.68.
• Consequences, rejected-content and convention reflect document genre, not coder drift.
• The small-team stratum is solo maintainers with many users (median 17 participants), not small groups.

Checks

• python3 -I evals/datasets/decisions-v1/validate.py: 127 cases, 132 decisions, 0 problems
• python3 -I evals/datasets/decisions-v1/documents/validate.py: 41 records, 0 problems, including the leak check
• bun test evals/datasets: pass
• bun run ci: pass
• bun run types: pass

Thread records, case files and the manifest are unchanged from #669.

🤖 Generated with Claude Code