Retrieval benchmark: record the questions, correct two scores

docs/70-79-ai-intelligence/70.13-retrieval-benchmark-reference.md

@@ -0,0 +1,102 @@
+# 70.13 Retrieval benchmark: five known-answer questions
+
+A fixed question set for comparing knowledge stores (ruflo memory, OKF-native
+progressive disclosure, qmd, OpenViking). Fixed is the point: the 2026-08-22
+run scored four stores and then **nobody wrote the questions down**, so the
+numbers in `.okf/log.md` cannot be reproduced or trusted for comparison. This
+file is the fix. Change the questions and you start a new series - say so, and
+re-run every store.
+
+## Method
+
+Each question has exactly one canonical home concept in `.okf/`. A store scores
+a hit when its **top result points at that concept**. Partial credit is not
+given: "it returned something related" is how a store looks good while being
+useless at the moment you need it.
+
+Run every store in the same session. A number from a store measured on a
+different day, against a different bundle state, is not comparable - the
+2026-08-22 OpenViking result was scored against a store silently missing the
+entire `build/` section.
+
+## The five questions
+
+| # | Question | Canonical home | Verified answer |
+|---|---|---|---|
+| 1 | What screenshot tolerance is allowed for a refactor, versus a new feature? | `build/test-gates.md` | 0.0 for refactors (a refactor must move zero pixels); <=0.03 for new features |
+| 2 | What founding date may we publish, and how must the Clutch rating be cited? | `content/claims-canon.md` | Founding 2008-09-01; "4.8/5 on Clutch" with the profile linked, and no review count |
+| 3 | How do we detect AI slop in a draft? | `content/voice-rules.md` | Cold-eyes four-lens panel of different agent types than the writer - never a score or a grep; each reviewer must name something it would cut |
+| 4 | When purging fabricated claims, what do we rank by, and what number proved `featured` was the wrong signal? | `content/fabrication-ratchet.md` | Rank by live GSC impressions; the `featured: true` post had 4 impressions in 90 days against 40,025 for the top carrier |
+| 5 | What font does mermaid render in, and what had to change to make it load? | `design/mermaid-theme.md` | Caveat handwritten cursive; it was never loaded before - now an explicit Google Fonts link gated on `features.mermaid`, awaited via `document.fonts.ready` |
+
+Answers verified against the source files on 2026-08-22, at
+`build/test-gates.md:86`, `content/claims-canon.md:27,30`,
+`content/fabrication-ratchet.md:85-90`, `design/mermaid-theme.md:17,22`.
+
+## Two question types, and why the old 5/5 was flattering
+
+A routing question ("which concept governs screenshot tolerance?") and a fact
+question ("what IS the tolerance?") measure different things, and mixing them
+produces a number that means nothing.
+
+The ruflo `okf` namespace holds 41 entries, one per concept, each ~130-400
+bytes of frontmatter title and description. `okf-build-test-gates` reads in
+full: *"Playbook: bin/qtest --changed is the routine gate; bin/rake
+test:critical at milestones; bin/test AND bin/dtest once at PR prep for
+themes/layouts/CSS changes. Details: .okf/build/test-gates.md"*. It ends by
+naming the file, because **it is a routing pointer by design**. Ask it where
+tolerance is governed and it hits at 0.365. Ask it what the tolerance is and it
+returns `okf-wf-css-plan`, the wrong concept. Question 2 returns nothing at all
+- no entry clears the 0.3 similarity floor.
+
+So the recorded "ruflo 5/5, OKF-native 5/5" measured **routing**. Both stores
+hold distilled pointers and cannot answer a fact question, by construction.
+That is not a defect - routing is what they are for - but it is not the same
+achievement the log implied, and it is not comparable to a full-content store.
+
+The table above is the **fact** suite. Keep the two suites separate and label
+every score with which one it is.
+
+## Results
+
+### 2026-08-22 - OpenViking: UNMEASURABLE, and that is the finding
+
+The re-run was blocked, not by a missing feature, but by throughput:
+
+| Signal | Measurement |
+|---|---|
+| Semantic-Nodes queue | 1,784 pending, 92 processed |
+| Measured throughput | 3 nodes / 5 min (two samples, 300s apart) |
+| ETA to drain | ~49 hours |
+| `ov find` on any benchmark question | request timeout, repeatedly |
+| Lifetime retrieval stats | 295 queries, **87.8% zero-result rate** |
+| Errors | 0 - nothing is broken, it is saturated |
+
+The query path needs the embedding model that the ingest queue is already
+saturating, so a `find` cannot complete while a backlog exists. A store that
+cannot answer while it is indexing has an availability property worth naming
+before adoption, separate from how good its answers are.
+
+The 87.8% zero-result rate over 295 lifetime queries is an independent signal
+and does not depend on this backlog.
+
+ruflo `okf` on the fact suite: 0/2 attempted before the run was abandoned as
+non-comparable (Q1 wrong concept, Q2 empty). Re-run all stores together on both
+suites once the backlog clears.
+
+### 2026-08-22 earlier run (SUPERSEDED - do not cite)
+
+ruflo 5/5 · OKF-native 5/5 · qmd 1/5 · OpenViking "2 partial, 3 unanswerable".
+Questions were never recorded, the two 5/5 scores were routing questions scored
+as if they were fact questions, and OpenViking was measured against a store
+silently missing the entire `build/` section. Every number in that line is
+uncomparable to anything.
+
+## Known trap
+
+`ov ls <uri> | grep -c 'viking://'` returns 1 for an **empty** directory - it
+counts the `cmd: ov ls ...` echo the CLI prints before results. A count is not a
+check. Confirm content with `ov grep` on a distinctive string, and never judge
+presence by directory name: OpenViking's directory ingest names directories from
+content **headings**, not frontmatter titles, so `claims-canon.md` is filed as
+`Product_canon_-_Evidence_on_Hand`.