diff --git a/docs/70-79-ai-intelligence/70.13-retrieval-benchmark-reference.md b/docs/70-79-ai-intelligence/70.13-retrieval-benchmark-reference.md new file mode 100644 index 000000000..be19a6084 --- /dev/null +++ b/docs/70-79-ai-intelligence/70.13-retrieval-benchmark-reference.md @@ -0,0 +1,102 @@ +# 70.13 Retrieval benchmark: five known-answer questions + +A fixed question set for comparing knowledge stores (ruflo memory, OKF-native +progressive disclosure, qmd, OpenViking). Fixed is the point: the 2026-08-22 +run scored four stores and then **nobody wrote the questions down**, so the +numbers in `.okf/log.md` cannot be reproduced or trusted for comparison. This +file is the fix. Change the questions and you start a new series - say so, and +re-run every store. + +## Method + +Each question has exactly one canonical home concept in `.okf/`. A store scores +a hit when its **top result points at that concept**. Partial credit is not +given: "it returned something related" is how a store looks good while being +useless at the moment you need it. + +Run every store in the same session. A number from a store measured on a +different day, against a different bundle state, is not comparable - the +2026-08-22 OpenViking result was scored against a store silently missing the +entire `build/` section. + +## The five questions + +| # | Question | Canonical home | Verified answer | +|---|---|---|---| +| 1 | What screenshot tolerance is allowed for a refactor, versus a new feature? | `build/test-gates.md` | 0.0 for refactors (a refactor must move zero pixels); <=0.03 for new features | +| 2 | What founding date may we publish, and how must the Clutch rating be cited? | `content/claims-canon.md` | Founding 2008-09-01; "4.8/5 on Clutch" with the profile linked, and no review count | +| 3 | How do we detect AI slop in a draft? | `content/voice-rules.md` | Cold-eyes four-lens panel of different agent types than the writer - never a score or a grep; each reviewer must name something it would cut | +| 4 | When purging fabricated claims, what do we rank by, and what number proved `featured` was the wrong signal? | `content/fabrication-ratchet.md` | Rank by live GSC impressions; the `featured: true` post had 4 impressions in 90 days against 40,025 for the top carrier | +| 5 | What font does mermaid render in, and what had to change to make it load? | `design/mermaid-theme.md` | Caveat handwritten cursive; it was never loaded before - now an explicit Google Fonts link gated on `features.mermaid`, awaited via `document.fonts.ready` | + +Answers verified against the source files on 2026-08-22, at +`build/test-gates.md:86`, `content/claims-canon.md:27,30`, +`content/fabrication-ratchet.md:85-90`, `design/mermaid-theme.md:17,22`. + +## Two question types, and why the old 5/5 was flattering + +A routing question ("which concept governs screenshot tolerance?") and a fact +question ("what IS the tolerance?") measure different things, and mixing them +produces a number that means nothing. + +The ruflo `okf` namespace holds 41 entries, one per concept, each ~130-400 +bytes of frontmatter title and description. `okf-build-test-gates` reads in +full: *"Playbook: bin/qtest --changed is the routine gate; bin/rake +test:critical at milestones; bin/test AND bin/dtest once at PR prep for +themes/layouts/CSS changes. Details: .okf/build/test-gates.md"*. It ends by +naming the file, because **it is a routing pointer by design**. Ask it where +tolerance is governed and it hits at 0.365. Ask it what the tolerance is and it +returns `okf-wf-css-plan`, the wrong concept. Question 2 returns nothing at all +- no entry clears the 0.3 similarity floor. + +So the recorded "ruflo 5/5, OKF-native 5/5" measured **routing**. Both stores +hold distilled pointers and cannot answer a fact question, by construction. +That is not a defect - routing is what they are for - but it is not the same +achievement the log implied, and it is not comparable to a full-content store. + +The table above is the **fact** suite. Keep the two suites separate and label +every score with which one it is. + +## Results + +### 2026-08-22 - OpenViking: UNMEASURABLE, and that is the finding + +The re-run was blocked, not by a missing feature, but by throughput: + +| Signal | Measurement | +|---|---| +| Semantic-Nodes queue | 1,784 pending, 92 processed | +| Measured throughput | 3 nodes / 5 min (two samples, 300s apart) | +| ETA to drain | ~49 hours | +| `ov find` on any benchmark question | request timeout, repeatedly | +| Lifetime retrieval stats | 295 queries, **87.8% zero-result rate** | +| Errors | 0 - nothing is broken, it is saturated | + +The query path needs the embedding model that the ingest queue is already +saturating, so a `find` cannot complete while a backlog exists. A store that +cannot answer while it is indexing has an availability property worth naming +before adoption, separate from how good its answers are. + +The 87.8% zero-result rate over 295 lifetime queries is an independent signal +and does not depend on this backlog. + +ruflo `okf` on the fact suite: 0/2 attempted before the run was abandoned as +non-comparable (Q1 wrong concept, Q2 empty). Re-run all stores together on both +suites once the backlog clears. + +### 2026-08-22 earlier run (SUPERSEDED - do not cite) + +ruflo 5/5 · OKF-native 5/5 · qmd 1/5 · OpenViking "2 partial, 3 unanswerable". +Questions were never recorded, the two 5/5 scores were routing questions scored +as if they were fact questions, and OpenViking was measured against a store +silently missing the entire `build/` section. Every number in that line is +uncomparable to anything. + +## Known trap + +`ov ls | grep -c 'viking://'` returns 1 for an **empty** directory - it +counts the `cmd: ov ls ...` echo the CLI prints before results. A count is not a +check. Confirm content with `ov grep` on a distinctive string, and never judge +presence by directory name: OpenViking's directory ingest names directories from +content **headings**, not frontmatter titles, so `claims-canon.md` is filed as +`Product_canon_-_Evidence_on_Hand`.