Retrieval benchmark: record the questions, correct two scores

[pftg]

Retrieval benchmark: write the questions down, and two corrections

The 2026-08-22 benchmark scored four knowledge stores and nobody wrote the
questions down. .okf/log.md says "queries in the benchmark entry" - they are
not there. So the numbers cannot be reproduced, which means they cannot be
compared to anything, which means they were not a measurement.

Five known-answer questions now live in 70.13, each with one canonical home
concept and its answer verified against the source line (build/test-gates.md:86,
claims-canon.md:27,30, fabrication-ratchet.md:85-90, mermaid-theme.md:17,22).

Correction 1: the two 5/5 scores were routing questions. The ruflo okf
namespace is 41 entries of frontmatter title+description, 130-400 bytes each.
okf-build-test-gates reads in full: "Playbook: bin/qtest --changed is the
routine gate; ... Details: .okf/build/test-gates.md". It ends by naming the
file because it is a pointer by design. Ask it which concept governs tolerance
and it hits at 0.365; ask it what the tolerance IS and it returns the wrong
concept, and a second fact question returns nothing above the 0.3 floor.
Routing and fact are different suites and mixing them flatters the distilled
stores. Both suites are now defined and every score must say which it is.

Correction 2: OpenViking is unmeasurable right now, and that is the finding.
Semantic-Nodes queue at 1,784 pending / 92 processed, measured at 3 nodes per
5 minutes across two samples 300s apart - about 49 hours to drain. ov find
times out on every question, because the query path needs the same embedding
model the ingest queue is saturating. Zero errors; it is not broken, it is
saturated. A store that cannot answer while it is indexing has an availability
property worth naming before adoption, separate from answer quality. Its
lifetime stats, which do not depend on this backlog: 295 queries, 87.8%
zero-result rate.

The old OV number was also scored against a store silently missing the entire
build/ section, backfilled earlier today.

Also recorded: ov ls <uri> | grep -c 'viking://' returns 1 for an EMPTY
directory - it counts the CLI's own cmd: echo. That nearly caused a wrong
delete.

Docs-only. No gates apply.

Co-Authored-By: Claude Opus 5 noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg

🤖 Generated with Claude Code

https://claude.ai/code/session_016PUkwFTsiv7EB2DYKogbpg

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 91893f0b-8e48-4d8d-bcbc-312aca801fe9

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}