saved_usd prices our own token count at the provider's rate, under-reporting savings by ~12% (measured)

[amiddavid]

request_components.saved_usd — and every figure derived from it — is computed by pricing our own message-text token count at the provider's rate, rather than the tokens the provider actually billed. On a measured live turn it under-reports by 12.4%.

This is the same root cause as two defects fixed in #234, at a third site. It predates that PR and affects every saved_usd consumer, so it gets its own issue.

The measured pair

From the acceptance run on #234 (full data) — a real Claude Code session on claude-haiku-4-5, where a cold turn arrived after the transcript had been summarized:

The full prefix that turn would have re-created	149,363 tokens
The compacted prefix it did re-create	32,123 tokens
Tokens the compaction removed, as the provider billed them	117,240
At haiku's 5-minute cache-write rate (1.25 × $1/MTok)	$0.14655
What saved_usd reported for the same removal	$0.12834875
Implied token count behind that figure	102,679
Under-report	12.42%

The savings pipeline counted 102,679 tokens where the provider billed 117,240 for the same removed content — a ratio of 0.876.

Why the gap exists

Event.baselineDeltaUSD prices SavedUnique and Saved, which come from schema.MessagesTokens: this repo's own tokenizer, over message text only. The provider bills the whole serialized prompt. dash/overview.go's EstimatorDivergence already measures the general form of this — a median 3.38x across all traffic — but that figure is dominated by small transcripts where the fixed overhead (system prompt, tool declarations, JSON envelope) is most of the request. On a large transcript, which is where compaction happens, the divergence narrows to roughly 1.1-1.5x, and 12.4% is what it comes to on this turn.

Why it is worth filing rather than leaving

The direction is the interesting part. It under-reports, so:

• Every savings figure this repo publishes is conservative, which is why it did not block feat(summarize): the pre-expiry summary gate — compact only when the write was already due, off the hot path #234's acceptance.
• But a fix moves every savings number in the product upward, which is the direction that needs the most justification. Doing it deliberately, with the arithmetic written down, is very different from someone later "correcting a tokenizer" and silently inflating the headline figure.

It is the third site of one root cause. #234 fixed the same class of error twice:

1. Trigger.Fires compared frac × window against MessagesTokens — a gate that could never fire, because a window is stated in billed tokens.
2. The compaction-episode span was sized in one quantity and evaluated against another, twice over.
3. This: saved_usd prices our token count at the provider's rate.

The first two were unreachable-gate and empty-measurement bugs. This one is a quiet 12% understatement, which is why it survived both.

Scope

Everything that reads saved_usd or baseline_cost_usd: the Components tab's per-component economics, NetSavedUSD/TotalSavedUSD, the savings waterfall, the compaction-episode panel's credit buckets, and /stats. So this is not a small change, and a fix wants:

• a decision on what to price — the provider's billed delta is available per request (fresh_input + cache_read + cache_write before and after is not, but the written and read columns are), so this may be answerable from stored data rather than by improving the tokenizer;
• the before/after figures published together, because the headline will move up;
• a test pinning the relationship on a known pair, so the next tokenizer change does not move it silently.

Not on #234's branch

#234's own two instances of this root cause are fixed there. This third one predates it and affects consumers that PR does not touch, so per the repo rule it gets its own issue and branch off main.

Found by the reviewing session on #234, which re-derived the headline from the raw token counts rather than reading the reported figure — which is how the 12.4% gap surfaced at all.

[OsherElhadad]

Review of this issue — the defect is real, the arithmetic is exact, but two of the three supporting arguments do not hold

Audited against origin/main @ e66f255, with a live paired token measurement on the IBM LiteLLM gateway (claude-haiku-4-5, ~$0.07 spend). Every claim below is marked CONFIRMED (verified against code, live data, or a test that ran) or REASONED (inference not independently verified).

1. The core claim and every step of its arithmetic: CONFIRMED

Step	Issue	Recomputed
149,363 − 32,123	117,240	117,240	✓
× 1.25 × $1/MTok	$0.14655	$0.14655	✓
$0.12834875 ÷ 1.25e-6	102,679	102,679.0	✓
1 − 102,679/117,240	12.42%	12.4198%	✓

The code path is exactly as described: dash/event.go:873 baselineDeltaUSD prices SavedUnique/Saved (from schema.MessagesTokens) at the provider's rate. Worth adding: that tokenizer is o200k_base, an OpenAI BPE, applied to Anthropic models (internal/tokens/tokens.go), and MessageText returns "" for non-text blocks (schema/schema.go:135-140), so images, audio, files and tool_result payloads count as zero.

Both endpoint figures are genuinely provider-billed (cache_creation_input_tokens), so "as the provider billed them" is legitimate. The 5-minute rate assumption is also safe — no 1h-tier rows are credited today.

2. "The direction is always conservative" is FALSE — this is the most important correction

internal/modelinfo/modelinfo.go:200 and table.go:118-119:

if p.CacheWrite == 0 { p.CacheWrite = p.Input * 1.25 }   // the Anthropic-family multiple

LiteLLM publishes no cache-write price for non-Anthropic families — verified by live fetch: gpt-4o, gpt-5, gemini-2.5-pro, gemini-2.5-flash all have cache_creation: None. So every OpenAI/Gemini row gets a fabricated 1.25× write rate on baselineDeltaUSD's unique term. Neither provider charges a write premium, so the correct counterfactual is 1.00× input:

reported 100_000 × 1.25 × Input vs truth 100_000 × Input → exactly +25%

CONFIRMED by a test pinning both signs on the real baselineDeltaUSD: 12.42% under on the measured Anthropic pair, 25.00% over on OpenAI rates. The replay term is unaffected (proxy/usage.go:228 never sets CacheWrite on the OpenAI dialect, so repeatRate correctly falls through to p.Input).

It also compounds on OpenAI: our tokenizer is o200k there, so the f > 1 undercount that partly masks this on Anthropic largely vanishes (f ≈ 1.00–1.06 for text).

Consequence for this issue's framing: the bias sign is model-dependent and unstated. The justification for deferring — "it under-reports, so every published figure is conservative" — does not hold, and a cost-savings product overstating savings is a different category of problem from understating them.

⚠️ Load-bearing caveat: "OpenAI/Gemini charge nothing to write a cache entry" is REASONED (domain knowledge), not verified in-session. The 25% figure depends on it. Please confirm against OpenAI's and Google's own pricing pages before acting.

3. The 3.38× EstimatorDivergence citation is the wrong instrument

Three independent reasons, all CONFIRMED from dash/overview.go:838-856:

• Disjoint population. ratioPop requires r.tokens_before = r.tokens_after — rows where nothing was compacted. saved_usd is nonzero exactly where those differ. It is measured on the strict complement of the affected population.
• Wrong quantity. It is a whole-request ratio carrying fixed overhead (system prompt, tool declarations, envelope). saved_usd's error is a difference of two counts on the same request, in which that overhead cancels. overview.go:362-364 says as much itself.
• The 3.38× is driven by the term that cancels. Fitting billed = K + f·ours to the two provider-billed points: f = 1.1442, K = 29,047, reproducing 149,363 exactly.

So the correct correction factor is f alone (≈1.14), which is size-independent — this strengthens the substantive claim (12.4% is not a small-transcript artifact) while invalidating the stated reasoning. Recommend dropping the 3.38× paragraph and stating f.

4. Two things the issue should state about its own evidence

• n=1. As written it reads as characterized across traffic. It is one turn. The divergence SQL has no size parameter or bucketing hook, so this could not be widened here.
• It is a cross-turn substitution. 149,363 is turn 15's write used as turn 17's counterfactual; ours grows 105,155 → 105,367 between them, so the true delta is ~117,483. Only 0.2%, but it is presented as a single-turn measurement.

Also worth noting the constituent errors (−44,208 and −29,435 tokens) are 3–4× larger than their difference (−14,773) — 12.42% is the residue of two large, partly offsetting errors, which is why it is not a stable global correction factor.

5. Attempted to flip the sign on Anthropic and could not — NEGATIVE RESULT

12 live paired samples (schema.MessagesTokens vs provider-billed prompt_tokens, same bodies, envelope differenced out):

sample	f		sample	f
hex_hashes	1.0001		tool_result_json	1.2445
big_numbers	1.0002		base64_blob	1.2799
uuids	1.0293		emoji	1.3753
english_prose	1.0606		go_code	1.3860
			cjk	1.3946
			many_tiny_msgs_120	1.9861
			single_char_run	2.6684

Every hypothesis tested came back conservative. Notably many_tiny_msgs_120 is the most conservative case, not the least — Anthropic bills ~3 tokens of role/envelope per message that MessagesTokens never counts. Non-text blocks are unreachable anyway: schema.Rewritable (schema.go:185-189) returns false, so components skip those messages and they never enter Saved() on either side.

Corollary: the issue likely understates its own understatement. Real Claude Code content is mostly code and tool JSON → f ≈ 1.24–1.39 (19–28%), against the issue's implied 1.144. But f ranges 1.0001–2.67 across content types, so no single global factor is defensible from one turn.

6. A fourth site of the same root cause, and one comment that must go

dash/event.go:642 adds the 1h premium (2·p.Input − p.CacheWrite) to CostUSD only. Since BaselineCostUSD = CostUSD + baselineDeltaUSD, a genuine 1h row gets its actual cost corrected and its counterfactual left at 1.25× — under-reporting that row's baseline delta by a further 60%. Empirically inert today, but the gateway does grant 1h on haiku. Structural confirmation from the panel side: modelinfo.Price has no 1h field at all, so the credit side cannot split 5m/1h even in principle, while the debit side does correctly.

dash/overview.go:362-365 currently asserts:

"Dollars are unaffected: they come from the provider's own reported usage."

True of cost_usd, false of baseline_cost_usd (= CostUSD + baselineDeltaUSD, event.go:647). A reader auditing exactly this is told not to worry about it. This sentence is arguably why the defect survived, and it should go with the fix.

Prior art worth citing: dash/declcredit.go:9-13 documents the same root cause with an infinite under-report on the largest measured lever, and contains the fix precedent — FilterUSD is priced from a measured billed count, and the file explicitly refuses to add a MEASURED figure to a MODELLED one.

7. The scope list is incomplete — 11 further readers

Site	Publishes
proxy/promexport.go:797-805	cg_baseline_cost_usd, cg_saved_usd, cg_net_saved_usd
proxy/promexport.go:862-883	per-tenant baseline/savings
dash/metrics_export.go:120-121,161-186	the alerting series
proxy/campaign.go, dash/campaignsavings.go, dash/ui/campaigns.js	per-strategy campaign verdicts
docs/reference/routes.md:404	GET /api/breakdown "spent vs saved"
context-guru-plugin/skills/statusline/SKILL.md:14	total_saved_usd — shown to the end user in their terminal
docs/results/kv-ttl-savings-reconciliation.md, docs/results/measured-2026-08.md, docs/dashboard.md, docs/hosted.md, docs/components/extract_llm.md	published headline figures

Sharpest: cg_net_saved_usd's own help text at promexport.go:803-804 reads "This is the number to alert on." Second sharpest: the statusline is the only surface a customer sees without opening a dashboard.

8. The proposed fix is not well-defined per component — please say so in the issue

• requests does persist provider usage (fresh_input, cache_read, cache_write, cache_write_1h, …).
• request_components (dash/schema.go:250-270) has no billed columns, and cannot have any.

The provider bills per request; saved_usd is per component. Several components act on one request, so attributing billed tokens to individual components needs a per-component counterfactual — not observable, not stored. Even per request a billed delta needs a counterfactual: you have what the request did bill, never what it would have. The measured pair here only exists because turn 15 accidentally supplied one.

The issue's parenthetical about the written/read columns is true about the columns and does not reach the per-component question at all. Leaving this as an open design question invites someone to attempt it.

Recommended sequencing:

1. Fix the over-report first (§2) — smallest diff, needs no f and no counterfactual, and it is the only finding that makes a published figure overstate. Confirm the zero-write-cost premise first.
2. Delete overview.go:365. Document the event.go:642 gap the way providerCacheTTLMs documents its own inert defect.
3. Widen n before moving any headline up. One turn is not a warrant to inflate cg_net_saved_usd.
4. For per-component economics, decouple unit from granularity: publish a per-request billed delta as the headline, and make per-component figures shares of it (which saved_gross/saved_unique already support). Stop quoting per-component dollars to 8 decimals — that precision is not attainable in principle.
5. Extend the scope list with §7.
6. Pin both directions in a test, or the next reader re-derives "always conservative" from a one-sided test.

Not verified here

Every production statistic quoted in this issue's neighbourhood (133,064-row corpus; median 3.38×, p25 2.43, p75 4.64, p90 6.80; 12,882 uncompacted rows) comes from source-code comments (dash/event.go:857, dash/overview.go:357-360) — cited as such. /var/lib/context-guru/ is mode 700; it was never read and no copy was made, so the real distribution is unavailable to this audit.