feat(pricing): keep models.dev context tiers and fast multipliers in the catalog - #2232
feat(pricing): keep models.dev context tiers and fast multipliers in the catalog#2232svarlamov-git-ai wants to merge 4 commits into
Conversation
…the catalog The catalog previously flattened models.dev to four flat rates, so long-context tiers and fast/priority speed pricing were unpriceable downstream. ModelPricing now carries the model's long-context band (whole-request above-threshold rates + threshold from models.dev cost.tiers, lowest context bound wins) and a fast-speed multiplier. Neither vendors nor models.dev publish fast-tier pricing, and models.dev's first-party Anthropic entries omit their long-context premium (it only appears under gateway spellings the provider allowlist drops), so both are hand-tracked override tables applied at trim time: fast multipliers ported verbatim from ccusage's overrides (MIT), and Anthropic's published 200K premium for the 1M-context sonnet/opus models. Cache rates become Option so unpublished rates stay distinguishable from explicit zeros; cache_read_rate()/cache_write_rate() accessors keep ccusage's models.dev-loader defaults (0.1x / 1.25x input). The snapshot is regenerated with the new fields; old snapshot/cache JSON still parses (missing fields default to None / 1.0). Cost math is unchanged in this commit beyond the 1.25x cache-write default for unpublished rates (cost-inert for Codex, which reports no cache writes); consumers of the new tier fields land separately. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
…terpreted The optional-rate schema reads a pre-tier cache's explicit 0.0 cache rates as published-free pricing, and such a cache carries no tiers or fast multipliers — yet it would win over the embedded snapshot until a refresh (only `git-ai usage` triggers one, 24h-throttled). Caches now carry a format version: other-format files are ignored (the embedded snapshot applies) and a failed refresh never relabels their models as current. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| /// "fast" tier is not its API priority pricing), so they are hand-tracked | ||
| /// against vendor price sheets. Exact entries apply to that catalog id only — | ||
| /// "gpt-5.5-pro" must not inherit gpt-5.5's multiplier. | ||
| const FAST_MULTIPLIER_EXACT: [(&str, f64); 6] = [ |
There was a problem hiding this comment.
[code-review] Fallback pricing leaks hand-tracked fast multipliers and long-context tiers to uncataloged model ids
The doc here says "Exact entries apply to that catalog id only — 'gpt-5.5-pro' must not inherit gpt-5.5's multiplier", but pricing_for's fallbacks return catalog entries verbatim, multiplier and tier included:
- Token-boundary containment (
.filter(|(id, _)| contains_at_token_boundary(&model, id)).max_by_key(...)): an uncatalogedgpt-5.5-codexmatches thegpt-5.5entry and bills fast requests at 2.5x (the codex-family fast multiplier is 2.0x pergpt-5.3-codex). family_fallbackmedian: any unknown opus id (e.g.claude-opus-4-9, the doc's own "successors newer than the catalog" scenario) resolves to the median entryclaude-opus-4-7—fast_multiplier: 6.0plus the 200K tier — so every fast-speed request on a new opus bills 6x, and the leaked threshold also flipsis_long_context, polluting the wire's long-context token splits with conjectured tier data.
Pre-existing fallbacks only leaked base rates; multipliers/tiers riding along is new here. Consider stripping fast_multiplier/tier fields (reset to defaults) on non-exact matches, or at least on family_fallback results.
There was a problem hiding this comment.
Fixed in d3ee577: the family-median fallback now neutralizes the fast multiplier and long-context tier to the median member's base rates (an unknown opus id no longer bills 6x or flips the long-context split). Token-boundary matches deliberately keep the full entry — that is how date-suffixed spellings inherit their base model's data, and ccusage's fuzzy lookup returns full entries (multiplier included) the same way.
| if let Ok(pricing) = serde_json::from_value::<ModelPricing>(cost.clone()) { | ||
| entries.insert(model_id.to_lowercase(), pricing); | ||
| } | ||
| let Ok(cost) = serde_json::from_value::<ModelsDevCost>(cost.clone()) else { |
There was a problem hiding this comment.
[code-review] Strictly-typed tier parsing silently drops whole models the old code kept at flat rates
ModelsDevCost.tiers: Vec<ModelsDevTier> makes a single unexpected tier shape (scalar element, string-encoded number like "input": "4.0", "size": "272000", or tiers: null) fail the whole from_value, and this else { continue; } then drops the model from the fetched catalog with no log. The previous from_value::<ModelPricing> ignored tiers as an unknown field and kept the model at flat rates. A models.dev format tweak now silently sends a model to the family-median fallback (or $0) even though its flat input/output rates were parseable.
Cheap hardening: deserialize tiers as Vec<serde_json::Value> (or #[serde(deserialize_with = ...)] that skips bad elements) so tier parse failures degrade to flat rates, and/or debug_log the skip.
There was a problem hiding this comment.
Fixed in d3ee577: tier bands are held as raw JSON and parsed per band, so a novel-shaped band (or tiers: null) degrades that model to its flat rates instead of dropping it from the catalog; parseable bands next to a malformed one still apply. Pinned by one_malformed_tier_band_degrades_to_flat_rates_not_a_dropped_model.
| let model_id = model_id.to_lowercase(); | ||
| let long_context = cost | ||
| .long_context_rates() | ||
| .or_else(|| prefix_override(&CLAUDE_LONG_CONTEXT_PREFIX, &model_id)); |
There was a problem hiding this comment.
[code-review] Overrides are stamped at fetch/trim time, so fetched caches freeze old override tables indefinitely
trim_catalog stamps FAST_MULTIPLIER_*/CLAUDE_LONG_CONTEXT_PREFIX into the entries it writes to the on-disk cache, and catalog_with_id returns cache.models verbatim whenever a current-format cache exists. PRICING_CACHE_VERSION only guards shape changes, so a binary upgrade that edits an override value (a new opus fast multiplier, a corrected tier) is inert on any machine holding a current-format cache until the next successful fetch — and the refresh path runs from git-ai usage, so a user who never runs that command keeps the stale multipliers forever while the daemon prices every entry with them (costs are event-time and never repriced). Stamping overrides at catalog-load time (in PricingCatalog::from_entries or catalog_with_id) would make them binary-versioned with no staleness window, and the cache would only need to carry the raw models.dev data.
There was a problem hiding this comment.
Fixed in d3ee577: the override tables now apply when a catalog is LOADED (one choke point for the embedded snapshot and the fetched cache), so a table edit shipped in a new binary takes effect immediately; the trim output and snapshot become pure models.dev data. Application is idempotent over pre-split caches with baked values, and an edited table wins over them.
…iers Three review findings on the catalog: - The hand-tracked override tables (fast multipliers, Claude 200K premium) were stamped at trim time and baked into the snapshot and fetched cache, so a binary upgrade editing them was inert against an existing cache until the next successful fetch — which only `git-ai usage` triggers. They now apply when a catalog is LOADED (one choke point for the embedded snapshot and the disk cache), the trim output stays pure models.dev data, and the snapshot is regenerated accordingly. Application is idempotent over pre-split caches that carry baked values, and an edited table wins over them. - The family-median fallback returned the median member's entry verbatim, leaking its hand-tracked fast multiplier and long-context tier onto uncataloged ids: an unknown opus id resolving to the 6x-multiplier median would bill a fast-speed request at 6x and flip the emitted long-context splits. Family estimates now neutralize multipliers and tiers to the base rates. (Token-boundary matches still return entries verbatim — that is how date-suffixed spellings inherit their base model's data, and ccusage's fuzzy lookup behaves the same way.) - One novel-shaped tier band (or tiers: null) made the whole model's cost object unparseable and silently dropped it from the fetched catalog. Tier bands now parse individually and leniently, so odd data degrades that model to its flat rates instead. ModelPricing becomes Copy and pricing_for returns it by value so the fallback can synthesize a neutralized entry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
95d32f4 to
d3ee577
Compare
CLAUDE_LONG_CONTEXT_PREFIX lists both "claude-sonnet-4" and its more specific generations, and first-match resolution let the shorter row shadow them — identical constants today, but an edit to one generation's premium would silently never apply. Prefix overrides now prefer the longest matching key (pinned by test). Also drops a needless per-band clone in tier parsing and fixes the module doc's stale claim that overrides apply at trim time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2 participants
The catalog previously flattened models.dev to four flat rates, so
long-context tiers and fast/priority speed pricing were unpriceable
downstream. ModelPricing now carries the model's long-context band
(whole-request above-threshold rates + threshold from models.dev
cost.tiers, lowest context bound wins) and a fast-speed multiplier.
Neither vendors nor models.dev publish fast-tier pricing, and
models.dev's first-party Anthropic entries omit their long-context
premium (it only appears under gateway spellings the provider allowlist
drops), so both are hand-tracked override tables applied at trim time:
fast multipliers ported verbatim from ccusage's overrides (MIT), and
Anthropic's published 200K premium for the 1M-context sonnet/opus
models.
Cache rates become Option so unpublished rates stay distinguishable
from explicit zeros; cache_read_rate()/cache_write_rate() accessors
keep ccusage's models.dev-loader defaults (0.1x / 1.25x input). The
snapshot is regenerated with the new fields; old snapshot/cache JSON
still parses (missing fields default to None / 1.0).
Cost math is unchanged in this commit beyond the 1.25x cache-write
default for unpublished rates (cost-inert for Codex, which reports no
cache writes); consumers of the new tier fields land separately.
Co-Authored-By: Claude Fable 5 noreply@anthropic.com
Stack created with GitHub Stacks CLI • Give Feedback 💬
🤖 Generated with Claude Code