feat(routing): prompt caching, session stickiness, and cache-regret cost gate#982
feat(routing): prompt caching, session stickiness, and cache-regret cost gate#982Spherrrical wants to merge 8 commits into
Conversation
Reconcile prompt caching with intelligent routing: zero-config implicit session affinity, cache_control preservation/injection across transforms, cache-adjusted routing economics, and a response-driven cache-hit feedback loop.
# Conflicts: # crates/brightstaff/src/handlers/llm/mod.rs # crates/brightstaff/src/router/model_metrics.rs # crates/common/src/configuration.rs
Spherrrical
changed the title
Add opt-in session_stickiness.record_counterfactual that emits the plano.switch.counterfactual_route OTEL attribute when the cache-regret gate retains the previous model, capturing the route it would have taken had the switch been allowed. Telemetry only; the candidate is never dispatched. Includes schema, demo config, and a getting-started guide.
|
@Spherrrical Thanks for pushing this PR!
|
…sive modules Extract the two concerns interleaved in the LLM handler into dedicated sibling modules for readability: prompt_caching.rs (cache-marker injection) and session_stickiness.rs (session key/prefix-hash resolution, pin lookup, cache-regret gate, and pin planning). handlers/llm/mod.rs is now a thin orchestrator that calls them at clear phase boundaries. Behavior-preserving.
It's not limited to OpenAI + Anthropic. The strategy is resolved per (gateway × model family × upstream API) in cache_marker_strategy: OpenAI-family caches automatically (no markers) across OpenAI/Azure/etc.; Anthropic-family gets native Messages breakpoints or, behind OpenAI-compatible gateways (DigitalOcean, OpenRouter), content-part cache_control; unimplemented backends (Bedrock) are an honest None. Crucially, we don't assume caching worked, we read it back from each provider's own response usage
Done!
The only change there is a one-line because the caching work added two fields to the shared Usage struct so every existing literal had to initialize them. It's not a revival of Arch-Function. The handler is technically still wired (/function_calling route), so if it's truly deprecated we can remove it in a separate cleanup PR rather than bundling that here.
It changed because session stickiness introduced stale pins: get() now returns
Done! |
Why only the first user message in determining the prompt prefix? Separately, I don't follow this notion of "auto injected". That whole sentence is slop.
How would we know that the cache is "plausibly" still warm? And I don't follow this whole regret cost gating math one bit. Not clear to me, how will it be clear to the developers building with plano.
I am not sure what this means? how does this work? what does the developer have to do? Separately I am not sure how can you compute prefix trees unless you are storing the prompt prefix somewhere in a database Please share the user experience in the config.yaml here. Don't link to the demo.
|
3 participants
Summary
Intelligent routing and prompt caching pull against each other: routing wants freedom to move traffic across models, while provider caches are per-model and only pay off when a conversation stays put. Every reroute silently abandons a warm cache and re-bills the whole prompt at the uncached rate — so a "cheaper" model can end up more expensive. This PR makes the two work together, without letting caching hijack routing quality.
X-Model-Affinityheader, Plano derives a stable session key from the prompt prefix (system + tools + first user message) so multi-turn conversations stay pinned and warm. Model-aware cache-control markers are auto-injected — Anthropic breakpoints or OpenAI content-partcache_control, idempotent and threshold-guarded — so OpenAI-shaped clients get caching on Anthropic-family upstreams. Caching never influences which model routing selects.context_tokens × (candidate_uncached_rate − previous_cached_rate)— against a developer-defined threshold, and either switches or retains the previous model. Input-only by design (no output-token gamble); negative regret always switches; drifted prefix means the cache is already cold, so the switch is free. Fully opt-in with no baked-in numbers (max_regret_usdormax_regret_pct_of_cached). The router still decides which model is better — the gate only vetoes a switch that isn't affordable.cache_read, with acache_read_discountfallback for feeds like DO that omit it) to drive the regret math; blended cost ranking is untouched.brightstaff_session_switch_decisions_total{decision}, pin-event and prompt-cache hit/miss counters, plus routing-span attributesplano.switch.regret_usd,plano.switch.threshold_usd, andplano.switch.decision— so a trace shows exactly why a request switched or stuck.endpoints.<name>.prefix_affinity(ring-hash onx-plano-prefix-hash), independent of prompt caching.Sample config + walkthrough under
demos/llm_routing/session_stickiness/.Test plan
cargo test --lib(brightstaff + hermesllm + common) — passcargo clippy --all-targets --all-features -- -D warnings— cleancd cli && uv run pytest— schema/template renderX-Plano-Cache: offdisables per request