feat(demos): ai-gateway demo with tiered token budgets and span-derived dashboards

[Ladas]

Adds demos/ai-gateway: the Praxis experimental gateway on its own KIND
cluster, in front of a real local model (Ollama), enforcing per-tier token
budgets, with Grafana dashboards built from Prometheus counters and Tempo span
metrics. Also enables the token-rate-limit-filter cargo feature on the
experimental server — this image exists to expose feature-gated upstream work,
and previously enabled only one of the four experimental AI filters.

What it demonstrates

Token budgets that a real workload actually reaches. Five rules in
configs/token-budget.yaml, evaluated in order, first match wins:

Rule	Match	Budget
premium	X-Tier: premium	20,000 tokens/min
agent-daily ×3	X-Model: one of qwen3.8:27b, qwen3-coder:30b, deepseek-r1:32b	10,000,000 tokens/day
free	catch-all	5,000 tokens/min

scripts/rate-limit-demo.sh sends 100 requests per tier and reproduces
53 admitted / 47 denied on free against 100 / 0 on premium — same load, same
model, same moment. Denials are decided at admission, before the upstream call,
so a rejected request costs no GPU time.

A daily budget is the same filter with a longer window; that is all a "total
budget" is here. window accepts ms/s/m/h, so a day is "24h" and a
week "168h" — "7d" is rejected and praxis keeps the previous config.

Tiering without client cooperation. model_to_header promotes the request
body's model into X-Model, and rules match on it, so Codex, opencode and
Claude Code land on the right budget without sending a custom header — none of
them let you.

Reservations sized from measurement. A bare curl costs ~406 tokens; one
trivial Codex turn costs ~9,400, because system prompts and tool schemas
dominate. At reserved_tokens: 1500 that recorded 7,971 tokens of overage over
four turns; at 10,000 a turn settles as
estimated 10000 / actual 9435 / refunded 565 / overage 0.

Dashboards

Four dashboards, 31 panels, every one carrying an i tooltip explaining
what it plots, which telemetry plane it comes from, and what a wrong value looks
like.

Dashboard	Panels
Praxis AI Gateway Overview	8
Praxis Token Budget & Rate Limiting	13
Praxis Filter Latency (from spans)	6
Praxis OTel Traces	4

They are built around a distinction that is easy to get wrong: Prometheus
counters cover 100% of requests and are sampling-independent; span metrics come
only from sampled traces. Prometheus therefore owns counts, rates and budget
state, while span metrics own latency distributions and the per-filter
breakdown. The Weighted Filter Cost panel takes its quantile from spans and its
rate from praxis_http_requests_total for exactly this reason — multiplying two
sampled series would under-report by the sampling factor while looking
plausible. A Sampling Ratio Cross-Check panel divides one plane by the other; a
drop means spans are being lost rather than sampled.

Traps the demo is built to avoid

• token_rate_limit must be declared before token_count. Response hooks
  run in reverse declared order, and it reconciles against the token.total
  metadata token_count writes. Declared after, it charges every request the
  flat estimate and the metrics look reasonable but are wrong — the signature is
  actual == estimated with zero refunds.
• Ollama stays on its default loopback binding. A KIND pod reaches it via
  host.docker.internal, which the runtime proxies from the host side.
  OLLAMA_HOST=0.0.0.0 is unnecessary and publishes the model server to the
  local network.
• The burst demo uses a small model deliberately. The limiter decides at
  admission, so the model is irrelevant to what is shown, but 100 requests must
  land inside the 1m window. qwen3.8:27b takes 74-78s under concurrency, so
  the budget would age out faster than it was consumed.

Caveats

• token_rate_limit is experimental, behind a cargo feature; its parent
  proposal is not accepted (ai#796) and the config surface may change.
• It does not authenticate — a header-matched rule trusts whatever reached
  it, so a real deployment needs an auth filter to set the tier header and strip
  client copies (grid#101).
• allow_public_admin and allow_private_endpoints are demo-only: the first
  exposes /api/log-level and /api/kv to the cluster, the second disables SSRF
  hardening so the gateway can reach the host. Both are called out in the README.
• The memory backend is per-process, and a sliding window retains one entry per
  request for the length of the window; backend.kind: valkey moves that state
  out of process.

Test plan

• praxis-forge up creates ai-gw-local; all five stacks deploy
• KIND pod reaches host Ollama via host.docker.internal
• qwen3.8:27b answers through the gateway
• Tiered rate limit reproduces 53/47 free and 100/0 premium across repeated runs
• Codex through the gateway lands on agent-daily, confirmed by the admitted counter
• All 31 panels render with data; queries verified against the live cluster
• cargo check --locked, make lint, shellcheck, taplo, markdownlint clean

Depends on praxis-proxy/forge#16 (extraPortMappings), merged.

pr-size-check counts 2,147 added lines against a 750 limit because its
exemption list covers examples/, tests/, benchmarks/ and docs/ but not
demos/. Needs skip/pr-conventions, or demos/** added to that list.

page@6aa31177458395a3dcd779d1c07cc299.webm

[Ladas]

merging this demo, I have followup with podman and tweked quickstart that need image built with this rate limitng feature enabled