Proposal: request-fit routing gate for two-tier routes

[panpan0000]

Design Proposal: Request-Fit Routing Gate

Status: draft.

Problem

Two-tier routes send everything to the efficient model unless a judge or signal
says otherwise. If the request is too long for that model, we only find out
after calling it: the upstream rejects with a context-window error, FallThrough
evicts the target for the rest of the session, and we retry on the capable
model. That works, but every overflowing session pays one wasted call plus its
latency.

There is a second problem. Small models handle long inputs badly even when the
input fits their window — retrieval and reasoning over long context degrade
faster for small models than for large ones. "It fits" does not mean "it will
be handled well".

What this is not

This gate does not guess task difficulty. Short prompts can be hard ("prove
this theorem"); long prompts can be trivial (a huge document, a simple
extraction question). Difficulty is the capability classifier's job — it reads
the task and is calibrated. The gate only escalates. It never says "this
request is short, send it to the cheap model".

When to turn it on

Only when the two tiers actually differ:

• Different context windows (32K vs 200K): the gate skips guaranteed rejections.
• Similar windows, but the small model is noticeably worse on long inputs.

If the two models have similar windows and similar long-context quality, there
is nothing to gain. Leave it off. The gate is opt-in per route.

Design

One new classifier plus the usual thin assembly — same pattern as
LlmTaskClassifier: classifier core, FallThrough wrapper, one RouteConfig
variant.

The whole logic:

est = estimate_input_tokens(request)   # chars/4 over instructions + messages + tools
est >= escalate_over_input_tokens  → route to strong tier
otherwise                          → abstain, the next classifier decides

One threshold. Above it, the strong tier is decided. Below it, the gate
abstains and the judge (if configured) or the default tier decides. The gate
sits first in the cascade because it is the cheapest check and skips guaranteed
failures before anyone pays for a judge call.

Config:

[routes.auto]
type = "request_fit"
weak_target = "small-model"
strong_target = "big-model"
escalate_over_input_tokens = 24000

The threshold is in tokens even though the estimator counts characters. No
tokenizer dependency. If chars/4 is too inaccurate for some model family, add
a per-route chars_per_token knob later.

The reactive context-window eviction stays. The gate makes overflows rare;
eviction catches the ones the estimate missed.

Alternatives I rejected

• Rule engine / predicate DSL: nobody asked for it.
• Per-target context_window with automatic filtering: cleaner in theory, but
  it changes the target config schema. Bigger change; separate proposal if
  someone wants it.
• Doing nothing: fine until you deploy a cheap small-window model as the
  efficient tier. Then every overflowing session wastes a call.

Later, if needed

• Output budget: gate on estimate + max_output_tokens. One more optional
  field, same classifier.
• Per-target windows: if targets ever declare context_window, the threshold
  can default to a fraction of the window instead of an absolute number.

Both are new config fields, not new types.

Tests

• Estimator: text, tool calls/results, multimodal blocks, instructions;
  behavior exactly at the threshold.
• Cascade: gate abstains and the judge decides; gate escalates and the judge is
  never called.
• Regression: weak target returns ContextWindowExceeded after the gate passed
  the request; eviction still kicks in.
• Server: TOML round-trip, unknown fields rejected, missing threshold rejected.

Open questions

1. Name: request_fit, length_gate, something else?
2. v1 standalone route only, or also expose the classifier for
   stage_router-style compositions?
3. Is chars/4 good enough across the tokenizers we actually deploy, or does
   v1 need the chars_per_token knob?