perf(bpe): hand the splitter's spans to the model in tiles

[ArthurZucker]

What

The pre-tokenizer wrote every span of a chunk into one array and the model read it back. For a 1 MB chunk of english that is ~229k spans × 8 B = 1.8 MB written and 1.8 MB read — 3.5 bytes of span traffic per input byte, none of which stays in cache. The emit now hands the model small tiles (1024 spans, 8 KB) as it finds them, and no per-document span array exists.

Result

tokbench, gpt2, 29 corpora, 3 interleaved rounds, medians:

	geomean vs gigatoken
base (poc/target-encode-clean)	0.9914x
this PR	1.0165x

1.0241x ours-vs-ours, 22/29 corpora faster. All 30 corpora byte-exact against the reference tokenizer; 370 tk-encode tests and bitsplit's parity suite pass.

How

In bitsplit, a SpanSink trait with two implementations — SliceSink for the existing whole-chunk API and TileSink, which flushes to a consumer when it fills — over one generic emit_contr.

Flushing from inside the emit's own control flow is what lets this need no resumable state: a contraction chain (y'all'd've) can emit an unbounded run of spans from a single block, so a caller-driven "emit N spans then return" API would have to resume mid-chain, and getting that wrong changes tokenization. The mask build is untouched and still one pass, so the grammar — contractions, the \s+(?!\S) steal, the block-edge carry — is exactly as before. Grammar::supports_tiled is asked before classifying, so a grammar without a tiled emit does not classify twice.

Two details are load-bearing, both found by profiling rather than reasoning:

• The consumer is resolved to &PipelineBPE once, outside the tile loop. Handing tiles to PipelineModel::tokenize_spans put the enum dispatch on the per-tile path and kept the probe loop from inlining — a profile showed that symbol newly outlined at 58% of self time, called once per tile instead of once per chunk. That alone made the first version 4% slower than the array it replaced.
• tokenize_spans reserves the fast path's worst case up front, so its capacity test almost never fires — but sized by batch shape. spans.len() * MAX_INLINE_IDS is right for an 8 KB tile and a 3× over-allocation for a whole chunk (2.7 MB of buffer for 0.94 MB of tokens), which cost 3–5% on the models that have no tiled emit and so still see whole chunks.

Measured and deliberately not included

• #[cold] #[inline(never)] fallback, in gigatoken's probe_emit_slow shape: −4% on gpt2 and llama-3 0.4157x with chinese 14× slower. Their slow path takes 3.8% of pretokens and calls out to merge; every cache miss reaches ours and merge_word inlines into it, so #[cold] has LLVM optimise the merge engine for size.
• One-deep probe prefetch (prfm pldl1keep on the next span's home line): 0.9363x, and 0.9015x on ASCII cells alone. The table is L1-resident at inference sizes, so there is no latency to hide and the duplicated hash costs real cycles. gigatoken affords D = 16 only because its span batch already holds every key.

Review notes

• The tile size is TK_SPAN_TILE-overridable; a sweep from 64 to 65536 spans could not distinguish sizes above noise, so 1024 (8 KB) is a comfortable-in-L1 default rather than a tuned optimum.
• bitsplit's emit_contr is now generic over the sink. cl100k goes through SliceSink and is unchanged in behaviour.
• Only the gpt2/byte-level grammar has a tiled emit so far; every other grammar takes the existing path untouched.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.