chore: first round of cleanup in preparation for 1.0.0rc0 by SBrandeis · Pull Request #2328 · huggingface/tokenizers

SBrandeis · 2026-08-10T09:01:03Z

PipelineTokenizer benchmark

10 / 10 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · pipeline: whole corpus · release: ~2 MB/fixture sample · ~10 kB chunks · cold caches · add_special_tokens on · 2/4-thread sweep per group

f82130ba4 · 2026-08-10 16:03 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 48 cores

Decode — the release encodes each fixture's decode sample (add_special_tokens=true) and both implementations decode those SAME ids with skip_special_tokens=false, so the comparison is decode alone. MB/s counts the input bytes the ids came from, the same denominator as the encode charts.

vs base branch (89c9377f4) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.

_{base numbers measured in this run's jobs, on the same runners as the PR's}

Work vs base (89c9377f4, allocation lane):

encode allocations: ⚠ counts changed on 38 of 150 fixtures: bert-base-uncased/added_normalized_sparse -99.29%, bert-base-uncased/added_special_sparse -86.88%, bert-base-uncased/tam_Taml -28.57%, bert-base-uncased/hin_Deva -28.53%, bert-base-uncased/tha_Thai -28.50%, bert-base-uncased/eng_Latn -28.44%, and 32 more
decode allocations: ✓ exactly identical on all 150 fixtures

_{allocation lane measured on: Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · glibc 2.39}

bert-base-uncased — normalizer-heavy WordPiece · ×5.04 vs v0.23.1 · ×1.04 vs base · decode ×0.99

bert-base-uncased thread scaling (modalities)

bert-base-uncased decode thread scaling (lang)

bert-base-uncased decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 7+11+0 (peak 37) · Pipeline 4+4+1 (peak 42)

Allocations (encode pass, whole corpus): v0.23.1: 159.5M allocs, 52.04 GB allocated, peak live 37 MB; Pipeline: 37.1k allocs, 0.32 GB allocated, peak live 45 MB

Allocations (decode pass, decode sample): v0.23.1: 98.5M allocs, 1.26 GB allocated; Pipeline: 98.5M allocs, 1.26 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	7.8	27.2	×3.47	×0.98	473 (×2,038)	×0.99	match
arb_Arab	lang	3.8	25.8	×6.75	×0.99	565 (×4,841)	×1.01	match
cmn_Hani	lang	3.6	18.1	×5.09	×0.99	665 (×5,959)	×0.92	match
eng_Latn	lang	4.4	19.4	×4.46	×0.97	479 (×4,440)	×1.02	match
hin_Deva	lang	6.4	28.7	×4.49	×0.98	459 (×3,405)	×1.01	match
jpn_Jpan	lang	4.2	27.3	×6.47	×0.99	672 (×4,297)	×1.08	match
rus_Cyrl	lang	3.5	25.0	×7.17	×0.98	564 (×6,302)	×1.09	match
tam_Taml	lang	6.9	38.8	×5.62	×0.98	460 (×3,035)	×1.02	match
tha_Thai	lang	8.4	32.8	×3.92	×0.99	445 (×2,162)	×1.01	match
added_normalized_sparse	modalities	5.3	19.4	×3.65	×0.99	490 (×3,280)	×1.06	match
added_special_sparse	modalities	3.9	23.1	×5.87	×0.99	11,767 (×296)	×1.33	match
agentic-traces	modalities	3.8	19.2	×5.04	×0.99	526 (×4,972)	×1.02	match
agentic_swe	modalities	3.9	19.7	×5.10	×0.99	566 (×4,641)	×1.02	match
code_mixed	modalities	3.8	19.6	×5.21	×0.99	582 (×4,826)	×1.01	match
math_latex	modalities	3.9	19.1	×4.87	×1.00	491 (×5,023)	×1.05	match

deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×14.95 vs v0.23.1 · ×1.01 vs base · decode ×13.28

deepseek-v4 decode thread scaling (lang)

deepseek-v4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 56+0+0 (peak 75) · Pipeline 16+0+0 (peak 56)

Allocations (encode pass, whole corpus): v0.23.1: 153.2M allocs, 38.67 GB allocated, peak live 66 MB; Pipeline: 28.0k allocs, 0.23 GB allocated, peak live 57 MB

Allocations (decode pass, decode sample): v0.23.1: 15.3M allocs, 0.77 GB allocated; Pipeline: 7.4k allocs, 0.06 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.1	62.9	×15.27	×21.24	570 (×3,582)	×1.00	match
arb_Arab	lang	4.1	53.3	×13.04	×12.29	378 (×5,022)	×1.00	match
cmn_Hani	lang	3.4	43.2	×12.86	×10.65	397 (×4,758)	×1.01	match
eng_Latn	lang	3.0	67.0	×21.98	×11.90	385 (×7,925)	×1.06	match
hin_Deva	lang	5.3	85.3	×16.09	×11.68	366 (×4,352)	×1.02	match
jpn_Jpan	lang	3.8	42.6	×11.32	×11.99	418 (×3,717)	×1.02	match
rus_Cyrl	lang	4.0	41.3	×10.44	×10.22	376 (×4,319)	×0.96	match
tam_Taml	lang	5.5	50.2	×9.12	×12.57	368 (×2,674)	×1.03	match
tha_Thai	lang	5.1	23.3	×4.55	×10.84	357 (×1,476)	×1.02	match
added_normalized_sparse	modalities	5.3	85.3	×16.11	×18.91	490 (×5,070)	×0.95	match
added_special_sparse	modalities	3.9	58.8	×15.06	×13.25	391 (×8,796)	×0.99	match
agentic-traces	modalities	2.6	66.3	×25.16	×14.46	430 (×8,982)	×1.03	match
agentic_swe	modalities	2.8	76.2	×27.19	×16.49	470 (×7,846)	×1.01	match
code_mixed	modalities	2.8	65.8	×23.55	×15.09	467 (×7,449)	×1.02	match
math_latex	modalities	2.5	60.9	×24.02	×12.18	387 (×9,256)	×1.01	match

gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×3.02 vs v0.23.1 · ×1.05 vs base · decode ×1.55

gemma-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 285+0+0 (peak 365) · Pipeline 0+0+0 (peak 365)

Allocations (encode pass, whole corpus): v0.23.1: 29.1M allocs, 23.69 GB allocated, peak live 310 MB; Pipeline: 65.2k allocs, 2.35 GB allocated, peak live 310 MB

Allocations (decode pass, decode sample): v0.23.1: 18.9M allocs, 1.79 GB allocated; Pipeline: 18.9M allocs, 1.79 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	10.7	43.9	×4.09	×1.53	780 (×521)	×0.98	match
arb_Arab	lang	6.9	22.6	×3.27	×1.50	753 (×514)	×1.02	match
cmn_Hani	lang	12.2	51.0	×4.19	×1.29	831 (×350)	×1.11	match
eng_Latn	lang	3.8	8.8	×2.34	×1.61	768 (×732)	×1.05	match
hin_Deva	lang	9.2	27.5	×2.99	×1.59	732 (×375)	×1.05	match
jpn_Jpan	lang	13.0	43.2	×3.34	×1.30	764 (×303)	×1.10	match
rus_Cyrl	lang	7.2	19.2	×2.67	×1.45	752 (×414)	×1.07	match
tam_Taml	lang	11.0	29.7	×2.71	×1.46	736 (×261)	×1.05	match
tha_Thai	lang	13.4	39.4	×2.95	×1.37	711 (×234)	×1.12	match
added_normalized_sparse	modalities	4.5	12.4	×2.77	×1.74	881 (×659)	×1.04	match
added_special_sparse	modalities	4.2	13.5	×3.22	×1.90	156,197 (×9.98)	×1.01	match
agentic-traces	modalities	4.3	11.6	×2.70	×1.57	788 (×825)	×1.04	match
agentic_swe	modalities	4.3	14.3	×3.33	×1.75	849 (×1,004)	×1.04	match
code_mixed	modalities	4.4	13.0	×2.97	×1.73	873 (×797)	×1.05	match
math_latex	modalities	4.0	9.8	×2.43	×1.60	740 (×792)	×1.04	match

gpt2 — gpt2 ByteLevel regex · ×22.35 vs v0.23.1 · ×1.00 vs base · decode ×29.87

Memory (RSS MB, load+encode+decode): v0.23.1 21+12+0 (peak 82) · Pipeline 2+1+0 (peak 75)

Allocations (encode pass, whole corpus): v0.23.1: 243.3M allocs, 38.17 GB allocated, peak live 79 MB; Pipeline: 34.6k allocs, 0.43 GB allocated, peak live 76 MB

Allocations (decode pass, decode sample): v0.23.1: 30.9M allocs, 1.36 GB allocated; Pipeline: 6.0k allocs, 0.07 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	3.9	78.0	×19.78	×41.26	570 (×4,658)	×1.01	match
arb_Arab	lang	3.8	72.4	×19.26	×22.95	564 (×5,616)	×0.99	match
cmn_Hani	lang	3.3	46.1	×13.81	×30.06	572 (×4,864)	×0.98	match
eng_Latn	lang	3.3	81.6	×25.06	×21.43	386 (×8,670)	×1.00	match
hin_Deva	lang	3.0	83.3	×27.94	×37.20	549 (×7,411)	×0.99	match
jpn_Jpan	lang	3.7	40.2	×10.87	×24.30	504 (×4,683)	×0.99	match
rus_Cyrl	lang	3.8	70.1	×18.45	×23.60	564 (×5,089)	×0.99	match
tam_Taml	lang	2.6	85.3	×33.04	×52.01	553 (×10,027)	×0.98	match
tha_Thai	lang	3.2	72.2	×22.60	×45.40	533 (×6,617)	×1.00	match
added_normalized_sparse	modalities	4.5	101.8	×22.65	×30.59	490 (×6,465)	×0.99	match
added_special_sparse	modalities	3.6	72.5	×20.19	×19.22	391 (×8,582)	×1.02	match
agentic-traces	modalities	2.8	78.3	×27.53	×26.40	441 (×9,456)	×1.02	match
agentic_swe	modalities	2.9	89.4	×31.03	×36.53	491 (×8,756)	×1.00	match
code_mixed	modalities	2.8	77.2	×27.75	×32.06	496 (×8,611)	×1.01	match
math_latex	modalities	2.5	72.6	×28.96	×24.03	406 (×9,294)	×0.99	match

gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×9.96 vs v0.23.1 · ×0.87 vs base · decode ×14.01

gpt-oss decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 236+0+0 (peak 309) · Pipeline 0+0+0 (peak 311)

Allocations (encode pass, whole corpus): v0.23.1: 127.8M allocs, 29.89 GB allocated, peak live 272 MB; Pipeline: 28.4k allocs, 0.23 GB allocated, peak live 272 MB

Allocations (decode pass, decode sample): v0.23.1: 14.5M allocs, 0.71 GB allocated; Pipeline: 7.4k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.1	60.6	×14.69	×30.33	570 (×3,431)	×0.97	match
arb_Arab	lang	4.3	49.1	×11.41	×13.27	378 (×4,522)	×1.01	match
cmn_Hani	lang	3.9	21.7	×5.60	×18.61	474 (×2,360)	×1.03	match
eng_Latn	lang	3.8	58.6	×15.57	×15.13	383 (×7,145)	×1.00	match
hin_Deva	lang	5.8	71.7	×12.30	×11.00	366 (×3,680)	×0.95	match
jpn_Jpan	lang	4.4	29.5	×6.67	×13.01	475 (×1,990)	×1.03	match
rus_Cyrl	lang	4.4	38.4	×8.77	×9.15	376 (×3,878)	×1.03	match
tam_Taml	lang	5.7	39.1	×6.86	×11.17	368 (×2,372)	×1.01	match
tha_Thai	lang	5.4	20.0	×3.67	×10.31	358 (×1,345)	×1.02	match
added_normalized_sparse	modalities	5.8	30.2	×5.21	×17.40	490 (×4,723)	×0.31	match
added_special_sparse	modalities	4.7	28.7	×6.05	×11.53	391 (×7,345)	×0.41	match
agentic-traces	modalities	3.8	65.4	×17.40	×14.40	414 (×7,406)	×0.97	match
agentic_swe	modalities	3.9	79.8	×20.41	×16.30	458 (×6,524)	×1.00	match
code_mixed	modalities	3.8	74.9	×19.81	×15.44	447 (×6,828)	×0.98	match
math_latex	modalities	3.5	56.5	×16.11	×12.64	386 (×7,848)	×1.00	match

glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×14.33 vs v0.23.1 · ×0.95 vs base · decode ×20.96

glm-5.2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 163+0+0 (peak 225) · Pipeline 0+0+0 (peak 226)

Allocations (encode pass, whole corpus): v0.23.1: 154.7M allocs, 33.00 GB allocated, peak live 212 MB; Pipeline: 29.5k allocs, 0.25 GB allocated, peak live 212 MB

Allocations (decode pass, decode sample): v0.23.1: 18.3M allocs, 0.90 GB allocated; Pipeline: 6.8k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.0	72.9	×18.38	×40.36	570 (×3,814)	×1.01	match
arb_Arab	lang	4.1	54.3	×13.10	×16.16	384 (×4,896)	×0.99	match
cmn_Hani	lang	4.3	21.7	×5.05	×15.92	425 (×2,520)	×1.06	match
eng_Latn	lang	3.7	68.5	×18.35	×17.50	384 (×7,146)	×1.06	match
hin_Deva	lang	3.9	88.1	×22.76	×17.88	458 (×6,443)	×1.00	match
jpn_Jpan	lang	4.7	24.4	×5.19	×17.51	433 (×2,153)	×0.98	match
rus_Cyrl	lang	4.2	38.3	×9.08	×12.61	376 (×3,881)	×1.00	match
tam_Taml	lang	3.8	95.0	×24.98	×27.62	464 (×6,456)	×0.98	match
tha_Thai	lang	4.5	70.7	×15.73	×28.25	445 (×3,747)	×1.01	match
added_normalized_sparse	modalities	4.5	58.5	×12.90	×29.26	490 (×4,755)	×0.58	match
added_special_sparse	modalities	3.8	42.0	×11.16	×18.38	391 (×7,379)	×0.74	match
agentic-traces	modalities	3.5	68.4	×19.64	×21.62	422 (×7,221)	×1.01	match
agentic_swe	modalities	3.6	80.1	×22.02	×23.15	460 (×6,451)	×1.00	match
code_mixed	modalities	3.5	70.4	×20.07	×22.33	446 (×6,650)	×1.01	match
math_latex	modalities	3.3	61.1	×18.81	×19.63	390 (×7,773)	×0.99	match

llama-2 — model-bounded BPE, no pre-tokenizer · ×4.95 vs v0.23.1 · ×1.03 vs base · decode ×1.52

llama-2 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 20+6+0 (peak 62) · Pipeline 0+3+1 (peak 63)

Allocations (encode pass, whole corpus): v0.23.1: 63.0M allocs, 28.31 GB allocated, peak live 61 MB; Pipeline: 67.7k allocs, 0.56 GB allocated, peak live 68 MB

Allocations (decode pass, decode sample): v0.23.1: 35.7M allocs, 3.03 GB allocated; Pipeline: 35.7M allocs, 3.03 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	5.6	87.2	×15.48	×1.34	850 (×3,793)	×1.02	match
arb_Arab	lang	12.4	76.5	×6.16	×1.71	845 (×844)	×1.06	match
cmn_Hani	lang	11.1	107.5	×9.67	×1.30	849 (×1,468)	×1.00	match
eng_Latn	lang	4.4	11.7	×2.68	×1.65	823 (×744)	×1.08	match
hin_Deva	lang	14.1	108.7	×7.69	×1.44	824 (×847)	×1.04	match
jpn_Jpan	lang	15.9	126.3	×7.94	×1.30	859 (×828)	×0.96	match
rus_Cyrl	lang	8.7	30.4	×3.48	×1.43	753 (×505)	×1.01	match
tam_Taml	lang	15.1	128.5	×8.49	×1.51	829 (×938)	×1.06	match
tha_Thai	lang	19.2	128.9	×6.72	×1.35	800 (×638)	×1.12	match
added_normalized_sparse	modalities	5.1	15.7	×3.06	×1.43	881 (×772)	×1.02	match
added_special_sparse	modalities	4.5	16.1	×3.56	×1.92	145,162 (×11.23)	×1.01	match
agentic-traces	modalities	4.6	13.7	×2.97	×1.53	791 (×916)	×1.01	match
agentic_swe	modalities	4.5	13.7	×3.04	×1.78	851 (×1,059)	×0.98	match
code_mixed	modalities	4.6	14.3	×3.10	×1.69	877 (×1,010)	×1.02	match
math_latex	modalities	4.4	12.5	×2.87	×1.57	762 (×860)	×1.00	match

llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×16.24 vs v0.23.1 · ×0.97 vs base · decode ×18.95

llama-3 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 67+2+0 (peak 89) · Pipeline 7+0+0 (peak 90)

Allocations (encode pass, whole corpus): v0.23.1: 152.3M allocs, 33.83 GB allocated, peak live 91 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 92 MB

Allocations (decode pass, decode sample): v0.23.1: 17.3M allocs, 0.88 GB allocated; Pipeline: 9.6k allocs, 0.06 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.2	80.1	×19.09	×29.35	570 (×3,818)	×0.99	match
arb_Arab	lang	4.3	60.2	×14.11	×13.49	379 (×4,900)	×0.95	match
cmn_Hani	lang	4.4	27.9	×6.36	×18.36	476 (×2,372)	×1.04	match
eng_Latn	lang	3.9	75.0	×19.19	×17.60	384 (×7,149)	×1.04	match
hin_Deva	lang	4.4	121.5	×27.55	×15.92	366 (×7,163)	×1.01	match
jpn_Jpan	lang	4.8	25.8	×5.33	×17.07	433 (×2,151)	×1.02	match
rus_Cyrl	lang	4.6	43.6	×9.40	×12.63	376 (×3,985)	×1.04	match
tam_Taml	lang	3.9	101.7	×25.82	×26.01	464 (×6,460)	×0.99	match
tha_Thai	lang	4.6	60.5	×13.16	×16.05	356 (×4,118)	×0.97	match
added_normalized_sparse	modalities	4.6	99.3	×21.55	×28.94	490 (×4,760)	×0.81	match
added_special_sparse	modalities	3.9	69.8	×18.11	×17.20	391 (×7,385)	×0.87	match
agentic-traces	modalities	3.3	75.8	×22.75	×20.93	414 (×7,327)	×0.98	match
agentic_swe	modalities	3.7	88.0	×24.04	×23.48	458 (×6,458)	×0.96	match
code_mixed	modalities	3.5	75.6	×21.50	×19.29	446 (×6,650)	×0.99	match
math_latex	modalities	3.2	64.8	×20.35	×16.85	388 (×7,791)	×0.98	match

mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×11.25 vs v0.23.1 · ×0.99 vs base · decode ×14.38

mistral-small-4 thread scaling (modalities)

mistral-small-4 decode thread scaling (lang)

mistral-small-4 decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 151+0+0 (peak 188) · Pipeline 0+0+0 (peak 189)

Allocations (encode pass, whole corpus): v0.23.1: 135.2M allocs, 31.26 GB allocated, peak live 185 MB; Pipeline: 28.7k allocs, 0.24 GB allocated, peak live 185 MB

Allocations (decode pass, decode sample): v0.23.1: 16.6M allocs, 0.81 GB allocated; Pipeline: 7.1k allocs, 0.05 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	4.1	59.7	×14.68	×26.41	570 (×3,847)	×0.97	match
arb_Arab	lang	4.4	45.1	×10.25	×10.36	378 (×4,557)	×0.95	match
cmn_Hani	lang	4.4	27.8	×6.38	×15.12	476 (×2,565)	×1.02	match
eng_Latn	lang	3.8	54.6	×14.27	×12.50	388 (×7,243)	×1.01	match
hin_Deva	lang	5.9	69.6	×11.80	×10.67	366 (×3,846)	×0.99	match
jpn_Jpan	lang	4.8	32.2	×6.67	×13.54	471 (×2,080)	×1.02	match
rus_Cyrl	lang	4.4	37.3	×8.44	×10.34	376 (×4,023)	×0.98	match
tam_Taml	lang	5.9	38.6	×6.50	×11.46	368 (×2,471)	×0.97	match
tha_Thai	lang	5.9	21.9	×3.67	×12.43	358 (×1,515)	×1.00	match
added_normalized_sparse	modalities	5.8	83.2	×14.39	×19.35	490 (×4,858)	×0.93	match
added_special_sparse	modalities	4.7	62.4	×13.23	×15.35	391 (×7,459)	×0.95	match
agentic-traces	modalities	3.3	61.9	×18.49	×16.17	438 (×7,687)	×1.02	match
agentic_swe	modalities	3.4	72.0	×20.90	×18.03	471 (×6,995)	×1.01	match
code_mixed	modalities	3.4	71.1	×21.11	×16.98	465 (×6,767)	×1.01	match
math_latex	modalities	3.2	55.2	×17.35	×14.32	396 (×7,918)	×1.03	match

t5-base — Unigram + Metaspace · ×3.16 vs v0.23.1 · ×1.09 vs base · decode ×0.95

t5-base decode thread scaling (modalities)

Memory (RSS MB, load+encode+decode): v0.23.1 31+10+0 (peak 56) · Pipeline 19+11+1 (peak 80)

Allocations (encode pass, whole corpus): v0.23.1: 260.7M allocs, 35.16 GB allocated, peak live 53 MB; Pipeline: 50.7M allocs, 4.68 GB allocated, peak live 98 MB

Allocations (decode pass, decode sample): v0.23.1: 12.0M allocs, 0.64 GB allocated; Pipeline: 12.0M allocs, 0.64 GB allocated

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Decode	allocs/MB	Δ base	Ids
amh_Ethi	lang	6.2	25.6	×4.12	×0.93	425,717 (×6.45)	×1.16	match
arb_Arab	lang	5.0	20.1	×4.03	×0.92	562,972 (×7.24)	×1.11	match
cmn_Hani	lang	10.1	17.5	×1.73	×0.94	933,157 (×1.44)	×1.11	match
eng_Latn	lang	2.7	9.8	×3.57	×0.96	856,308 (×7.71)	×1.05	match
hin_Deva	lang	6.6	27.8	×4.24	×0.93	208,681 (×15.21)	×1.16	match
jpn_Jpan	lang	10.3	16.2	×1.58	×0.95	825,891 (×1.52)	×0.94	match
rus_Cyrl	lang	4.2	14.1	×3.36	×0.98	962,881 (×4.39)	×1.11	match
tam_Taml	lang	8.6	26.5	×3.07	×0.93	484,771 (×4.28)	×1.09	match
tha_Thai	lang	10.2	17.9	×1.75	×0.94	769,749 (×1.71)	×1.00	match
added_normalized_sparse	modalities	4.9	19.6	×4.01	×0.98	130,242 (×49.37)	×1.05	match
added_special_sparse	modalities	5.0	21.3	×4.23	×0.97	132,882 (×39.65)	×1.13	match
agentic-traces	modalities	3.0	10.4	×3.51	×0.97	1,041,900 (×5.73)	×1.13	match
agentic_swe	modalities	3.5	12.1	×3.44	×0.96	863,676 (×6.10)	×1.07	match
code_mixed	modalities	3.2	10.9	×3.45	×0.97	994,914 (×5.88)	×1.09	match
math_latex	modalities	2.9	10.4	×3.61	×0.97	890,581 (×7.35)	×1.11	match

HuggingFaceDocBuilderDev · 2026-08-10T09:04:43Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

SBrandeis added 2 commits August 10, 2026 10:46

fix: decoder added token fix

c7f99bd

chore: PipelineToken type

58a0b70

SBrandeis changed the base branch from main to feat/train_encode_split August 10, 2026 09:01

remove stages, rename encode_generic -> encode_sequence

5bd0663

SBrandeis changed the title ~~Chore/cleanup~~ chore: cleanup in preparation for 1.0.0rc0 Aug 10, 2026

SBrandeis added 2 commits August 10, 2026 12:19

rm tokenize_spans and reuse scratch pool for pre-tokens

e4486f8

more thorough testing fot the ScratchPool

b27431f

SBrandeis force-pushed the chore/cleanup branch 2 times, most recently from 5ad9a2b to da313a2 Compare August 10, 2026 12:47

SBrandeis added the run-pipeline-bench label Aug 10, 2026

github-actions Bot removed the run-pipeline-bench label Aug 10, 2026

SBrandeis added 4 commits August 10, 2026 15:43

bump dependency: wide

a39cd5c

pretok: remove thread local, use the scratch pool

310547f

Sequence: use scratch

ce867ba

simplify metaspace

6d64f78

SBrandeis force-pushed the chore/cleanup branch from 47e0968 to 6d64f78 Compare August 10, 2026 13:44

unsafe hygiene

ae449f9

SBrandeis added the run-pipeline-bench label Aug 10, 2026

github-actions Bot removed the run-pipeline-bench label Aug 10, 2026

SBrandeis added 3 commits August 10, 2026 16:50

clippy

6f3fde6

rewrite queue-based BPE merge

4c7c337

wordcache: logs

f82130b

SBrandeis added the run-pipeline-bench label Aug 10, 2026

github-actions Bot removed the run-pipeline-bench label Aug 10, 2026

SBrandeis changed the title ~~chore: cleanup in preparation for 1.0.0rc0~~ chore: first round of cleanup in preparation for 1.0.0rc0 Aug 11, 2026

SBrandeis marked this pull request as ready for review August 11, 2026 08:55

SBrandeis merged commit 62a8976 into feat/train_encode_split Aug 11, 2026
56 of 62 checks passed

SBrandeis deleted the chore/cleanup branch August 11, 2026 11:14

Navigation Menu

Uh oh!

chore: first round of cleanup in preparation for 1.0.0rc0 - #2328

chore: first round of cleanup in preparation for 1.0.0rc0#2328
SBrandeis merged 13 commits into
feat/train_encode_splitfrom
chore/cleanup

SBrandeis commented Aug 10, 2026 •

edited by github-actions Bot

Loading

Uh oh!

HuggingFaceDocBuilderDev commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

SBrandeis commented Aug 10, 2026 • edited by github-actions Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PipelineTokenizer benchmark

Uh oh!

HuggingFaceDocBuilderDev commented Aug 10, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SBrandeis commented Aug 10, 2026 •

edited by github-actions Bot

Loading