Skip to content

Pull requests: huggingface/tokenizers

New pull request New
134 Open 1,016 Closed
134 Open 1,016 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

feat: wire in the thread pool
#2331 opened Aug 10, 2026 by McPatate Member Loading…
1 task
5
chore: first round of cleanup in preparation for 1.0.0rc0
#2328 opened Aug 10, 2026 by SBrandeis Contributor Loading…
1
fix(bpe): stop merging pairs whose count has gone negative
#2327 opened Aug 9, 2026 by yentur Loading…
1
Add missing return type hints to ByteLevelBPETokenizer.from_file, train, train_from_iterator
#2326 opened Aug 9, 2026 by RudrenduPaul Loading…
Add missing return type hints to CharBPETokenizer.from_file, train, train_from_iterator
#2325 opened Aug 9, 2026 by RudrenduPaul Loading…
perf(vocab): pack words up to 15 bytes exactly, in two tiers
#2324 opened Aug 7, 2026 by ArthurZucker Collaborator Loading…
1
perf(bpe): hand the splitter's spans to the model in tiles
#2323 opened Aug 7, 2026 by ArthurZucker Collaborator Loading…
1
perf(bpe): Last missing nits to get up to speed
#2313 opened Aug 7, 2026 by ArthurZucker Collaborator Loading…
3
perf: simd literal matching
#2312 opened Aug 6, 2026 by SBrandeis Contributor Loading…
1
[BUG] | The docs listing page for tokenizers is throwing 404
#2311 opened Aug 6, 2026 by xiaoyaoqilan Loading…
1
TARGET POC: bitsplit run-pipeline-bench target
#2306 opened Aug 6, 2026 by ArthurZucker Collaborator Loading…
1
TARGET MINIMAL RUST CRATE + .tok FORMAT target
#2293 opened Aug 5, 2026 by ArthurZucker Collaborator Loading…
1
Make the encode pool opt-out behind a parallel feature
#2289 opened Aug 5, 2026 by ArthurZucker Collaborator Loading…
1
feat(node): expose the pipeline encode path over napi
#2281 opened Aug 4, 2026 by ArthurZucker Collaborator Loading…
1
feat(atomsplit): run deepseek's pre-tokenizer with no regex backend
#2265 opened Jul 29, 2026 by SBrandeis Contributor • Draft
1
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245 opened Jul 27, 2026 by weixlu Loading…
3
Update fancy-regex dependency to 0.18
#2243 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
1
Update monostate dependency to v1
#2242 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
1
Fix panic in Strip decoder on short all-content tokens
#2239 opened Jul 24, 2026 by v-code01 Loading…
feat(pipeline+bindings): Support encoding pairs of sequences
#2237 opened Jul 23, 2026 by SBrandeis Contributor • Draft
2
implement ultra-basic, erroneous decode
#2232 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
feat(bindings): Python bindings rework, plugging PipelineTokenizer
#2229 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
2
Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
ProTip! Add no:assignee to see everything that’s not assigned.