Hi maintainers,
I've been building a Zig 0.16 reimplementation of the huggingface/tokenizers JSON format (SMC17/tokenizers-zig) primarily as a substrate exercise — coverage of BPE / WordPiece / Unigram against the same tokenizer.json files your Rust path consumes, with the same canonical token outputs on real-tokenizer fixtures (BERT, TinyLlama, T5).
On a 123-byte BERT-base-uncased input, our Zig path is measuring ~15 μs/encode against ~82 μs/encode for tokenizers 0.21 on the same hardware (consumer x86_64-linux, ReleaseFast, median of N=100). Bench harness lives at bench/bench_rust_bert/ in the repo above and is re-runnable against any installed Python wheel for verification.
Two methodology questions before I'd interpret this as a real finding rather than a fixture artifact:
-
Python boundary cost. Is the per-call Python boundary considered fixed-cost overhead by the maintainers, or in-scope for future optimisation? At 123-byte inputs we'd expect that boundary to dominate proportionally.
-
Bench harness fairness. The Rust path is doing more than ours — TemplateProcessing wrap + Python boundary + lifetime management for the returned Encoding. Is there an existing way (or appetite for a PR) to expose an encode_no_template variant on the Rust side so a fair apples-to-apples bench could exist for pre_tokenizer + model + post_processor without the wrap?
If question 2 is "yes, would accept a PR" I'd happily file one. Either answer to question 1 helps me interpret the existing numbers correctly.
Not asking you to merge or endorse our reimplementation — just asking whether the methodology gap is interesting enough that the mainline bench harness should expose a no-wrap variant.
Reference:
- Reimplementation (Zig, AGPL-3.0): https://github.com/SMC17/tokenizers-zig
- Bench harness: same repo,
bench/bench_rust_bert/
- Per-family head-to-head numbers (BPE / WordPiece / Unigram) in the README
Thank you for the canonical implementation — every other library in this space owes you the JSON schema.
Hi maintainers,
I've been building a Zig 0.16 reimplementation of the
huggingface/tokenizersJSON format (SMC17/tokenizers-zig) primarily as a substrate exercise — coverage of BPE / WordPiece / Unigram against the sametokenizer.jsonfiles your Rust path consumes, with the same canonical token outputs on real-tokenizer fixtures (BERT, TinyLlama, T5).On a 123-byte BERT-base-uncased input, our Zig path is measuring ~15 μs/encode against ~82 μs/encode for
tokenizers 0.21on the same hardware (consumer x86_64-linux, ReleaseFast, median of N=100). Bench harness lives atbench/bench_rust_bert/in the repo above and is re-runnable against any installed Python wheel for verification.Two methodology questions before I'd interpret this as a real finding rather than a fixture artifact:
Python boundary cost. Is the per-call Python boundary considered fixed-cost overhead by the maintainers, or in-scope for future optimisation? At 123-byte inputs we'd expect that boundary to dominate proportionally.
Bench harness fairness. The Rust path is doing more than ours — TemplateProcessing wrap + Python boundary + lifetime management for the returned
Encoding. Is there an existing way (or appetite for a PR) to expose anencode_no_templatevariant on the Rust side so a fair apples-to-apples bench could exist forpre_tokenizer + model + post_processorwithout the wrap?If question 2 is "yes, would accept a PR" I'd happily file one. Either answer to question 1 helps me interpret the existing numbers correctly.
Not asking you to merge or endorse our reimplementation — just asking whether the methodology gap is interesting enough that the mainline bench harness should expose a no-wrap variant.
Reference:
bench/bench_rust_bert/Thank you for the canonical implementation — every other library in this space owes you the JSON schema.