chore(deps): bump tiktoken-rs from 0.11.0 to 0.12.0 in /src-tauri in the cargo-majors group

[dependabot]

Bumps the cargo-majors group in /src-tauri with 1 update: tiktoken-rs.

Updates tiktoken-rs from 0.11.0 to 0.12.0

Release notes

Sourced from tiktoken-rs's releases.

v0.12.0

Summary

This release backports OpenAI tiktoken 0.13.0 into tiktoken-rs. The main reason to upgrade is better alignment with upstream tokenization behavior, especially the upstream Rust core changes for large BPE pieces and error-aware encoding.

For most users who call the high-level model/token counting helpers, this should behave the same aside from the new Rust compiler requirement. Users who call lower-level CoreBPE encoding methods directly should review the breaking changes below.

What Changed

• Backported the vendored OpenAI tiktoken Rust core from 0.9.0 to 0.13.0.
• Added the upstream large-piece BPE merge path. Functionally, this improves behavior for very large or repetitive inputs that previously stressed the merge algorithm.
• Changed CoreBPE::encode to return Result<(Vec<Rank>, usize), EncodeError>, matching upstream. Regex/tokenization failures can now be reported instead of being hidden behind infallible APIs.
• Updated encode_as and count to return Result because they call encode.
• Re-exported EncodeError so callers can handle encode failures directly.
• Aligned the vendored core with Rust 2024 and raised the crate MSRV to Rust 1.85.
• Synced model-to-tokenizer mappings with upstream tiktoken 0.13.0 while keeping local extra prefixes isolated.
• Hardened asset downloads with SHA-256 checks and a repo-root-aware asset path.

Breaking Changes

If your code calls CoreBPE::encode, unwrap or propagate the result before using the tokens:

let allowed = bpe.special_tokens();
let (tokens, last_piece_token_len) = bpe.encode("hello <|endoftext|>", &allowed)?;

The generic helpers changed similarly:

let (tokens, last_piece_token_len) = bpe.encode_as::<usize>(text, &allowed)?;
let token_count = bpe.count(text, &allowed)?;

encode_ordinary, encode_ordinary_as, encode_with_special_tokens, and count_ordinary remain infallible.

Projects must now build with Rust 1.85 or newer.

Practical Impact

• Applications processing long repeated text should see more robust tokenization behavior.
• Code that only uses helpers like get_chat_completion_max_tokens, get_text_completion_max_tokens, bpe_for_model, or singleton tokenizer constructors should not need call-site changes.
• Code using low-level CoreBPE::encode, encode_as, or count needs a small migration to handle Result.

Links

• PR: zurawiki/tiktoken-rs#164
• Upstream tiktoken 0.13.0: https://github.com/openai/tiktoken/releases/tag/0.13.0
• Full changelog: zurawiki/tiktoken-rs@v0.11.0...v0.12.0

Commits

• 32de8dc Bump version minor to "0.12.0"
• ac7e2e8 Backport tiktoken 0.13.0 (#164)
• See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

---

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

• @dependabot rebase will rebase this PR
• @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
• @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
• @dependabot ignore <dependency name> major version will close this group update PR and stop Dependabot creating any more for the specific dependency's major version (unless you unignore this specific dependency's major version or upgrade to it yourself)
• @dependabot ignore <dependency name> minor version will close this group update PR and stop Dependabot creating any more for the specific dependency's minor version (unless you unignore this specific dependency's minor version or upgrade to it yourself)
• @dependabot ignore <dependency name> will close this group update PR and stop Dependabot creating any more for the specific dependency (unless you unignore this specific dependency or upgrade to it yourself)
• @dependabot unignore <dependency name> will remove all of the ignore conditions of the specified dependency
• @dependabot unignore <dependency name> <ignore condition> will remove the ignore condition of the specified dependency and ignore conditions

[github-actions]

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

OpenSSF Scorecard

Package	Version	Score	Details
cargo/tiktoken-rs	0.12.0	Unknown	Unknown

Scanned Files

• src-tauri/Cargo.lock