tokenizer upgrade ? #50
-
|
which tokenizer is best to upgrade ? |
Beta Was this translation helpful? Give feedback.
Answered by
codeaddict-119
May 23, 2026
Replies: 1 comment
-
|
o200k_harmony (BPE) tokenizer with a vocabulary of 201,088 tokens tiktoken library. |
Beta Was this translation helpful? Give feedback.
0 replies
Answer selected by
Eamon2009
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
o200k_harmony (BPE) tokenizer with a vocabulary of 201,088 tokens tiktoken library.