Skip to content

Add missing return type hints to CharBPETokenizer.from_file, train, train_from_iterator - #2325

Open
RudrenduPaul wants to merge 1 commit into
huggingface:mainfrom
RudrenduPaul:fix/char-level-bpe-return-type-hints
Open

Add missing return type hints to CharBPETokenizer.from_file, train, train_from_iterator#2325
RudrenduPaul wants to merge 1 commit into
huggingface:mainfrom
RudrenduPaul:fix/char-level-bpe-return-type-hints

Conversation

@RudrenduPaul

Copy link
Copy Markdown

What this PR does

Adds missing return type hints to three methods in
bindings/python/py_src/tokenizers/implementations/char_level_bpe.py:

  • from_file(vocab_filename: str, merges_filename: str, **kwargs) -> "CharBPETokenizer"
  • train(...) -> None
  • train_from_iterator(...) -> None

This follows the same typing-consistency pattern as #2211, which added
the equivalent missing return hints to BaseTokenizer.save,
save_model, and to_str. CharBPETokenizer (a BaseTokenizer
subclass) had the identical gap in its own methods: every argument was
already annotated, but the return type was never added.

Why these types are correct

  • from_file is a @staticmethod that returns CharBPETokenizer(vocab, merges, **kwargs),
    i.e. a new instance of the class — hence -> "CharBPETokenizer", matching the quoted
    self-referential forward-reference convention already used elsewhere in this codebase
    (e.g. Tokenizer.from_file -> "Tokenizer" and BPE.from_file -> "BPE" in the generated
    .pyi stubs).
  • train and train_from_iterator both call self._tokenizer.train(...) /
    self._tokenizer.train_from_iterator(...) without a return statement, so they implicitly
    return None — hence -> None.

Scope note

Checked gh pr list --repo huggingface/tokenizers --search "char_level_bpe" before starting;
no open or closed PR currently touches this file.

No behaviour change

Type-hint-only change on method signatures. No logic, docstrings, or other files were modified.

Testing

  • ruff check bindings/python/py_src/tokenizers/implementations/char_level_bpe.py — all checks passed.
  • ruff format --check bindings/python/py_src/tokenizers/implementations/char_level_bpe.py — passes, file already formatted.
  • python3 -c "import ast; ast.parse(...)" on the modified file — passes.

Note: Claude Code was used to assist in drafting this change. All changes were reviewed by the submitter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant