Skip to content

[Fix] Preserve native BPE training with protected separators - #262

Open
baihchou8787 wants to merge 2 commits into
BUPT-GAMMA:mainfrom
baihchou8787:fix/graph-bpe-protected-native
Open

[Fix] Preserve native BPE training with protected separators#262
baihchou8787 wants to merge 2 commits into
BUPT-GAMMA:mainfrom
baihchou8787:fix/graph-bpe-protected-native

Conversation

@baihchou8787

@baihchou8787 baihchou8787 commented Aug 28, 2026

Copy link
Copy Markdown

Description

Fixes GraphBPE so protected component separators split BPE input sequences without forcing backend=cpp or backend=auto onto the Python training path. Explicit cpp now fails during fit when the native extension is unavailable.

The native bridge accepts an initial vocabulary floor so merge IDs cannot collide with protected token IDs excluded from training segments.

Checklist

  • The PR title starts with [Fix]
  • Changes are complete
  • All changes have test coverage
  • Code is well-documented
  • Examples are not affected
  • Related issue is referred in this PR

Changes

  • Segment protected tokens before BPE training and encoding.
  • Keep native training available for protected GraphTokenizer separators.
  • Add regressions for native dispatch and fail-fast behavior.

Verification

TL_BACKEND=torch /local/wrq/gammagl-graphtokenizer/bin/python -m pytest tests/transforms/test_graph_tokenizer.py tests/transforms/test_graph_bpe_cpp_install.py -q

Result: 36 passed.

@baihchou8787
baihchou8787 force-pushed the fix/graph-bpe-protected-native branch from 3bb5810 to a0d8004 Compare August 31, 2026 02:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant