Record PLM tokenizer normalization blocker - #662
Merged
Conversation
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a “fail-closed” evidence path for GGUF tokenizer materialization, capturing a pinned PLM GGUF artifact whose embedded tokenizer behavior (llama.cpp qwen2 handling of decomposed Unicode) is provably incompatible with the official HuggingFace tokenizer, and ensuring the pipeline errors with a precise, evidenced reason instead of silently producing a mismatched tokenizer.
Changes:
- Introduces
GGUFTokenizerBlockerEvidenceand a small blocker registry, plus selection logic inmatching_tokenizer_evidence()that raises an explicit “materialization is blocked” error on an exact identity match. - Adds a pinned llama.cpp oracle JSON fixture + unit tests binding the blocker record to the fixture and asserting the fail-closed behavior.
- Extends generated/public GGUF docs to include a “Fail-closed tokenizer evidence” table.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/data/gguf_plm_qwen2_tokenizer_blocker.json | Adds pinned oracle fixture describing the normalization mismatch witness and case hashes. |
| src/mobius/integrations/gguf/_tokenizer_evidence.py | Adds blocker evidence dataclass + registry and enforces fail-closed behavior on exact identity matches. |
| src/mobius/integrations/gguf/_tokenizer_evidence_test.py | Adds tests validating blocker scoping, fixture binding, and the explicit error message path. |
| src/mobius/integrations/gguf/_docs.py | Renders a new fail-closed tokenizer evidence table into the generated GGUF API documentation. |
| docs/api/build_from_gguf.md | Updates published docs with the new fail-closed evidence section/table. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Pin the official PLM GGUF and tokenizer identities, preserve complete header/token closure evidence, and bind a multilingual/adversarial llama.cpp oracle to the pinned dispatch revision. Fail closed for the exact PLM artifact because the official NFC normalizer diverges from llama.cpp qwen2 behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d7199b2-384e-4547-8c01-1a71172ce681 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Commit every pinned llama.cpp tokenization result and derive the corpus, full-result, and per-mode hashes in network-free tests. Bind the corpus digest into the architecture-scoped blocker evidence so input changes cannot preserve qualification by count alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d7199b2-384e-4547-8c01-1a71172ce681 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Render fail-closed tokenizer evidence as compact entries so the generated support document remains within its enforced line budget after rebasing onto current main. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 0d7199b2-384e-4547-8c01-1a71172ce681 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-exact-plm-gguf
branch
from
August 27, 2026 05:31
5b650ea to
c0749be
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
PLM-Team/PLM-1.8B-Instruct-gguf@7bec6546983bcf0d99526c943580bd49e2237445and artifact SHA-256b38570ee56ebec82a1e9ef45ab408c0d8230ececef1d7f1b267c49cff35638b8PLM-Team/PLM-1.8B-Instruct@62d188c7d58843d7013d5b3ffe198db4487878600f3c77d24208f2ac0833668128cb2a00adcb7e6b4b4eedc6e4783e3ec7b41f5aand result SHA-25666513168812575ccac974ecb454e916def5f4492d558c1866b8811d4f587a41dto llama.cpp8d9af256337d1a501250f9bbf4c0859a654bddd6éto token 963, while pinned llama.cpp preserves it as tokens 68 and 53839Tests
python -m pytest src/mobius/integrations/gguf/_tokenizer_evidence_test.py src/mobius/integrations/gguf/_docs_test.py -q --tb=short(36 passed)python -m pytest tests/build_graph_test.py tests/cli_test.py src/ -q -k "not phi4mm and not apply_weights_unknown" --tb=short(8,471 passed, 62 skipped, 12 deselected)lintrunner f --output oneline --all-filesReview
GPT-5.6 Sol, medium reasoning: final review found no code/evidence issue; its sole observation was to commit the already validated concise generated-doc changes, now included in
c0749bef.