Prove Qwen3.5 MoE GGUF explicit-float runtime route - #672
Merged
Conversation
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR promotes a pinned reduced Qwen3.5-MoE GGUF artifact into the “promoted runtime evidence” suite, establishing an explicit-float CPU correctness route (including hybrid KV/conv/recurrent state semantics) and surfacing that structured evidence through the GGUF capability docs and registries.
Changes:
- Add a new promoted real-artifact runtime case for
qwen3.5-moe-0.87b-q2-k(explicit dequantize + dense-MoE opt-in) with deterministic generation evidence. - Extend the structured runtime evidence schema/output to include
source_fidelity,storage_quantized,target_storage_format,compute_mode, and optionallimitations. - Fix Qwen3.5 M-RoPE channel interleaving mapping by introducing/setting
mrope_interleavedindependently of flatrope_interleave.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/model_coverage_test.py | Updates coverage note for qwen3_5_moe to reflect the new bounded GGUF runtime fixture. |
| tests/gguf_small_model_runtime_integration_test.py | Adds promoted Qwen3.5-MoE GGUF runtime evidence case and same-value HF reference loader for parity. |
| testdata/evidence/gguf_quantization_capabilities.json | Adds the pinned Qwen3.5-MoE reduced artifact and runtime results to the evidence corpus. |
| testdata/cases/causal-lm/qwen3.5-moe-0.87b-q2-k.yaml | Introduces the new L4+L5 case definition for the reduced Qwen3.5-MoE GGUF artifact. |
| src/mobius/integrations/gguf/_runtime_evidence.py | Extends GGUFRuntimeEvidence with fidelity/storage/compute fields and optional limitations, adds Qwen3.5-MoE record. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Updates evidence record construction to satisfy the new schema fields. |
| src/mobius/integrations/gguf/_route_census_test.py | Adjusts census expected counts after reclassifying one architecture runtime status. |
| src/mobius/integrations/gguf/_quant_capabilities.py | Emits additional structured runtime evidence fields into the generated capabilities JSON. |
| src/mobius/integrations/gguf/_quant_capabilities_test.py | Updates artifact budget expectations and asserts new runtime evidence fields for the Qwen3.5 record. |
| src/mobius/integrations/gguf/_docs.py | Appends runtime evidence limitations text into the generated runtime evidence table output. |
| src/mobius/integrations/gguf/_docs_test.py | Adds qwen35moe runtime evidence pin coverage to the documentation tests. |
| src/mobius/integrations/gguf/_config_mapping.py | Sets mrope_interleaved for qwen35/qwen35moe during GGUF→config mapping. |
| src/mobius/integrations/gguf/_config_mapping_test.py | Updates/extends tests to assert mrope_interleaved behavior while keeping rope_interleave false. |
| src/mobius/integrations/gguf/_builder_test.py | Adds a fail-closed test to ensure default quantized routes don’t requantize source-float MoE experts. |
| src/mobius/integrations/gguf/_arch_registry.py | Promotes qwen35moe runtime from deferred to supported with explicit evidence scope and IDs. |
| docs/api/build_from_gguf.md | Updates generated API docs and tables to reflect the new runtime-supported architecture and evidence entry. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
justinchuby
force-pushed
the
justinchuby-prove-qwen35-moe-runtime
branch
from
August 27, 2026 09:41
80d8de7 to
fe08549
Compare
Pin the bounded Q2_K artifact and publish exact explicit-float CPU runtime evidence across ONNX Runtime and ORT GenAI. Cover full logits, hybrid state semantics, deterministic generation, tokenizer provenance, package identities, and fail-closed mixed quantization. Copilot-Session: ad2de961-a4dc-4ed1-b9c4-ac9a826214cb Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Materialize all floating reference parameters and buffers as float32 after loading the exact dequantized GGUF values. Copilot-Session: ad2de961-a4dc-4ed1-b9c4-ac9a826214cb Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-prove-qwen35-moe-runtime
branch
from
August 27, 2026 11:00
3cf8023 to
fd4e8b0
Compare
This was referenced Aug 27, 2026
justinchuby
added a commit
that referenced
this pull request
Aug 28, 2026
## Summary - promote exact-artifact ORT GenAI runtime support for six graph-supported GGUF architectures: GPT-2, GPT-NeoX/Pythia, MPT, OLMo, StarCoder, and StarCoder2 - bind every route to immutable source/config/tokenizer revisions, exact artifact identity, tensor/value closure, full logits, dynamic-cache replay/rollback/reorder, deterministic 20-token generation, tokenizer identity, and graph/package/report roundtrips - keep all six routes explicitly dequantized to float storage and float operators; this PR makes no native or source-faithful quantized-runtime claim - add network-free case fixtures and a machine-readable batch manifest, then regenerate the capability matrix and GGUF documentation - route GPT-2 through the generic decoder ABI, permit exact-evidenced portable/default graphs for CPU packages, and close dense Qwen3 Q/K norm tensor mapping - preserve the global revision policy: `revision=None` remains unpinned and arbitrary explicit revisions continue to forward unchanged; immutable revisions appear only in tests/evidence ## Selected artifacts | Architecture | Artifact bytes | SHA-256 | |---|---:|---| | GPT-2 | 81,196,544 | `4234545f917ec1df10dab4d926796a83422b68e9010d85a4c111b8b541f32892` | | GPT-NeoX/Pythia | 38,508,192 | `8e331c8c8016bed8ff1863b78fafe51e86b2364f32c2d7f3e201687e081cf7f7` | | MPT | 8,734,304 | `5627dcb0ff18f6f7200f83c0aed2056a6a7c86b5f2d865833e1b5f00b00e4daa` | | OLMo | 33,860,576 | `be1c5a22ac0e75cd5874467ffd80bcd2c8500609d3bf7ccdb5b269373e4d6da4` | | StarCoder | 103,899,456 | `aa8c2170bb9172447baba14309916cfc0d901dbffaf10f1448f4f631e10c1f41` | | StarCoder2 | 68,039,904 | `ab0a4b4e79c906520808db065a00a317c8b097e2176638b34088975eada6e0ed` | Artifact payload is **334,238,976 bytes**; tokenizer assets add **12,586,075 bytes**, for **346,825,051 downloaded bytes** total, below the 16 GiB policy ceiling. ## Fail-closed outcomes - Qwen3: `TOKENIZER_CHAT_TEMPLATE_IDENTITY_MISMATCH` - BLOOM: `FULL_LOGIT_PARITY_DIVERGENCE` - StableLM: `TENSOR_SHAPE_CLOSURE_FAILED` - Gemma/Gemma2/Gemma3/Phi3: `TOKENIZER_COMPILED_SEMANTICS_UNAVAILABLE` - Falcon: `INVALID_GGUF_HEADER` - Qwen3.5 MoE and Nemotron-H MoE remain excluded as recently handled by #672/#674; tokenizer-only and ABI-blocked route classes remain out of scope ## Runtime evidence - ONNX Runtime 1.29.0 / ORT GenAI 0.15.2 / CPU - exact tensor census and value closure - independent full-logit parity with per-route bounded tolerances - prefill, cached decode, full replay, rollback, reorder, and deterministic generation - tokenizer vocabulary/special-token identity - graph package and `quantization_report.json` roundtrip identity - all downloaded payloads and temporary ONNX packages removed after collection ## Validation - promoted real-artifact runtime cases: 6 passed - focused GGUF unit suite: 1,441 passed, 1 skipped - schema/dashboard/docs coverage: 1,135 passed, 255 skipped - broad serial non-integration suite: 8,626 passed, 64 skipped, 12 deselected - post-rebase focused suite: 413 passed - Ruff check/format and generated-doc check: passed - changed source type check: clean except six pre-existing `auto_export.py` errors outside changed lines; repository-wide mypy remains a known non-clean baseline - independent high-confidence review: no findings --------- Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com> Copilot-Session: e2a8f3cf-109b-47c5-bc61-0a91ffd27ebf Copilot-Session: f5f288c7-b4e5-4371-ad46-9d163511ade9
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Flexan/kshitijthakkar-qwen3.5-moe-0.87B-d0.8B-GGUF@a9b8adbec2cc87479c772dac1944f313b4036c26/qwen3.5-moe-0.87B-d0.8B.Q2_K.gguf(626,599,552 bytes, SHA-256e8a84df1a50ce65cf80c2b55bba8c6e80f913679fdf9e9439f2c3b52ef3145d5)Evidence
8c1aa1075cee03ffd5ce5bbd283ee88b2466e6c1d645f0a41e542230951d6f09bd69cf72acffd393a706bb012c23990acfdaf3a43e25ad919ef1148afac42a15[198, 198, 198, 321, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198, 198]Validation