Preserve pinned Qwen4Exp FP8 checkpoint with QDQ - #658
Merged
Conversation
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-integrate-qwen4exp-fp8
branch
from
August 26, 2026 18:21
266176a to
6d8c317
Compare
justinchuby
changed the base branch from
justinchuby-finish-qwen4exp-core
to
justinchuby-add-qwen4exp-multimodal
August 26, 2026 18:22
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 26, 2026 18:34
a38afdc to
1eeb756
Compare
justinchuby
force-pushed
the
justinchuby-integrate-qwen4exp-fp8
branch
from
August 26, 2026 19:02
6d8c317 to
6d23fa1
Compare
justinchuby
changed the title
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 27, 2026 06:20
1eeb756 to
822335e
Compare
Dequantize strict 128-by-128 inverse-scale grids without materializing expanded scale tensors, while preserving scalar scales. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Stream the immutable Qwen3.8 Flash-Next FP8 checkpoint into a dense ONNX fallback with strict 128x128 scale validation and explicit non-native reporting. Preserve PLE shards as separate graph initializers, classify composite visual and MTP keys, and record header-only schema evidence without downloading the 185.5 GB payload.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Validate every FP8 and inverse-scale header, including ignored MTP tensors, reject orphan scales and revision drift, and persist the dense fallback report with the package. Add preference, sidecar, report round-trip, and immutable revision tests.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Use an independent reduced-matrix reconstruction oracle, support upstream combined PLE tables alongside checkpoint shards, and pass pinned Transformers prefill/decode parity. Account for ONNX IR output-shard buffering by defaulting dense streaming packages to 1 GiB shards, rejecting limits above 5 GB, and reporting the real serialization memory bound.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Apply the pinned shared BF16 scalar to every FP8 PLE shard, validate its exact payload, and report MTP and visual exclusions separately. Reject duplicate safetensors keys, serialize dense fallbacks with one worker, correct QSA RoPE frequency expansion, and add pinned L2/L3/YAML coverage.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Preserve the multimodal registry and config tests while selecting the immutable FP8 text checkpoint for L2 validation. Remove duplicate metadata introduced by the restack and keep the combined core, multimodal, and FP8 surfaces lint-clean.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Require E4M3 PLE shards with the single pinned BF16 scalar, reject per-shard scale bypasses, and use in-place scalar reconstruction. Report the largest simultaneously live source+dense+scale working set so the single-worker serialization bound is accurate.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Restore the defaultdict import required by the inherited GGUF indexer split/fusion path after restacking FP8 on the updated core, GGUF, and multimodal branches.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reconstruct the pinned per-expert FP8 matrices one projection at a time into the final packed rank-3 graph initializers. Preserve the final core QSA implementation while deduplicating rebased registry and synthetic-parity metadata.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Validate every BF16 PLE graph shard after the FP8 integration changes the shared graph contract from one combined table to checkpoint-native shard initializers.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Select routed token coordinates per expert, run only those token rows through constant-index expert weights, and scatter weighted outputs back. This avoids materializing token-shaped copies of full expert matrices during prefill.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Make storage-preserving standard ONNX QDQ the default for the pinned FP8 checkpoint. Keep exact E4M3 code and BF16 scale initializers, reconstruct block tiles invertibly in-graph, retain dense BF16 reconstruction only behind explicit dequantization, and record runtime capability separately from storage fidelity.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Propagate the Transformers CLI quantization choice, prevalidate complete QDQ source and target closure transactionally, and account for simultaneous source and cast-destination buffers in serialization memory bounds. Strengthen failure regressions with complete graph snapshots.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Assert the persisted single-worker serialization bound incorporates planned source and cast-destination overlap for QDQ packages.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Record the required attribution metadata for the independent review-fix series. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Deduplicate build and CLI quantization controls against merged compressed-tensors support, preserve the dedicated FP8 QDQ route ahead of ordinary Qwen loading, and align the inherited BF16 streamer test with sharded PLE graph storage.\n\nCo-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>\nCopilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Queue dense bindings until the complete checkpoint plan validates, and preflight the persisted weight-loading report path before any package artifact is written. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-integrate-qwen4exp-fp8
branch
from
August 27, 2026 08:18
2c59cd0 to
797a830
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR adds a storage-preserving FP8 (F8_E4M3 + BF16 scale) import path for the pinned unsloth/Qwen3.8-Flash-Next-FP8@41cc25f… checkpoint by streaming safetensors shards and emitting standard ONNX QDQ to reconstruct logical weights, while keeping dense reconstruction only behind an explicit --dequantize / keep_quantized=False choice.
Changes:
- Add FP8 streaming weight-loading infrastructure in
mobius.integrations._weight_loadingwith a strict, fail-closed planning contract and a QDQ-preserving path (stream_qdq_safetensors_to_model) plus a dense fallback (stream_preprocessed_safetensors_to_model). - Extend Qwen4-Exp text model support to classify FP8 sources (including packed expert banks and PLE scalar scaling) and produce a detailed weight-loading report, with new evidence fixtures and regression tests.
- Persist
weight-loading-report.jsoninModelPackage.save()/load()and force serial external-data saving for streaming packages to keep memory bounds predictable.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/synthetic_parity_test.py | Adjust synthetic HF-config adaptation for Qwen4-Exp text. |
| tests/cli_test.py | Add CLI test asserting --dequantize toggles keep_quantized. |
| testdata/evidence/causal-lm/qwen3.8-flash-next-fp8-schema.json | New pinned header/schema evidence for the FP8 checkpoint. |
| testdata/cases/causal-lm/qwen3.8-flash-next-fp8.yaml | New (skipped) L4/L5 case documenting the pinned FP8 artifact and waiver. |
| src/mobius/models/qwen4_exp.py | Add FP8 header validation + streaming plan + sharded PLE embedding + routed-expert dispatch changes. |
| src/mobius/models/qwen4_exp_test.py | Add extensive FP8/QDQ/dense-fallback streaming tests and fixture validation. |
| src/mobius/integrations/transformers/_qwen4_exp_weights_test.py | Update streaming loader tests for sharded PLE embedding parameters. |
| src/mobius/integrations/transformers/_builder.py | Enforce pinned revision and route FP8 block-quant configs into QDQ/dense streaming loaders. |
| src/mobius/integrations/transformers/_builder_test.py | Add tests for loader selection and pinned revision enforcement. |
| src/mobius/integrations/_weight_loading.py | Implement streaming plan dataclasses, strict shard indexing, FP8 block dequant, and QDQ graph rewriting. |
| src/mobius/integrations/_weight_loading_test.py | Update generic FP8 dequant tests to fail closed on missing scales; add 2D-grid coverage. |
| src/mobius/integrations/_weight_loading_stream_test.py | Add regression ensuring duplicate tensor keys across shards fail closed. |
| src/mobius/_registry.py | Point qwen4_exp_text test model id/revision to the pinned FP8 checkpoint. |
| src/mobius/_model_package.py | Persist and validate weight-loading-report.json; force serial external-data saving for streaming packages. |
| src/mobius/_model_package_test.py | Add roundtrip + save-policy tests for the weight-loading report behaviors. |
| src/mobius/_configs/_base.py | Add allow_block_fp8_dense_fallback to allow parsing FP8 block schemes for Qwen4Exp. |
| docs/design/qwen4-exp-text-core.md | Document the pinned FP8 checkpoint route, QDQ recipe, and reporting semantics. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Parse the composite parent before stripping multimodal fields so its sole quantization_config reaches the Qwen4Exp text route. Cover default QDQ, explicit dense fallback, and unchanged multimodal config derivation without network or weights. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Inherit parent-only quantization metadata before stripping text-only multimodal fields. Replace per-expert and per-shard runtime expansion with constant-size fused MoE and one logical PLE gather, and cover both storage and multimodal routing contracts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Cast float32 routing probabilities to the model dtype before fused MoE, keeping graph size expert-count independent. Restore per-shard PLE lookup and serialize shard dequantization with explicit dependencies so only one dense shard is live at a time. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Describe the implemented full-shard DQ then Gather order and its code, dense, scale, and optional-cast peak. Gate full BF16 QDQ model load and inference only on an independent exact DequantizeLinear capability probe. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Skip BF16 FP8 QDQ validation only for ORT's exact NotImplemented class and canonical DequantizeLinear(24) missing-kernel signature. Re-raise matching ValueError text, near matches, wrong op versions, and subclasses. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e0bfa582-e6d5-4722-b02b-70b18f639ef2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
unsloth/Qwen3.8-Flash-Next-FP8@41cc25fe32cc20053a59c89716196897580cddf6sourceFLOAT8E4M3FNcodes and BF16 scales byte-for-byte0x3951)com.microsoft.MoE, and assemble packed rank-3 expert banks from independently scaled QDQ projections--dequantize/keep_quantized=False, with BF16-first semanticsquantization_configthrough text-only config parsing, then strip all multimodal fields before graph constructionstorage_preserving=true,native_fp8=false,runtime_execution_proven=falseBase integration
Based directly on current main, preserving merged truthful GGUF reports, compressed routes, runtime capabilities, Qwen4Exp GGUF/multimodal behavior, and tokenizer evidence. The main-relative diff is FP8-QDQ-only.
Validation
142 passed304 passed, 13 skipped9 passed8612 passed, 62 skipped, 12 deselected, 4 subtests passedRuntime and scope
Stock ORT may reject FLOAT8
DequantizeLinear(24)because no current kernel is available. This is recorded as a runtime capability gap and does not trigger dense serialization or invalidate the checker-valid storage artifact. Native FP8 compute is not claimed.Full 185.5 GB payload export and L4/L5 generation are waived; all 131 headers and the complete 152,089-tensor schema were validated without payload download, with independent reduced QDQ/dense and pinned-Transformers parity.
This PR remains text-route only for the FP8 checkpoint. Merged multimodal support supplies the architecture, but visual FP8 loading is not claimed. MTP remains excluded because no authoritative forward equation or NextN cache ABI exists.