Add Qwen4Exp multimodal pipeline - #657
Merged
Merged
Conversation
Performance Comparison
|
🏗️ Architecture Diff
No architecture changes detected. ✅ Legend: ⚪ No change · 🔵 Minor (attrs/inits) · 🟡 Moderate (nodes added/removed) · 🔴 Major (interface changed) |
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 26, 2026 18:00
2cd112a to
c24fe82
Compare
justinchuby
changed the base branch from
justinchuby-finish-qwen4exp-core
to
justinchuby-add-qwen4exp-gguf
August 26, 2026 18:00
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 26, 2026 18:10
c036046 to
a38afdc
Compare
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-gguf
branch
from
August 26, 2026 18:22
c1b734e to
1c09bee
Compare
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 26, 2026 18:34
a38afdc to
1eeb756
Compare
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-gguf
branch
4 times, most recently
from
August 27, 2026 05:58
f1a5be7 to
fbc3370
Compare
Build the Qwen3.8 Flash-Next composite as decoder, vision encoder, and embedding graphs. Reuse the source-identical no-DeepStack Qwen vision tower, keep image and video streams distinct, and expose PLE token IDs separately from fused embeddings with the complete recurrent, sparse-index, PLE, and four-axis position state ABI. Fail closed for unsupported vision variants and MTP state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Add pinned composite config and processor extraction, graph-derived ORT GenAI metadata for every heterogeneous state role, native four-axis position metadata, deterministic image/video mixer and decoder parity tests, and an immutable L2 case. Preserve text-only routing despite the multimodal parent architecture and document the released runtime limitation. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Require the source RGB channel geometry, pin L2 coverage for the architecture alias, and add the rejection check requested by final review. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Cast float32 processor patches at the BF16 vision boundary, keep the text position axis independent from M-RoPE-driven QSA, fail closed for unsupported ORT GenAI packaging, and embed an explicit role-to-layer state manifest. Preserve exact pinned processor constants and add pinned HF, ORT load, packaging, and parity regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Describe text-only and multimodal position state separately, and cover channel-zero invariance across cached Mobius and pinned HuggingFace decode. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reject both text and multimodal packages before writing artifacts because OGA cannot represent their heterogeneous state contract. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Remove duplicate registry and synthetic-parity entries after restacking on the final core and GGUF heads, and normalize the combined test imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Exclude the multimodal parent during Qwen4Exp text-only config conversion and cover the complete builder path so vision state cannot leak into the text package. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-add-qwen4exp-multimodal
branch
from
August 27, 2026 06:20
1eeb756 to
822335e
Compare
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Pull request overview
This pull request adds full Qwen3.8 Flash-Next / Qwen4Exp multimodal export support to Mobius by introducing a three-model VLM package (decoder, vision_encoder, embedding), wiring it through the registry + Transformers builder, and extending metadata/export integrations to correctly model Qwen4Exp’s heterogeneous cache/state (including four-axis positions and per-layer role membership).
Changes:
- Add a Qwen4Exp vision-language task that builds a 3-model package and emits explicit
mobius.state_manifestmetadata for direct ORT orchestration. - Extend the Qwen4Exp model implementation with a multimodal decoder variant (fused
inputs_embeds+ independentple_input_ids), plus a vision encoder boundary cast for BF16/FP16 checkpoints. - Update integrations (Transformers builder, ORT GenAI export fail-closed behavior, onnx-genai inference metadata) and add/adjust targeted test + testdata coverage.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/_test_configs.py | Adds tiny composite Qwen4Exp VL configs (vision + token IDs) and adjusts test parametrization. |
| testdata/cases/vision-language/qwen3_8-flash-next.yaml | Adds an L4/L5 vision-language case record for Qwen3.8 Flash-Next (CI-skipped). |
| src/mobius/tasks/_qwen4_exp.py | Refactors Qwen4Exp decoder build, adds VL task, and emits mobius.state_manifest. |
| src/mobius/tasks/init.py | Exposes and registers the new qwen4-exp-vision-language task. |
| src/mobius/models/qwen4_exp.py | Adds multimodal decoder + vision encoder boundary cast + composite wrapper module. |
| src/mobius/models/qwen4_exp_test.py | Adds focused tests for 3-model IO, ordering, state manifest, processor contract, and parity invariants. |
| src/mobius/models/init.py | Exports Qwen4ExpForConditionalGeneration from the models package. |
| src/mobius/integrations/transformers/_builder.py | Allows multimodal parent architectures by default; ensures text_only=True ignores composite parent config. |
| src/mobius/integrations/transformers/_builder_test.py | Adds/updates tests for text-only resolution and parent-config exclusion. |
| src/mobius/integrations/ort_genai/auto_export.py | Improves state-kind detection and fails closed for Qwen4Exp ORT GenAI export; pins processor fallback constants. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Updates semantic past-input filtering expectations. |
| src/mobius/integrations/onnx_genai/inference_metadata.py | Extends state/position metadata to cover Qwen4Exp index_key + PLE roles and 4-axis positions. |
| src/mobius/integrations/onnx_genai/inference_metadata_test.py | Adds targeted metadata tests for Qwen4Exp 4-axis positions and state pairing. |
| src/mobius/_registry.py | Routes qwen4_exp / Qwen4ExpForConditionalGeneration to the new VLM task; keeps text_only mapping. |
| src/mobius/_configs/_vision_defaults.py | Extends vision-default extraction to include video + vision start/end token IDs and hidden_act. |
| src/mobius/_configs/_base.py | Extends Qwen4ExpConfig extraction/validation for the pinned composite vision tower and multimodal constraints. |
| docs/design/qwen4-exp-text-core.md | Updates design doc to describe the full multimodal pipeline and ORT GenAI limitations. |
Suppressed comments (2)
src/mobius/tasks/_qwen4_exp.py:147
- Same as above: using
assertforlayer_typesvalidation can be optimized out. Returning a clearValueErrormakes misconfigurations easier to diagnose in production builds.
assert config.layer_types is not None
src/mobius/tasks/_qwen4_exp.py:180
- Same as above: prefer an explicit exception over
assertfor required config fields so failures remain consistent even underpython -O.
assert config.layer_types is not None
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Stream decoder, embedding, and vision weights transactionally from one safetensors index with lazy payloads, deterministic PLE integrity checks, and clone-based fold rollback. Normalize the current multimodal package to explicit image-only support and fail closed for video capability and incomplete ORT/ONNX GenAI workflows. Restore three-component inspection and add loader memory, corruption, rollback, processor, graph, and dispatch regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Align the VL decoder override with the base forward contract, make layer-type validation explicit, and consolidate the public Qwen4Exp imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Reject unsupported video tokens inside the embedding graph before vocabulary semantics can leak, publish the unsupported-token contract, and prevent explicit config overrides or spoofed package metadata from bypassing ONNX GenAI preflight. Add processor-native mixed-input failure and output-snapshot regressions. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Replace CUDA's fail-open out-of-range Gather guard with a dynamic Reshape element-count violation shared by CPU and CUDA ORT. Preserve runtime dependency after optimization and cover valid plus processor-native mixed execution on CPU and CUDA when available. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
decoder,vision_encoder,embedding)ple_input_ids, fusedinputs_embeds, four-channel text + M-RoPE positions, and every heterogeneous recurrent/KV/QSA/PLE statemobius.state_manifestandmobius.unsupported_token_idscontracts for direct ONNX Runtime orchestrationBounded multimodal streaming
Both
qwen4_expandqwen4_exp_textuse the bounded safetensors path; the eager_download_weightsroute is unreachable. One shard index validates config-derived deterministic PLE buffers and plans decoder, embedding, and vision bindings. Each graph is cloned, lazily bound, and initializer-folded in isolation; the package is updated only after every staged component succeeds. Missing tensors, corrupt PLE hash buffers, or fold failures leave the original package untouched.Parameter payloads remain
LazyTensors and no source state dict is retained. The approximately 95 GiB PLE table is allocated once only when materialized/serialized and populated one checkpoint shard at a time. Loader accounting reports model/initializer/lazy counts, deterministic bytes validated eagerly, and zero retained source tensors.Enforced image-only boundary
The checkpoint's source video token is validated, then removed from normalized runtime metadata. Explicit configs requesting video capability fail closed and the embedding graph exposes no
video_featuresinput. It sanitizes any source video token before vocabulary lookup, computes a runtimehas_video, and reshapes one zero element to[1 + has_video]. Valid requests reduce/add zero; video requests demand two elements and fail ONNX Reshape's element-count invariant before an EP executes. ORT CUDA uses the sharedReshapeHelperfor this check (onnxruntime@b1f76d58,cuda/tensor/reshape.h), avoiding CUDA Gather's out-of-range zero-fill behavior. Optimized CPU is exercised locally and the same test executes CUDA when available; processor-native mixed input also fails through the guard.Runtime workflow boundary
onnxruntime-genai and the current ONNX GenAI workflow schema cannot represent Qwen4Exp's
ple_input_ids, four-axis positions, and heterogeneous state membership. Both exporters reject before writing artifacts. ONNX GenAI detection checks package config, explicit config, and the structural decoder ABI, so a non-Qwen override cannot bypass preflight or modify an existing output directory. Direct ONNX Runtime users consumemobius.state_manifestinstead.Source pins
Qwen/Qwen3.8-Flash-Next@f5d08274bafd880402bd16f5e3e6c514136ec06chuggingface/transformers@598d8ba8baaec7fec5a22da0e2844c7bf4ea20e1Final-base rebase
Lower-layer PR #656 was squash-merged to
mainas8f902ac2. This branch replays only #657's multimodal work onto that commit; no pre-squash lower-layer or GGUF implementation files remain in the PR diff.Validation
Waivers
This PR targets
main, is intentionally non-draft, and must not be merged by this session.