Prove GGUF quantized runtime capabilities - #661
Merged
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Performance Comparison
|
justinchuby
force-pushed
the
justinchuby-audit-remaining-dense-gguf
branch
from
August 27, 2026 05:20
0037a86 to
ab69b56
Compare
Base automatically changed from
justinchuby-audit-remaining-dense-gguf
to
main
August 27, 2026 05:22
Generate an exhaustive qtype-by-role capability matrix that separates parsing, exact dequantization, affine conversion, byte preservation, and evidenced execution. Reclassify Q4_K and Q6_K as lossy dequantize/requantize routes, retain runtime support only for the pinned Q8_0 CPU ABI, and add synthetic ORT kernel, transform, package, and real-artifact evidence gates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Separate supported packed target storage from source fidelity for lossy GGUF routes, retain Q4_K and Q6_K as affine target conversions, and parse runtime import routes as strict duplicate-free JSON before accepting preservation evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: da8e7ec7-2747-41aa-99eb-05a1c5227ce2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Bind every pinned graph and runtime package to its post-#668 quantization_report.json identity. Validate report save/load equality, qualify lossy Q4_K_M-derived INT4 target storage separately from source fidelity, and retain exact Q8/full-state runtime evidence. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: da8e7ec7-2747-41aa-99eb-05a1c5227ce2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Release ORT GenAI objects before interpreter teardown so the complete pinned runtime-evidence group exits cleanly, and retain the generated-document size guard after the latest tokenizer census landed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: da8e7ec7-2747-41aa-99eb-05a1c5227ce2 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-prove-gguf-quantization-runtime
branch
from
August 27, 2026 05:47
b29986c to
4c4d0bd
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR strengthens Mobius’ GGUF quantization claims by (1) introducing a machine-readable capability/evidence matrix for all stored qtypes and tensor roles, and (2) tightening the definition of “runtime supported” to require concrete, preserved-quantization, full-stateful, CPU execution evidence. It fits into the mobius.integrations.gguf subsystem by making quantization support and runtime proof auditable, reproducible, and test-closed.
Changes:
- Add a generated, canonical JSON capability matrix (
testdata/evidence/gguf_quantization_capabilities.json) and closure tests to ensure it exactly matches live registries/evidence. - Introduce qtype-level runtime evidence IDs (currently only
Q8_0) plus stricter validation ofpreserve_quantizationroutes (duplicate-free JSON + booleantrue). - Persist and validate
quantization_report.jsonin evidenced graph/runtime packages; update docs generation and CI runtime pinning accordingly.
Reviewed changes
Copilot reviewed 14 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gguf_small_model_runtime_integration_test.py | Extends integration assertions to require persisted quantization_report.json and validates lossy INT4 target-storage reporting vs explicit-float parity. |
| testdata/evidence/gguf_quantization_capabilities.json | Adds the generated, machine-readable stored-qtype capability/evidence matrix (large canonical JSON). |
| src/mobius/integrations/gguf/_spec.py | Adds runtime_evidence_ids to quant specs and validates they only accompany runtime=SUPPORTED. |
| src/mobius/integrations/gguf/_runtime_evidence.py | Expands evidence records (EP/ORT version/result), adds iter_runtime_evidence(), and enforces strict preserved-quantization evidence requirements for qtype support. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Adds coverage for quantized runtime evidence validation and strict import-route JSON requirements. |
| src/mobius/integrations/gguf/_repacker_test.py | Adds ORT round-trip + execution checks for exact affine ABI routes (MatMulNBits) using profiling evidence. |
| src/mobius/integrations/gguf/_quant_registry.py | Introduces qtype-level runtime support mapping (Q8_0 only) and updates the rendered quantization matrix columns/content. |
| src/mobius/integrations/gguf/_quant_registry_test.py | Tightens runtime support expectations and asserts lossy target-storage routes are never treated as source-preserving. |
| src/mobius/integrations/gguf/_quant_capabilities.py | New: renders/checks the canonical capability matrix JSON derived from live registries + runtime evidence. |
| src/mobius/integrations/gguf/_quant_capabilities_test.py | New: closure tests ensuring the committed matrix is current, role-complete, budget-bounded, and consistent. |
| src/mobius/integrations/gguf/_docs.py | Enhances runtime evidence table with ORT/EP/result fields and links the new machine-readable capability matrix. |
| src/mobius/integrations/gguf/_docs_test.py | Ensures docs + capability matrix are both current; adjusts doc length constraint. |
| scripts/generate_gguf_support_docs.py | Updates generator/check to also emit/verify gguf_quantization_capabilities.json. |
| docs/api/build_from_gguf.md | Regenerates docs to include updated runtime-evidence table and the expanded stored-qtype matrix. |
| .github/workflows/ort_genai_e2e.yml | Pins onnxruntime==1.29.0 alongside onnxruntime-genai==0.15.2 for the e2e workflow. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This was referenced Aug 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to merged #659 and rebased onto current
main(cc3647ea).testdata/evidence/gguf_quantization_capabilities.jsonfor all 25 stored qtypes and every projection, affine-only, output, embedding, expert-major, and non-MatMul routesource_fidelity=false; dequantize/requantize is never labeled preservedpkg.nxrt::BlockQuantizedMatMul/v1execution deferred; byte ABI compatibility is not execution evidencepreserve_quantizationto be exactly booleantruequantization_report.jsonin every evidenced graph/runtime packagePinned artifacts
Selected total: 1,763,532,768 bytes (1.64 GiB), below the 16 GiB policy.
Qwen/Qwen2.5-0.5B-Instruct-GGUF@9217f5db79a29953eb74d5343926648285ec7e67/qwen2.5-0.5b-instruct-q8_0.ggufca59ca7f13d0e15a8cfa77bd17e65d24f6844b554a7b6c12e07a5f89ff76844eunsloth/SmolLM2-135M-Instruct-GGUF@9e6855bc4be717fca1ef21360a1db4b29d5c559a/SmolLM2-135M-Instruct-Q4_K_M.ggufed5fa30c487b282ec156c29062f1222e5c20875a944ac98289dbd242e947f747source_fidelity=false; explicit float import passed same-artifact full-logit prefill/decode and deterministic generation; runtime support remains deferredneopolita/smollm-135m-gguf@22cca988936eafe92908e7558907c3964e10bba7/ggml-model-f16.ggufec8c775c16944a7e4b5251f97b3f848500dcc3e701b0d492ce9055cea42138a2LiquidAI/LFM2-350M-GGUF@8fdc9d526b7ed346b19257551b05816c7912ecc2/LFM2-350M-F16.gguf379ffdcbf08147c0313f6f1ce7ff558a2bc935eda633f4b46c52347032419c42Post-#668 package identities
4b608b099fb17471f342c925c20173f297abd0f8456c9e96a11b1d044272d1ad57038e28e83a2b4251e334b6098f59b4344c7e526746e29d9fa42eeaebfcbddc15d0218a4b326648d514a98c4f073251c674312b1faa83bec9022cc91daa0a53240e5e374803c94efdb17eee39c09b0d3e9aed10b6d8b4e1c92e39918ea2155e5029bbfcdd8d1ae1d2b0ed9587cc288a68cb38bb7f92b9e10f9bf64a436b176227e4ebb4c0c8b6c01ee57fa7825f34c5ddadc8ca5dc0c75d989e4507d4dcdfdbef3816d4f93c7061fd4653248629f0439d6ad1c623bdd0f27d31fe0349cb3505Every graph and runtime identity includes validated
quantization_report.json; tests assert exact report save/load equality.Validation
Limitations
Do not merge directly.