Promote first low-cost GGUF runtime evidence batch - #681
Merged
Conversation
Performance Comparison
|
Contributor
There was a problem hiding this comment.
Pull request overview
Promotes the first “low-cost” GGUF runtime-evidence batch by wiring six GGUF architectures into the ORT GenAI evidence pipeline (explicit-float only), and updating the runtime/type plumbing and documentation so the new evidence is treated as runtime-supported in the GGUF registry/capability surfaces.
Changes:
- Add six promoted real-artifact ORT GenAI runtime-evidence cases (GPT-2, GPT-NeoX/Pythia, MPT, OLMo, StarCoder, StarCoder2) with explicit cache-vs-full-sequence parity checks and tokenizer identity assertions.
- Register/bind the new evidence across GGUF runtime-evidence records, batch manifest + capability JSON, architecture registry runtime support flags, and route census counts.
- Normalize GPT-2 ORT GenAI packaging to the generic decoder model type (since the exported cache ABI is the generic separate K/V form), and document the new batch in generated GGUF docs.
Reviewed changes
Copilot reviewed 26 out of 26 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gguf_small_model_runtime_integration_test.py | Adds six promoted runtime-evidence cases, reference loading for low-cost artifacts, cache parity tolerances, and stricter tokenizer identity checks. |
| testdata/evidence/gguf_quantization_capabilities.json | Updates selected-artifact totals and appends new runtime-evidenced artifacts/records. |
| testdata/evidence/gguf_low_cost_runtime_batch.json | New machine-readable manifest describing the selected low-cost batch, budget totals, exclusions, and fail-closed reasons. |
| testdata/cases/causal-lm/gpt2-q2-k.yaml | New network-free case fixture for the GPT-2 low-cost runtime-evidence route. |
| testdata/cases/causal-lm/pythia-70m-q2-k.yaml | New network-free case fixture for GPT-NeoX/Pythia low-cost runtime-evidence route. |
| testdata/cases/causal-lm/tiny-mpt-q2-k.yaml | New network-free case fixture for MPT portable/default-graph runtime-evidence route. |
| testdata/cases/causal-lm/tiny-olmo-q2-k.yaml | New network-free case fixture for OLMo low-cost runtime-evidence route. |
| testdata/cases/causal-lm/tiny-starcoder-q2-k.yaml | New network-free case fixture for StarCoder low-cost runtime-evidence route. |
| testdata/cases/causal-lm/tiny-starcoder2-q2-k.yaml | New network-free case fixture for StarCoder2 low-cost runtime-evidence route. |
| src/mobius/integrations/ort_genai/auto_export.py | Removes GPT-2 from architecture-specific ORT GenAI types and documents why GPT-2 normalizes to generic decoder. |
| src/mobius/integrations/ort_genai/auto_export_test.py | Updates expectations so GPT-2 emits generic decoder type instead of specialized GPT-2 type. |
| src/mobius/integrations/gguf/_tensor_mapping.py | Updates comments to reflect Q/K norm mapping applicability to dense Qwen3 as well as MoE variants. |
| src/mobius/integrations/gguf/_tensor_mapping_test.py | Adds a focused unit test proving Qwen3 attention Q/K norm tensor name mapping. |
| src/mobius/integrations/gguf/_runtime_package.py | Allows “default” as an evidenced execution provider and maps it to CPU for ORT GenAI session options. |
| src/mobius/integrations/gguf/_runtime_package_test.py | Adds a unit test ensuring portable/default graphs still target ORT GenAI CPU session construction. |
| src/mobius/integrations/gguf/_runtime_evidence.py | Adds six new GGUFRuntimeEvidence records and includes them in the evidence registry. |
| src/mobius/integrations/gguf/_runtime_evidence_test.py | Adds a closure/budget test for the new low-cost batch manifest and evidence binding. |
| src/mobius/integrations/gguf/_route_census.py | Records PR dependency dispositions for #672 and #674 to keep evidence batching explicit. |
| src/mobius/integrations/gguf/_route_census_test.py | Updates census totals and reconciled PR dependency expectations. |
| src/mobius/integrations/gguf/_quant_capabilities_test.py | Updates artifact-count and byte-total assertions to match the expanded capabilities set. |
| src/mobius/integrations/gguf/_exact_legacy_decoder_test.py | Updates expected runtime support for exact-legacy decoder architectures now evidenced (gptneox/mpt). |
| src/mobius/integrations/gguf/_conventional_decoder_test.py | Updates expected runtime support for StarCoder now evidenced. |
| src/mobius/integrations/gguf/_docs.py | Documents the new low-cost batch in the generated GGUF documentation output. |
| src/mobius/integrations/gguf/_docs_test.py | Relaxes the doc line-count ceiling and asserts new runtime-evidence IDs are present. |
| src/mobius/integrations/gguf/_arch_registry.py | Marks the six architectures as runtime-supported with tight runtime_evidence_ids scoping; adds Qwen3 tensor recipe for Q/K norms. |
| docs/api/build_from_gguf.md | Regenerates the rendered GGUF docs: updated runtime counts, evidence table, and recent PR dependency line. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
justinchuby
force-pushed
the
justinchuby-promote-low-cost-gguf-runtime
branch
2 times, most recently
from
August 27, 2026 18:21
7fa755d to
bc00350
Compare
Promote exact-artifact explicit-float ORT GenAI routes for GPT-2, GPT-NeoX, MPT, OLMo, StarCoder, and StarCoder2. Bind each route to immutable artifact, tokenizer, graph, package, full-logit, state, and deterministic-generation evidence while preserving fail-closed routing for all unmatched sources. Add portable CPU package support for evidenced default graphs, use the generic decoder ABI for GPT-2, close dense Qwen3 Q/K norm mapping, and update the network-free batch manifest, capability matrix, docs, and regression coverage. Production revision routing remains unchanged and revision-neutral. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: e2a8f3cf-109b-47c5-bc61-0a91ffd27ebf Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Map GPT-2 and StarCoder2 to their tanh-GELU defaults, tighten parity with activation-drift mutations, and replace circular reference loading with independent upstream-GGUF oracles. Reconcile route census and runtime documentation with current repository state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f5f288c7-b4e5-4371-ad46-9d163511ade9 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
Regenerate evidence fingerprints against the fully landed architecture registry, preserve truthful explicit-float runtime claims, and reconcile the route census. Enable dense Qwen3 Q/K norm consumers so its deferred tensor-closure record remains accurate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f5f288c7-b4e5-4371-ad46-9d163511ade9 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
justinchuby
force-pushed
the
justinchuby-promote-low-cost-gguf-runtime
branch
from
August 28, 2026 03:25
bc00350 to
951517b
Compare
Apply StarCoder2's architecture-owned 4096-token sliding window when GGUF metadata omits it. Add exact-artifact cached-position parity that rejects full causal attention, and refresh the config, graph, and runtime-package fingerprints. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f5f288c7-b4e5-4371-ad46-9d163511ade9 Signed-off-by: Justin Chu <justinchuby@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
revision=Noneremains unpinned and arbitrary explicit revisions continue to forward unchanged; immutable revisions appear only in tests/evidenceSelected artifacts
4234545f917ec1df10dab4d926796a83422b68e9010d85a4c111b8b541f328928e331c8c8016bed8ff1863b78fafe51e86b2364f32c2d7f3e201687e081cf7f75627dcb0ff18f6f7200f83c0aed2056a6a7c86b5f2d865833e1b5f00b00e4daabe1c5a22ac0e75cd5874467ffd80bcd2c8500609d3bf7ccdb5b269373e4d6da4aa8c2170bb9172447baba14309916cfc0d901dbffaf10f1448f4f631e10c1f41ab0a4b4e79c906520808db065a00a317c8b097e2176638b34088975eada6e0edArtifact payload is 334,238,976 bytes; tokenizer assets add 12,586,075 bytes, for 346,825,051 downloaded bytes total, below the 16 GiB policy ceiling.
Fail-closed outcomes
TOKENIZER_CHAT_TEMPLATE_IDENTITY_MISMATCHFULL_LOGIT_PARITY_DIVERGENCETENSOR_SHAPE_CLOSURE_FAILEDTOKENIZER_COMPILED_SEMANTICS_UNAVAILABLEINVALID_GGUF_HEADERRuntime evidence
quantization_report.jsonroundtrip identityValidation
auto_export.pyerrors outside changed lines; repository-wide mypy remains a known non-clean baseline