diff --git a/.agents/skills/adding-a-new-model/SKILL.md b/.agents/skills/adding-a-new-model/SKILL.md index 9ec1c8887..0069c07c0 100644 --- a/.agents/skills/adding-a-new-model/SKILL.md +++ b/.agents/skills/adding-a-new-model/SKILL.md @@ -286,7 +286,8 @@ see the [quality-checklist skill](../quality-checklist/SKILL.md). - [ ] Integration test in `tests/integration_test.py` (if small checkpoint available) - [ ] L4 golden file generated and committed (`testdata/golden/`) - [ ] L5 generation golden file generated and committed -- [ ] ORT GenAI test added to `tests/ort_genai_test.py` (text-generation and VLM models) +- [ ] Graph-derived ORT GenAI metadata is tested; downstream load/generation is + optional evidence and never an export capability gate - [ ] CLI build works (`mobius build --model ...`) - [ ] Multi-dtype correctness verified (fp32, fp16, bf16) - [ ] Pinned revision reaches every Hub/processor/weight/golden call diff --git a/.agents/skills/diffusion-models/SKILL.md b/.agents/skills/diffusion-models/SKILL.md index 959de5499..1246d68de 100644 --- a/.agents/skills/diffusion-models/SKILL.md +++ b/.agents/skills/diffusion-models/SKILL.md @@ -363,8 +363,9 @@ Before advertising runtime support, execute the complete path: Verify sample/output ranks, source-vs-target token slicing, CFG semantics, and the runtime's actual scheduler identifiers/equations. Reload final metadata and verify every file path, port, preprocessing step, loop edge, and postprocess. -If the runtime cannot express the dataflow, reject it before emitting artifacts; -keep direct component export separate. +Emit faithful component graphs and metadata even when the tested downstream +runtime cannot yet express the complete dataflow. Record that runtime/version +limitation separately; it must not become a Mobius export capability gate. Validate required VAE statistics against latent channel count before graph construction (`len(mean) == len(std) == z_dim`, positive standard deviations). diff --git a/.agents/skills/multimodal-models/SKILL.md b/.agents/skills/multimodal-models/SKILL.md index 156dff435..d3beed9e2 100644 --- a/.agents/skills/multimodal-models/SKILL.md +++ b/.agents/skills/multimodal-models/SKILL.md @@ -255,9 +255,10 @@ must scale with rows/windows, not total-patches × media-count. Keep dense masks only as a portable fallback and assert optimized CUDA/DML graphs lack the quadratic construction. -Config generation is not runtime support. Execute real media through ORT GenAI; -if it cannot supply a required encoder input or position-ID rank, reject export -before writing artifacts and use the same evidence for any Foundry waiver. +Config generation is not runtime support. Emit graph-derived metadata faithfully +and, when useful, execute real media through ORT GenAI as a downstream probe. +If it cannot supply a required encoder input or position-ID rank, record the +exact runtime/version limitation without blocking Mobius export. ## GQA for KV-shared layers (Gemma4) diff --git a/.agents/skills/onnx-export-quantization/SKILL.md b/.agents/skills/onnx-export-quantization/SKILL.md index 5d533aaab..132293f14 100644 --- a/.agents/skills/onnx-export-quantization/SKILL.md +++ b/.agents/skills/onnx-export-quantization/SKILL.md @@ -151,8 +151,11 @@ Before implementing or claiming a direct conversion: pinned upstream tokenizer. A self-contained package is invalid when padding, EOS, or BOS semantics disagree. 7. Require real-weight full-logit parity and deterministic multi-token - generation through both ONNX Runtime and the declared GenAI runtime. Graph, - config, and session creation are not acceptance evidence. + generation directly through ONNX Runtime. Graph, config, and session creation + alone are not acceptance evidence. +8. Optionally run the same generation through the declared GenAI runtime as + downstream evidence. Record its version and outcome, but do not gate direct + GGUF artifacts on its acceptance. Use Hub GGUF architecture metadata to fail before downloading multi-gigabyte unsupported files, then repeat the guard from the local GGUF header so local @@ -242,6 +245,16 @@ pip install cupy-cuda12x # Olive auto-detects cupy and uses GPU when available ``` +### Isolate Olive from unrelated provider DLLs + +Olive 0.13 may auto-register every provider DLL bundled in an ORT GPU wheel +even when a weight-only pass explicitly targets CPU. A missing TensorRT DLL can +then abort K-quant before the pass starts. Keep the accelerator CPU-only and, +for programmatic workflows, suppress `olive.systems.local` EP-library +registration around `olive.workflows.run`; restore it immediately afterward. +This is safe for `OnnxKQuantQuantization`, which does not create an inference +session. Still load and execute the resulting package with the intended EP. + ### Quantizing multi-model exports Quantize each sub-model independently. Typically only the decoder is @@ -425,9 +438,9 @@ compare_golden( ) ``` -### L5: End-to-end smoke test +### Optional ORT GenAI downstream smoke test -Run inference with the quantized model through ORT GenAI: +When useful, run inference with the quantized model through ORT GenAI: ```python import onnxruntime_genai as og @@ -442,6 +455,10 @@ output_ids = model.generate(params) print(tokenizer.decode(output_ids[0])) ``` +ORT GenAI acceptance is not a Mobius export gate. Always validate the final +quantized ONNX package directly; treat ORT GenAI load/generation as optional +downstream evidence and record its version/outcome without blocking export. + ### Numerical parity verification Quantized models will have some numerical divergence from the diff --git a/.agents/skills/ort-genai-config/SKILL.md b/.agents/skills/ort-genai-config/SKILL.md index cb94def19..6ab46aa10 100644 --- a/.agents/skills/ort-genai-config/SKILL.md +++ b/.agents/skills/ort-genai-config/SKILL.md @@ -282,12 +282,14 @@ processor doesn't provide. Either: 1. Compute them externally and inject via NamedTensors, or 2. Modify the vision model to compute them from `image_grid_thw` internally -### Config writes successfully but runtime cannot execute - -Config/schema success is not runtime support. Run load plus generation through -the exported contract. If the runtime cannot route required feature inputs, -position-ID rank, cache state, scheduler, or multimodal metadata, reject export -before writing artifacts and report the exact runtime version/limitation. +### Config metadata and downstream runtime acceptance + +Mobius owns metadata correctness, not ORT GenAI capability decisions. Emit the +most accurate package from graph metadata: filenames, semantic graph inputs and +outputs, every cache template the current config schema can represent, and the +global cache-slot count. Preserve intrinsic schema/config validation, but do +not gate or reject export based on the current GenAI model registry, runtime +version, topology support, or cache executor capability. ### "input_ids size exceeds max length" diff --git a/.agents/skills/quality-checklist/SKILL.md b/.agents/skills/quality-checklist/SKILL.md index 9f5b6453e..f539eb275 100644 --- a/.agents/skills/quality-checklist/SKILL.md +++ b/.agents/skills/quality-checklist/SKILL.md @@ -3,10 +3,10 @@ name: quality-checklist description: > Use this skill when verifying that a new model is truly done and ready to merge. Provides a Definition-of-Done checklist covering all five test - confidence levels (L1 graph build through L5 Foundry Local smoke-test), - ORT GenAI runtime validation, Olive quantization compatibility, multi-dtype - (f32/f16/bf16) and multi-EP (CPU/CUDA/DML) correctness, documentation - requirements, and code review criteria. + confidence levels (L1 graph build through L5 generation), faithful runtime + metadata, optional downstream runtime probes, Olive quantization + compatibility, multi-dtype (f32/f16/bf16) and multi-EP (CPU/CUDA/DML) + correctness, documentation requirements, and code review criteria. --- # Skill: Quality Checklist @@ -14,8 +14,8 @@ description: > ## When to use Use this checklist before marking a new model addition as **done**. -Every item must be checked — or explicitly waived with a written reason — -before the PR is merged. +Every required item must be checked — or explicitly waived with a written +reason — before the PR is merged. --- @@ -148,12 +148,6 @@ python examples/_text_generation.py --compare-hf --dtype bf16 opt-in via `MOBIUS_ORT_LOWER_OPSET_FOR_EP=1`) - [ ] Dead graph inputs removed after EP-aware optimization (`RemoveDeadGraphInputsPass` in Stage 4 of `optimize_model()`) -- [ ] For models with dual head_dim (e.g. Gemma4: 256 for sliding, - 512 for full attention), verify whether ORT GenAI requires - `search.past_present_share_buffer=false` because GenAI allocates - uniform KV cache shapes; if so, explicitly override the generated - `genai_config.json` before runtime validation rather than assuming - the default generated setting is correct - [ ] Vision/audio graph inputs match the real processor (normally float32); reduced-precision encoders cast once at graph entry - [ ] Representative graph evidence covers raw/post-Mobius/post-weight and @@ -162,15 +156,29 @@ python examples/_text_generation.py --compare-hf --dtype bf16 - [ ] Final fusion claims use loaded weights/constants; no-weight graphs cannot prove initializer folding -### 9. ORT GenAI runtime +### 9. ORT GenAI metadata and downstream runtime -- [ ] Model can be loaded with `ort_genai.Model(output_dir)` without error -- [ ] Actual generation with required media/features produces coherent output; - schema/config emission alone is not runtime support -- [ ] ORT GenAI test added to `tests/ort_genai_test.py` - (or confirmed covered by an existing parametrized test) -- [ ] Structurally unsupported contracts fail before artifacts are emitted and - have a version-specific, evidence-based waiver +- [ ] Generated metadata faithfully reflects graph filenames, semantic I/O, + representable cache templates, and global cache-slot indices +- [ ] Intrinsic schema/config errors are tested without inferring downstream + runtime capability + +Downstream evidence: + +- [ ] If ORT GenAI load/generation is run, record the exact runtime version and + result. Failures document limitations but never gate Mobius export based + on the runtime registry, topology support, or cache executor capability. +- [ ] If ORT GenAI validation is run for a model with dual head dimensions, + determine whether that runtime requires + `search.past_present_share_buffer=false` for its uniform KV-cache + allocation, and override the generated config only for that probe. +- [ ] Add an ORT GenAI integration test when useful, but do not require one for + export acceptance. + +Omitting these downstream probes requires no waiver or TODO. + + +Waiver needed if any of the steps are not possible. Run the ORT GenAI integration test: @@ -178,14 +186,12 @@ Run the ORT GenAI integration test: python -m pytest tests/ort_genai_test.py -m integration_slow -k "" -sv ``` -### 10. Foundry Local smoke test - -- [ ] Model exported package can be loaded and run in Foundry Local -- [ ] At minimum, verify that the `genai_config.json` and all ONNX files - are present and the model responds to a short prompt +### 10. Foundry Local package check -> If Foundry Local is not available in the current environment, document the -> skip with a `# TODO: verify with Foundry Local` comment in the PR. +- [ ] `genai_config.json` and all ONNX files are present and internally + consistent +- [ ] If Foundry Local is available, record its version and load/generation + result as optional downstream evidence; limitations do not block export ### 11. Olive quantization compatibility @@ -223,8 +229,8 @@ novel weight layouts (e.g. fused QKV, non-standard expert routing). after optimization or rebase changes - [ ] Rebase linearly onto `origin/main`; resolve shared registries/helpers semantically, run shared-surface tests, and push with `--force-with-lease` -- [ ] After rebase, rerun config generation and actual runtime load/generation; - parity-only tests do not catch changed cache/runtime contracts +- [ ] After rebase, rerun metadata generation and intrinsic config validation. + If downstream runtime evidence is included, rerun that exact probe too. - [ ] Confirm the remote PR head SHA, mergeability, replacement lint, and architecture checks after the final push - [ ] Triage red CI at check/job/test granularity against the exact base SHA; @@ -236,14 +242,13 @@ novel weight layouts (e.g. fused QKV, non-standard expert routing). ## Waiver policy -Any item that cannot be completed must be waived explicitly in the PR +Any required item that cannot be completed must be waived explicitly in the PR description: ``` **Waivers:** - L5 golden: Model is 70B — generating golden data exceeds CI resources. skip_reason added to YAML. -- Foundry Local: Not available in this environment. Tracked in issue #NNN. ``` Unchecked items without a waiver are grounds to request changes before merge. diff --git a/.agents/skills/writing-tests/SKILL.md b/.agents/skills/writing-tests/SKILL.md index 2f92dd4f0..ab7ca84d1 100644 --- a/.agents/skills/writing-tests/SKILL.md +++ b/.agents/skills/writing-tests/SKILL.md @@ -266,6 +266,20 @@ integration test alongside any new custom function or Scan op.** runner termination from a model assertion failure. - Reference goldens must come from an independently invoked upstream pipeline, never from the implementation under test or ad-hoc intermediate features. +- When the checkpoint is too large, HTTP-range-read safetensors headers and the + exact tensors for a production-dimension reduced fixture. Cover every layer + family and cache contract, record the source layer/row derivation, and create + L4/L5 goldens from the independently invoked HuggingFace model. Treat this as + reduced real-weight evidence, not as a claim of full-checkpoint parity. +- Before accepting an architecture xfail, verify config vocabulary, epsilon, + and layer-kind translation. A stale `mlp`->`moe` mapping can look like an SSM + numerical failure while loading the wrong weights entirely. +- Compare full prefill and every token-by-token prompt/decode logit on the + target EP. Fused cache kernels can match multi-token prefill yet diverge on + the first reused-state step. If full-precision and quantized packages fail at + the same reused-state step, suspect source cache semantics rather than Olive. + Test the equivalent standard-ONNX cache graph before assigning blame; prefer + the portable graph when it restores the numeric gate. ### Recurrent state ≠ KV cache