[QNN EP] Add QDQ selector for GRU op to enable INT8 HTP execution by yuhuchua-qti · Pull Request #721 · onnxruntime/onnxruntime-qnn

yuhuchua-qti · 2026-08-13T03:55:30Z

Summary

Enable genuine quantized QDQ GRU execution on HTP (uint8 and native uint16) where it
finalizes, and fp-degrade the configs HTP cannot run quantized so they execute fp32 on QNN
(finalizes and is accurate) instead of hard-failing. Reworked per reviewer feedback: the QDQ
node-group selector is now structural-only and all op-semantic policy lives in the GRU op
builder.

OrtGRUNodeGroupSelector is structural-only — folds any well-formed DQ → GRU → Q
boundary into a single QDQGroup NodeUnit and returns. GRU's Y / Y_h are optional, so an
absent slot is accepted: CheckQDQNodes uniformly skips a nullptr output slot (the
graph-output and consumer-count guards still apply to the present slots). The selector makes
no op-semantic decision.
All fp-fallback policy lives in the GRU op builder — ProcessAttributesAndOutputs
fp-degrades a QDQ GRU when linear_before_reset=0, direction is non-forward, an optional
output is missing, or the input dtype combo is unsupported. fp-degrade emits an explicit
Dequantize → fp32 GRU → Quantize, all on QNN, so the group stays fully assigned
(ExpectedEPNodeAssignment::All). Genuine quantized GRU is kept for the tested/spec combos:
- u8: X/W/R uint8, B uint8-or-int32, initial_h uint8
- native u16: X/initial_h uint16, W/R uint16-or-uint8, B int32
both forward, both outputs, linear_before_reset=1.
IsOpSupported unchanged — still rejects what QNN genuinely cannot run (dynamic/unequal
seq_lens, layout=1, clip, non-default activations); those fall to CPU.

Why fp-degrade instead of decline

Reviewers (minfhong-qti, huaychou) asked to separate structural checks from op-semantic policy.
Previously the selector declined LBR=0 / missing-output / non-tested dtype, leaking policy
into the shared CheckQDQNodes helper. Now the selector is thin; the builder owns policy and
keeps the group on QNN by degrading to fp rather than declining.

HTP quantized-GRU limitations handled by fp-degrade

linear_before_reset=0: a u8 LBR=0 cell fails QNN finalize with Code 1002
(QNN_COMMON_ERROR_MEM_ALLOC) — HTP force-widens the gate activation u8 → QUint16Crouton,
leaving no constructible u8 kernel. Measured on real v73 and v81.
Missing optional output: a Y_h-only u8 fold drifts ~8.8% on v73 (per-step recurrence
requantized at the single present output's scale; a Y-only fold was ~0.75%). Measured on v73.
Non-forward direction / unsupported dtype combo.

Temporary — genuine quantized execution can replace the fp-degrade once HTP registers the
corresponding kernels.

Tests (onnxruntime/test/providers/qnn/gru_test.cc)

Genuine quantized (native HTP, no fp-degrade):

GRU_QDQ_linear_before_reset — u8, LBR=1, both outputs (u8 bias)
GRU_QDQ_linear_before_reset_int32_bias — u8, LBR=1, both outputs, int32 bias
GRU_QDQ_u16_linear_before_reset — native u16, LBR=1, both outputs (int32 bias)

fp-degraded by the builder (run fp on QNN, assert All):

LBR=0 sanity (7): GRU_QDQ_sanity_forward, _forward_seq1, _reverse, _bidirectional,
_bidirectional_wo_B, _bidirectional_wo_H, _bidirectional_all_initializer
GRU_QDQ_u16_sanity_forward (u16, LBR=0)
missing-output: GRU_QDQ_Y_only_bidirectional, GRU_QDQ_Y_h_only_bidirectional (both also
non-forward), GRU_QDQ_Y_h_only_forward (forward + LBR=1 → isolates the missing-output trigger)

Rejected by IsOpSupported (assert None, runs on CPU):

GRU_QDQ_layout1_forward (layout=1)

Genuine-u8 fingerprint: GRU_QDQ_linear_before_reset drifts ~2.24% (u8 vs fp reference) → needs 3%, fails at the default 0.4%. An fp path passes at 0.4%, so the 0.4%→3% gap confirms genuine u8 is engaged.

Verification

Build gate (Windows ARM64 Release, QAIRT 2.49.40.260810): PASS.
Lint: PASS.
Accuracy (QAIRT 2.49.40.260810, seed 2345; full v73 UT sweep of this rework in progress):
- u8 genuine (GRU_QDQ_linear_before_reset): peak normalized error vs qdq@CPU_EP = 2.24%
  on v81 / 2.24384% on v73 (within the test's 3.0% silicon tolerance; 6.0% on the
  linux-x86_64 emulator).
- native u16 (GRU_QDQ_u16_linear_before_reset): peak Y=2.37%, Y_h=2.52% on real v73
  (within 3.0%).
- fp-degraded tests pass at the default 0.4% QDQ tolerance on v73.

Test plan

CPU fp32 GRU tests pass (no regression)
Build gate (ARM64 Release, QAIRT 2.49.40.260810) + lint
On-device v73 UT sweep of this rework (in progress; owner's harness)
Numerical accuracy on the customer model on real device (owner validates with their harness)
Remove the fp-degrade paths once HTP registers the corresponding quantized kernels

🤖 Generated with Claude Code

Without a registered QDQ selector, Q/DQ nodes around GRU are built as separate Quantize/Dequantize ops in the QNN graph. The StridedSlice ops inserted by the GRU builder then operate on float tensors, preventing HTP from using quantized GRU kernels. This registers OrtGRUNodeGroupSelector so that DQ+GRU+Q are recognized as a single QDQ NodeUnit. The GRU builder then receives quantized tensor info and creates all internal ops (StridedSlice, Gru cell, Concat, Reshape) natively in UFIXED_POINT_8 — no separate Q/DQ ops needed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

A u8 QDQ GRU exposing only one of its two optional outputs (Y-only or Y_h-only) hard-crashed with a 0xc0000005 access violation -- a regression from the GRU QDQ selector added in 4b54b1b. The absent output slot is reported by ORT core as a nullptr OrtValueInfo*, and it faulted at two sites: - Stage 1 (qnn_ep_utils.cc CheckQDQNodes): the selector dereferenced the nullptr slot while matching Q nodes. Add an allow_missing_optional_outputs path (set by OrtGRUNodeGroupSelector) that skips empty slots and matches only present outputs, so u8 fusion is still selected without the deref. - Stage 2 (gru_op_builder.cc): the absent slot's internal per-step tensor was emitted as u8 with an UNDEFINED quant encoding -> invalid QNN graph. Backfill each absent output's quant_param from a present output so every emitted tensor carries a valid encoding. Verified on v81 / QAIRT 2.48.40.260702: both crashes now surface as the catchable HTP-finalize Code 1002 that every LBR=0 u8 GRU test hits (u8->QUint16Crouton gate-matmul widening), rather than an access violation. Tests (gru_test.cc): - DISABLE GRU_QDQ_Y_only_bidirectional and GRU_QDQ_Y_h_only_bidirectional: LBR=0 u8 GRU cannot finalize on real HTP silicon (Code 1002; measured on arch v73 and v81). x86 HTP-emulator behavior for this LBR=0 path is not verified. Re-enable when HTP registers a u8 LBR=0 gate kernel. - Relax GRU_QDQ_linear_before_reset tolerance 0.4% -> 3.0%: LBR=1 finalizes and runs on silicon, but the per-timestep unrolled recurrence accumulates u8 quantization drift; measured peak normalized error vs qdq@CPU_EP is 2.24% (Y) / 1.96% (Y_h) on v81. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

OrtGRUNodeGroupSelector declines two u8 GRU configs that fail HTP finalize (Code 1002), so they run fp32 + separate Q/DQ instead: - linear_before_reset=0 (HTP widens the u8 gate matmul to QUint16Crouton). - only one of {Y, Y_h} present (Y_h-only drifts ~8.8%). The 9 re-enabled tests assert the default 0.4% tol on the fp path; the LBR=1 both-output control keeps genuine u8 coverage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

minfhong-qti · 2026-08-26T04:02:10Z

  for (size_t i = 0; i < num_outputs; i++) {
    const OrtValueInfo* value_info = outputs[i];
+    // An empty slot -- nullptr OrtValueInfo* from ORT core, e.g. GRU's optional Y / Y_h -- makes
+    // the group ill-formed; decline (also avoids the nullptr deref in the C API calls below).


Not really get what does ill-formed mean here.
Take GRU for example, the first output is nullptr but the second output isn't. I though this is valid case?
Therefore, directly return false here behaves differently from existing logic which ignores it.

Addressed. CheckQDQNodes now uniformly skips an absent optional output slot (nullptr OrtValueInfo*) instead of rejecting the group — qnn_ep_utils.cc:755-757. Only present slots are counted (present_outputs, :771) and still validated. So a GRU with Y present / Y_h absent (or vice versa) is accepted; the builder decides genuine-quantized vs fp-degrade.

minfhong-qti · 2026-08-26T04:04:28Z

  return true;
 }

+bool OrtGRUNodeGroupSelector::Check(const OrtGraph* graph, const OrtApi& ort_api, const OrtNode* node,


Prefer not to consider HTP-related constraint here since here is a general place for all backends. Move to op builder instead to explicitly construct Q / DQ nodes if want to fallback to FP.

Done.
The selector is now structural-only (qnn_ep_utils.cc:1525-1537): it folds a well-formed DQ→GRU→Q boundary and returns. All op-semantic policy (LBR=0, missing-output, dtype combo) moved into gru_op_builder.cc ProcessAttributesAndOutputs, which fp-degrades unsupported configs on QNN.
Thanks.

huaychou · 2026-08-26T04:20:28Z

+      {ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT8, ONNX_TENSOR_ELEMENT_DATA_TYPE_INT8},  // in[1]: W
+      {ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT8, ONNX_TENSOR_ELEMENT_DATA_TYPE_INT8},  // in[2]: R
+      {ONNX_TENSOR_ELEMENT_DATA_TYPE_UINT8, ONNX_TENSOR_ELEMENT_DATA_TYPE_INT8,
+       ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32},  // in[3]: B


[N-1 Minor] in[3]: B allows UINT8/INT8 but the comment on line 1570 says INT32. Conv/Gemm selectors enforce INT32-only for bias (see qnn_ep_utils.cc:1081). In QDQ models, GRU bias is typically INT32 (accumulated multiply-add precision); allowing UINT8/INT8 may let a config that HTP cannot finalize pass the selector, failing silently at graphFinalize with error 1002 or dtype mismatch.

Suggested fix: verify QnnOpDef.h for the actual dtype HTP accepts for GRU bias. If INT32 only:

{ONNX_TENSOR_ELEMENT_DATA_TYPE_INT32}, // in[3]: B (INT32 per HTP requirement)

If other dtypes are valid, document the SDK reference in a comment.

huaychou · 2026-08-26T04:20:28Z

  ort_selectors_.RegisterSelector(matmulnbits_ops, std::make_unique<OrtMatMulNBitsNodeGroupSelector>());
+
+  // Register GRU ops
+  OrtOpVersionsAndSelector::OpVersionsMap gru_ops = {{"GRU", {}}};


[T-1 Nit] GRU (G) is registered after MatMulNBits (M); the existing registrations follow alphabetical order. Suggest moving it to appear after Gemm and before InstanceNormalization to match convention and avoid duplicate-registration oversights in the future.

Done. Moved to after Gemm (before InstanceNormalization) — qnn_ep_utils.cc:1758-1760.

huaychou · 2026-08-26T04:20:28Z

+// The linux x86_64 HTP emulator's u8 GRU kernel is not bit-accurate to silicon and drifts further
+// (observed peak ~4.29% vs f32@CPU_EP), so relax to 6.0% there while keeping the tight 3.0% bound on
+// real silicon. TODO: Remove the platform-aware tolerance once the emulator u8 kernel matches silicon.
 TEST_F(QnnHTPBackendTests, GRU_QDQ_linear_before_reset) {


[T-2 Nit] The PR description includes tolerance measurements (2.24%/4.29% on v81) but does not explicitly document a kill-test fence: confirming that GRU_QDQ_linear_before_reset fails (accuracy exceeds the pre-existing 0.4% threshold) when OrtGRUNodeGroupSelector is disabled.

Suggested addition to PR description: "Kill-test confirmed: with OrtGRUNodeGroupSelector disabled, GRU_QDQ_linear_before_reset fails (accuracy exceeds 0.4% tolerance)."

Thanks — one subtlety: this kill-test is inverted.

Selector ON (genuine u8): ~2.24% drift (v73/v81) → needs 3%, fails at 0.4%.

Selector OFF (fp GRU): matches CPU-QDQ → passes at 0.4%.

So it's enabling genuine u8 that breaks 0.4%, not disabling. The real fence is the tolerance gap: 2.24% needs 3% and fails at the default 0.4%; an fp path passes at 0.4%.

huaychou · 2026-08-26T05:55:12Z

      output_tensor_infos[i].qnn_data_type = input_tensor_infos[0].qnn_data_type;
    }
  }
+  // TODO: an absent output slot keeps the default UNDEFINED quant_param, emitted verbatim below.


Should we move the comment to L332?

huaychou · 2026-08-26T05:58:34Z

  ORT_RETURN_FALSE_ON_ERROR(ort_api.Node_GetOutputs(node, outputs.data(), outputs.size()), ort_api);

-  // Check if any of the outputs are graph outputs
+  // Walk the output slots and validate them against the Q nodes.


I think we still check if any of the outputs are graph outputs

Kept.
The graph-output guard is retained on present slots: produces_graph_output set at qnn_ep_utils.cc:760-764, enforced in the return (!produces_graph_output, :773). Only absent slots are skipped; present slots' graph-output/consumer-count are still validated.

huaychou · 2026-08-26T05:59:42Z

+  // selector). LBR=1 (what customer models use) finalizes and stays accelerated. Remove once HTP
+  // registers a u8 kernel for the LBR=0 gate matmul.
+  OrtNodeAttrHelper node_helper(*node);
+  if (node_helper.Get("linear_before_reset", static_cast<int64_t>(0)) == 0) {


I think we shouldn't check attribute in QDQ selector.

Done.
The linear_before_reset check is removed from the selector; it's now an fp-degrade trigger in the builder (gru_op_builder.cc:567-587).

huaychou · 2026-08-26T06:02:59Z

+
+  // Per-input data type constraints (index matches ONNX GRU input position)
+  // Empty set means "skip this input" (not quantized)
+  const std::vector<std::unordered_set<int32_t>> input_constraints = {


This should be validated by opbuilder or backend instead QDQ selector

Done. All dtype/semantic validation moved to the GRU op builder; the selector no longer inspects inputs or attributes.

Keep the QDQ node-group selector structural-only (reviewer request) and move every GRU fp-fallback decision into the op builder; wire native u16. - Selector (qnn_ep_utils): OrtGRUNodeGroupSelector::Check folds any well-formed DQ -> GRU -> Q group; CheckQDQNodes uniformly skips an absent optional output (valid ONNX), keeping the graph-output and consumer-count guards on the present slots. - Builder (gru_op_builder): fp-degrades LBR=0, non-forward, missing output, or non-supported dtype via explicit Dequantize -> fp32 GRU -> Quantize (all on QNN); genuine u8 / native u16 kept for the spec combos (both forward, both outputs, linear_before_reset=1). - Tests (gru_test): native-u16, missing-output forward, and int32-bias genuine-u8 coverage. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

GRU_QDQ_u16_linear_before_reset exercises the genuine native-INT16 Gru kernel (LBR=1). The linux x86_64 HTP emulator has no faithful INT16 Gru kernel: it emits a degenerate constant output (every mismatching element collapses to one value, err/output_range up to ~100%), so no tolerance can bracket it -- the previous 6% emulator tolerance could not. On the same model qdq@CPU_EP matches f32 to 0.0003%, confirming the QDQ math is correct and the fault is emulator-kernel-specific, not accuracy drift. Skip the test under __linux__ && __x86_64__ (mirrors the x86-sim skips in cast_test.cc and framework_op_trace_test.cc); real silicon keeps the tight 3.0% bound. The u8 mirror is unaffected -- its emulator INT8 kernel only drifts (bounded), so it retains its 6% emulator tolerance and passes. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

yuhuchua-qti force-pushed the dev/yuhuchua/qdq_gru_selector branch 6 times, most recently from 39d6d49 to 91704c8 Compare August 25, 2026 08:21

yuhuchua-qti marked this pull request as ready for review August 25, 2026 08:22

yuhuchua-qti requested review from qti-ashwshan, qti-chuteng, qti-jkilpatrick, qti-kromero, qti-yuduo, tirupath-qti and yath1 as code owners August 25, 2026 08:22

yuhuchua-qti and others added 2 commits August 25, 2026 16:29

yuhuchua-qti marked this pull request as draft August 25, 2026 08:52

yuhuchua-qti force-pushed the dev/yuhuchua/qdq_gru_selector branch from 91704c8 to 1716f68 Compare August 25, 2026 10:08

yuhuchua-qti marked this pull request as ready for review August 25, 2026 10:53

yuhuchua-qti force-pushed the dev/yuhuchua/qdq_gru_selector branch from 1716f68 to 837da89 Compare August 25, 2026 15:20

yuhuchua-qti force-pushed the dev/yuhuchua/qdq_gru_selector branch from 837da89 to fd0faaa Compare August 25, 2026 17:07

minfhong-qti reviewed Aug 26, 2026

View reviewed changes

huaychou reviewed Aug 26, 2026

View reviewed changes

yuhuchua-qti marked this pull request as draft August 27, 2026 03:59

yuhuchua-qti force-pushed the dev/yuhuchua/qdq_gru_selector branch from e4ec635 to fab9220 Compare August 27, 2026 08:06

yuhuchua-qti marked this pull request as ready for review August 27, 2026 08:34

yuhuchua-qti marked this pull request as draft August 27, 2026 08:34

yuhuchua-qti marked this pull request as ready for review August 27, 2026 10:22

yuhuchua-qti marked this pull request as draft August 27, 2026 10:22

Uh oh!

Conversation

yuhuchua-qti commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why fp-degrade instead of decline

HTP quantized-GRU limitations handled by fp-degrade

Tests (onnxruntime/test/providers/qnn/gru_test.cc)

Verification

Test plan

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yuhuchua-qti commented Aug 13, 2026 •

edited

Loading