fix(amdgpu): forward custom op domains from the hip backend too - #102
fix(amdgpu): forward custom op domains from the hip backend too#102AMDmoore wants to merge 2 commits into
Conversation
|
|
| RETURN_IF_ERROR(hip_ep_factory_->GetNumCustomOpDomains(hip_ep_factory_, &hip_num_domains)); | ||
| hip_num_domains = std::min(hip_num_domains, num_domains - offset); | ||
| if (hip_num_domains > 0) { | ||
| RETURN_IF_ERROR(hip_ep_factory_->GetCustomOpDomains(hip_ep_factory_, domains + offset, hip_num_domains)); |
There was a problem hiding this comment.
What happens if HIP EP returns custom op domains on an unsupported GPU? HIP EP is only built to support gfx1151 at the moment.
There was a problem hiding this comment.
Good question. Whether the custom op or standard op, I think hip-ep has the same behavior for an unsupported GPU. If profiler is configured as hip, ORT can recognize the custom ops that hip-ep registers, then hip-ep compiler can compile successfully but fails at runtime due to unsupported GPU, either hipmalloc error or kernel launch failure.
There was a problem hiding this comment.
Have you tested on non-gfx1151 hardware with a gpu ep produced build that only has gfx1151 tensile/custom_operator files? We have to make sure there is good fallback behavior that doesn't crash.
There was a problem hiding this comment.
Hi @tperry-amd, do you know if anyone has tested this base case before? a GPU EP produced build that only has gfx1151 standard ops on non-gfx1151 hardware, what's the base behavior?
As you requested, I did a test with prebuilt gfx1151 package running on gfx1150 GPU and gfx1150 theRock for win.
case1: profile as auto, EP for Hipep backend is not created.
(1) standard op model like Llama-3.1-8B, fallback to CPU EP and run successfully.
(2) custom op model like com.amd::QMoE, fallback to CPU EP but ORT throws exception, message like "hip-ep-registered custom op node 'qmoe_amd_L0' fell back to CPU: this op is implemented only as a GPU kernel in the hip-ep backend and has no CPU kernel."
case2: profile as hip, only Hipep backend is selected.
(1) standard op model like Llama-3.1-8B, ORT session initialization fails.
(2) custom op model like com.amd::QMoE, also ORT session initialization fails.
These two cases both fails like [2026-08-31 01:59:39.8219620 [E:onnxruntime:, inference_session.cc:2812 onnxruntime::InferenceSession::Initialize::<lambda_7>::operator ()] Exception during initialization: ].
So, as I replied some days ago, the amdgpu behavior has no difference between standard op and custom op on an unsupported GPU platform, it will fall back to CPUEP for case1 and throw exception for case2. By this PR, it only changes the exception from ORT model load (op not supported) to the other places.
For case1-(2), I think there should be more effort to ease the arch restrictions from gfx1151.
cc: @amd-mingw @zz002
There was a problem hiding this comment.
Hi, @tperry-amd which fallback path is expected? Thanks! cc @AMDmoore
Have you tested on non-gfx1151 hardware with a gpu ep produced build that only has gfx1151 tensile/custom_operator files? We have to make sure there is good fallback behavior that doesn't crash.
There was a problem hiding this comment.
Hi @amd-mingw and @AMDmoore, I am mostly referring to a GPU that HIP EP doesn't support today like Navi 31 or Navi 48. Does the hip ep register ops in those cases? Does it only register ops for compatible hardware?
I'm wondering if we need to gate this behind a similar heuristic that we have today. Currently we gate HIP EP on gfx1151 + LLM model arch. (Obviously the gfx1151 will expand in the future)
There was a problem hiding this comment.
Hi @tperry-amd , Custom OP register only has implementation for HIP EP now, If heuristic not route to hip-ep, then it will fallback to migraphx EP? DML EP? CPU EP? If want the fallback EP not crash, they will need support this custom op, too, but that wasn’t the original design intent. The intent was only for the HIP EP to support this custom op.
There was a problem hiding this comment.
@tperry-amd , Custom OP register happen before ORT complete register EP,
for example, VitisAI EP register 2 custom_ops, and the OP register happen before VitisAI EP register completed:
for our AMDGPU EP, the OP register will happen before AMDGPU EP register completed:
so for your question:
Does the hip ep register ops in those cases? --Yes
Does it only register ops for compatible hardware? --No
for a model with custom op, If not register it, it will report error and exit. If no this PR, for the fallback EP(migraphx EP/DML EP/CPU EP), it will report error and exit, too, the behavior is similar with/without this PR.
any suggest approach to support this custom op feature? Thanks!
There was a problem hiding this comment.
My main concern is here that this change doesn't break cert on non-hipep use-cases (like Stable Diffusion) and non-hipep hardware (like Navi4x).
Based your answer "Does it only register ops for compatible hardware? --No", it seems to be that you register ops for incompatible hardware which may result in hardware crashes, TDR, etc.
Please understand that HIP EP is not the only provider, other providers need to function as normal. If they don't, then the entire software cannot ship (including HIP EP) since it will fail cert.
I think the question I need answered. Do the ops get registered if hip backend isn't selected?
| #ifdef USE_DML | ||
| if (dml_ep_factory_ != nullptr && dml_ep_factory_->GetCustomOpDomains != nullptr) { | ||
| RETURN_IF_ERROR(dml_ep_factory_->GetCustomOpDomains(dml_ep_factory_, domains, num_domains)); | ||
| size_t dml_num_domains{}; |
There was a problem hiding this comment.
Can you reduce the code duplication here? It should just be a function that accepts a factory pointer since the logic between the dml and hip ep factory are identical it seems, it's just the factory pointer that's different.
| RETURN_IF_ERROR(hip_ep_factory_->GetNumCustomOpDomains(hip_ep_factory_, &hip_num_domains)); | ||
| hip_num_domains = std::min(hip_num_domains, num_domains - offset); | ||
| if (hip_num_domains > 0) { | ||
| RETURN_IF_ERROR(hip_ep_factory_->GetCustomOpDomains(hip_ep_factory_, domains + offset, hip_num_domains)); |
There was a problem hiding this comment.
Have you tested on non-gfx1151 hardware with a gpu ep produced build that only has gfx1151 tensile/custom_operator files? We have to make sure there is good fallback behavior that doesn't crash.
dddde34 to
4911385
Compare
GetNumCustomOpDomains/GetCustomOpDomains only forwarded to the DirectML backend factory. Schema registration happens once, at Model::Load, before any session/EP or profile selection exists, so the union of every built backend's domains must be reported regardless of which backend a given model eventually routes to. Without this, a graph containing a HIP-only custom op (e.g. com.amd::QMoE) fails schema resolution during Model::Load even though the HIP backend is built and would otherwise handle the node. Extend both functions to also forward to hip_ep_factory_ (guarded by USE_HIP, matching the existing USE_DML guard), tracking an offset across backends in GetCustomOpDomains so each backend's domains are written into its own slice of the caller's buffer. Co-authored-by: Cursor <cursoragent@cursor.com>
4911385 to
288144b
Compare
GetNumCustomOpDomains and GetCustomOpDomains each repeated the same null-check, count, clamp and append sequence once per backend, with only the factory pointer differing, behind their own USE_DML/USE_HIP guards. Collect the backends that can contribute custom op schemas into custom_op_backends_ as each factory is created, and iterate that member in both functions. The push_back sites sit inside the USE_DML/USE_HIP blocks the constructor already has, so no new preprocessor guard is introduced, and the build with neither backend no longer produces unused variables needing (void) casts. Both functions now walk the same list in the same order, so the offset-sliced layout GetCustomOpDomains writes cannot drift from what GetNumCustomOpDomains counted. Adding a backend is one push_back at its load site. No behavior change. Verified warning-free at /W4 with USE_DML+USE_HIP, USE_HIP only, USE_DML only, and neither backend. Checked end to end through the amdgpu EP on an 8-layer nemotron submodel carrying com.amd::QMoE: it loads and compiles when built with USE_HIP, and Model::Load fails with 'com.amd:QMoE(-1) is not a registered function/op' when built without it -- before backend routing runs, which is why the union has to be reported at Model::Load. Load, compile and run results are identical to the pre-refactor build. Co-authored-by: Cursor <cursoragent@cursor.com>
4 participants
Summary
GetNumCustomOpDomains/GetCustomOpDomainson the AMDGPU umbrella EP factory only forwarded to the DirectML backend factory, never to the HIP backend factory.Model::Load, before any session/EP exists and before theprofileprovider option selects a backend. The union of every built backend's custom op domains must therefore be reported here, regardless of which backend a given model eventually routes to.com.amd::QMoE, used by fused Nemotron-H MoE models) fails schema resolution atModel::Loadeven though the HIP backend is built and would otherwise handle the node.hip_ep_factory_(guarded byUSE_HIP, matching the existingUSE_DMLguard), tracking a byte offset across backends inGetCustomOpDomainsso each backend's domains are written into its own slice of the caller-provided buffer.Test plan
amdgpu-ep.dll+hip-backend.dll(fresh, from this branch) plusdirectx-backend.dll/migraphx-backend.dllon Windows/gfx1151 (TheRock ROCm SDK), deployed alongside a pre-existing known-goodhipgpu.dll/hip-compiler.dll/custom kernels build.model_benchmarkend-to-end pass against a real NVIDIA-Nemotron-3-Super-120B QMoE-fused model (genai_config.jsonprovider =AMDGPU, profile =hip) withHIPDNN_EP_STRICT=1(no silent CPU fallback):Model::Loadsucceeded (thecom.amd::QMoEschema resolved through the AMDGPU factory's forwarded HIP domains).hipMalloc failed for constants blob (68000755968 bytes)insideInferenceState::create-- the model's 68GB of constants exceed the 34GB VRAM on the test GPU (gfx1151). This is an orthogonal hardware-capacity limitation, not a schema/domain-forwarding regression, and matches the exact known failure signature from prior HIP-EP-direct validation of the same model.