Skip to content

[QNN EP] Support htp_reused_io_limit_mb provider option for HTP AOT context load - #752

Merged
qti-chuteng merged 26 commits into
mainfrom
dev/chunghow/reused-io-limit
Sep 10, 2026
Merged

qti-chuteng merged 26 commits into
mainfrom
dev/chunghow/reused-io-limit

Conversation

@chunghow-qti

@chunghow-qti chunghow-qti commented Aug 23, 2026

Copy link
Copy Markdown
Collaborator

Description

  • Adds a new htp_reused_io_limit_mb provider option for the HTP backend, wired through to
    QNN_HTP_CONTEXT_CONFIG_OPTION_REUSED_IO_LIMIT when loading a cached QNN context binary (LoadCachedQnnContextFromBuffer).
  • Usage: set ep.qnnexecutionprovider.reused_io_limit_mb value in MB. Default is 0, which leaves the SDK's default (unbounded) behavior unchanged.
  • This option allow client specify the effective I/O buffer size used by a context instead of QNN HTP default estimation. QNN HTP uses this value for memory estimation and DSP PD placement decisions when it loads a context binary.

Motivation and Context

QNN HTP assumes no I/O reuse and estimates the context's memory using the total I/O size of all graphs in the context. If you actually share (reuse) one or more I/O buffers across multiple graphs in a context, or across multiple contexts, that default estimate can overshoot what the context really needs.

This overestimation matters because HTP loads each context into one of several process domains (PDs), each with a limited memory budget. An inflated estimate can cause a context to be placed in its own PD instead of sharing one with others (which is slower, since contexts on different PDs pay extra cost to talk to each other) — or it can cause the context to fail to load at all (QNN error 1002, "Failed to find available PD") even though it would actually fit.

By explicitly specifying the reused I/O size, you let QNN HTP use a smaller, more accurate estimate instead.

Test plan

  • Context that doesn't fit in a PD → fits with the option
    Take a context binary that fails to load by default (QNN error 1002, "Failed to find available PD"). Load it
    • (a) without htp_reused_io_limit_mb → expect 1002;
    • (b) with a known-safe value → expect load success and correct inference output.
  • Verify htp_reused_io_limit_mb is applied on both AOT context-load paths
    • contextCreateFromBinary: load four static-1024 contexts with
      htp_reused_io_limit_mb=2; confirm QNN verbose log reports
      Setting Reused IO memory estimation to 2 MB and all contexts load successfully.
    • contextCreateFromBinaryListAsync: run
      QnnHTPBackendTests.HtpSharedResourceOptimization_HtpReusedIoLimitMb_LoadsSucceeds;
      confirm QNN verbose log reports
      Setting Reused IO memory estimation to 128 MB.

@chunghow-qti
chunghow-qti force-pushed the dev/chunghow/reused-io-limit branch from 5e980d1 to 42490e9 Compare August 25, 2026 07:00
@chunghow-qti

Copy link
Copy Markdown
Collaborator Author

/ci

Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.h Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated

@huaychou huaychou left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we need to add htp_reused_io_limit_mb in onnxruntime_perf_test?

Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc Outdated
Comment thread docs/execution_providers/QNN-ExecutionProvider.md
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
Comment thread docs/execution_providers/QNN-ExecutionProvider.md Outdated
@chunghow-qti chunghow-qti changed the title [QNN EP] Expose reused_io_limit_mb provider option for HTP AOT context load [QNN EP] Expose htp_reused_io_limit_mb provider option for HTP AOT context load Aug 27, 2026
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.cc Outdated
@chunghow-qti

chunghow-qti commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator Author

will rebase on PR#778 and PR#677
Also updated the doc from owners info.

@chunghow-qti chunghow-qti changed the title [QNN EP] Expose htp_reused_io_limit_mb provider option for HTP AOT context load [QNN EP] Support htp_reused_io_limit_mb provider option for HTP AOT context load Aug 28, 2026
@chunghow-qti
chunghow-qti force-pushed the dev/chunghow/reused-io-limit branch from 1c79f42 to 95e4a80 Compare September 7, 2026 03:15
Comment thread onnxruntime/core/providers/qnn/builder/qnn_def.h
Comment thread onnxruntime/core/providers/qnn/builder/qnn_backend_manager.h Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc Outdated
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc Outdated
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc Outdated
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc Outdated
Comment thread onnxruntime/core/providers/qnn/builder/qnn_def.h
Comment thread docs/execution_providers/QNN-ExecutionProvider.md Outdated
Comment thread docs/execution_providers/QNN-ExecutionProvider.md Outdated
@chunghow-qti

Copy link
Copy Markdown
Collaborator Author

Update the doc to use htp_share_resource_optimization to clarify default path or not, which is more clear to user instead of using QNN API explaination.

Comment thread docs/execution_providers/QNN-ExecutionProvider.md Outdated
Comment thread onnxruntime/core/providers/qnn/qnn_execution_provider.cc
Comment thread onnxruntime/test/providers/qnn/qnn_ep_context_test.cc
@qti-chuteng
qti-chuteng merged commit 9371367 into main Sep 10, 2026
177 of 181 checks passed
@qti-chuteng
qti-chuteng deleted the dev/chunghow/reused-io-limit branch September 10, 2026 05:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants