Skip to content

Fix rocSPARSE handle and test buffer leaks under ASAN (#388) - #460

Open
zjin-lcf wants to merge 2 commits into
ORNL:developfrom
zjin-lcf:fix/issue-388-rocsparse-handle-leaks
Open

Fix rocSPARSE handle and test buffer leaks under ASAN (#388)#460
zjin-lcf wants to merge 2 commits into
ORNL:developfrom
zjin-lcf:fix/issue-388-rocsparse-handle-leaks

Conversation

@zjin-lcf

@zjin-lcf zjin-lcf commented Jul 23, 2026

Copy link
Copy Markdown

Summary

Fixes handle/memory leaks in the ROCm (HIP) backend that LeakSanitizer reports under -DRESOLVE_USE_ASAN=yes. LSan attributes the allocation sites to librocsparse.so / librocsolver.so, which makes them look like AMD defects, but they are actually caused by ReSolve creating rocSPARSE/rocSOLVER handles (and a device matrix) that it never destroys.

Note (updated): The earlier version of this PR claimed the only remaining leaks were external ROCm runtime-init allocations. That was only true for the reduced test matrix used at the time (KLU was disabled, so the rocSOLVER-Rf refactorization tests were never built or run). Enabling KLU surfaced an additional ReSolve-owned leak in LinSolverDirectRocSolverRf, now fixed here.

Changes

  • resolve/LinSolverDirectRocSparseILU0.cpp: setup() creates descr_A_, descr_L_, descr_U_, info_A_, and a device analysis buffer_, but the destructor only freed the ILU value/aux device arrays. Destroy the three matrix descriptors and the info object and free buffer_ on teardown.
  • resolve/LinSolverDirectRocSolverRf.cpp: setup() creates the rocSOLVER refactorization info object infoM_ (rocsolver_create_rfinfo) and the combined-factor matrix M_ (new matrix::Csr in combineFactors), but the destructor freed neither, and repeated setup() calls leaked both again. Now: destroy infoM_ and delete M_ in the destructor, destroy any previous infoM_ before recreating it in setup(), and delete any previous M_ before reallocating it. This mirrors the existing CUDA teardown (LinSolverDirectCuSolverGLU deletes M_; CholeskySolverHip destroys its rfinfo).
  • tests/unit/matrix/SparseTests.hpp: copyValues allocated a host scratch buffer with new[] in the device path and never freed it. Free it in the device case (on host it aliases matrix-owned data and must not be freed).
  • tests/lsan.supp (new): LSan suppression file for the residual, external ROCm runtime-init allocations that are not fixable from within ReSolve.

Test plan

Environment: AMD Instinct MI300A (gfx942), ROCm 7.2.4.

cmake -DRESOLVE_USE_ASAN=yes -DRESOLVE_USE_UBSAN=yes -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DRESOLVE_USE_HIP=yes -DCMAKE_HIP_ARCHITECTURES=gfx942 -DRESOLVE_USE_KLU=yes ..
make -j && ctest -j
  • ILU0 / test buffer: before, all rocsparse_create_mat_descr / rocsparse_create_mat_info leaks present and sparse_matrix_test leaks a host buffer; after, gone.
  • rocSOLVER-Rf (rocsolver_rf_test, requires KLU): before 1169156 B / 139 allocs (LSan names combineFactors ... :338M_ and its Csr::allocateMatrixData children, plus rocsolver_create_rfinfoinfoM_); after 6192 B / 108 allocs.
  • After all fixes, no ReSolve-attributed leaks remain in any test.

Remaining leaks (external ROCm runtime init, suppressed)

The only residual LSan output is a fixed ~6 KB inside libhsa-runtime64.so / libamdhip64.so. Symbolizing against the library's embedded .gnu_debugdata:

rocr::AMD::hsa_amd_signal_create        4704 B / 84 objs + 864 B / 12 objs
hsaKmtCreateEventCtx                     480 B / 10 objs (indirect; backs the signals)
rocr::HSA::hsa_queue_create -> GpuAgent::QueueCreate -> AqlQueue::AqlQueue   72 B (+ children)

These are one-time HSA/HIP runtime-initialization allocations owned by the process-global rocr::core::Runtime singleton; HIP never calls hsa_shut_down() at exit, so LSan flags them. They are:

AMD's own GPU sanitizer docs classify these language-runtime allocations as "not considered to be leaks" and recommend exactly this suppression-file approach, which tests/lsan.supp implements.

Relationship to #388

This does not fully close #388, but it eliminates all ReSolve-owned leaks in the HIP backend (ILU0 handles, rocSOLVER-Rf infoM_/M_, and the test host buffer). The remaining LSan reports are exclusively the external, one-time ROCm runtime-init allocations covered by tests/lsan.supp.

LeakSanitizer attributed leaks to librocsparse.so, but they were caused
by ReSolve creating rocSPARSE handles it never released:

- LinSolverDirectRocSparseILU0 created descr_A_/descr_L_/descr_U_,
  info_A_, and a device buffer_ in setup() but its destructor only
  freed the ILU value/aux device arrays. Destroy the matrix
  descriptors and info object and free the analysis buffer on
  teardown.
- SparseTests::copyValues allocated a host scratch buffer with new[]
  in the device code path and never freed it. Delete it (only in the
  device case, where it is owned; on host it aliases matrix data).

A minimal create/analysis/destroy reproducer shows rocSPARSE 7.2.4
itself does not leak, so this is a ReSolve ownership bug, not a
rocSPARSE defect.

Co-authored-by: Cursor <cursoragent@cursor.com>

@superwhiskers superwhiskers left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this will not close #388, but it does reduce the leak load

LinSolverDirectRocSolverRf::setup() creates the rocSOLVER refactorization
info object infoM_ (rocsolver_create_rfinfo) and the combined-factor matrix
M_ (new matrix::Csr in combineFactors), but the destructor freed neither, and
repeated setup() calls leaked both again. Under LeakSanitizer this shows up as
allocations attributed to librocsolver.so plus large indirect leaks from
matrix::Csr::allocateMatrixData (~1.16 MB per solve in klu_rf_test).

- Destroy infoM_ and delete M_ in the destructor.
- Destroy any previous infoM_ before recreating it in setup().
- Delete any previous M_ before reallocating it in combineFactors().

This mirrors the existing CUDA teardown (LinSolverDirectCuSolverGLU deletes M_,
CholeskySolverHip destroys its rfinfo). Verified with a KLU + HIP + ASAN build:
rocsolver_rf_test drops from 1169156 B / 139 allocs to 6192 B / 108 allocs,
with the residual being only one-time libhsa-runtime64.so/libamdhip64.so
runtime-init allocations.

Also add tests/lsan.supp, an LSan suppression file for those external ROCm
runtime init allocations (see ORNL#388), which are not fixable from
within ReSolve.

Co-authored-by: Cursor <cursoragent@cursor.com>
@superwhiskers

Copy link
Copy Markdown
Collaborator

i think the leak suppression list added there is too broad. unless i'm misunderstanding how the suppression list works, that will, in addition to hiding the runtime globals, also hide future rocm adjacent leaks because it targets the shared objects and not the individual functions responsible for the leak

@pelesh
pelesh self-requested a review July 31, 2026 17:04

@pelesh pelesh left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@zjin-lcf thanks for your contribution. Please provide your before and after test logs with instructions how to reproduce them. Also, please go through Re::Solve PR checklist available in the PR template.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: 35/46 tests fail on ROCm when using address sanitizers.

3 participants