Skip to content

[BUG]: VirtualMemoryResource host_numa allocations always fail #2694

Description

@fedonman

Is this a duplicate?

Type of Bug

Runtime Error

Component

cuda.core

Describe the bug

Every VirtualMemoryResource allocation with location_type="host_numa" fails with CUDA_ERROR_INVALID_VALUE, on a system that reports host_numa_id 0 and host_numa_virtual_memory_management_supported True. VirtualMemoryResource.allocate passes prop.location.id = -1 whenever the resource has no device, which __init__ sets for every host location type, and cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA with that id. cuda_core/cuda/core/typing.py documents HOST_NUMA as "host memory pinned to a specific NUMA node", and VirtualMemoryResourceOptions has no field that carries a node id.

location_type="host_numa_current" also fails, for a separate reason: cuMemCreate rejects CU_MEM_LOCATION_TYPE_HOST_NUMA_CURRENT for every node id I passed it, including 0.

test_vmm_host_location_types_report_host_accessible in cuda_core/tests/test_memory.py parametrizes over host, host_numa and host_numa_current, but it only checks mr.device and mr.is_host_accessible, so it stays green without ever calling allocate.

How to Reproduce

from cuda.core import Device, VirtualMemoryResource, VirtualMemoryResourceOptions

dev = Device()
dev.set_current()
print("host_numa_id =", dev.properties.host_numa_id)
for location_type in ("host", "host_numa"):
    opts = VirtualMemoryResourceOptions(location_type=location_type, handle_type=None)
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
    print(location_type, "allocated", buf.size, "bytes")
    buf.close()
host_numa_id = 0
host allocated 2097152 bytes
Traceback (most recent call last):
  File "/tmp/vmm_host_numa.py", line 8, in <module>
    buf = VirtualMemoryResource(dev, config=opts).allocate(4096)
  File "/home/vyron-vasileiadis/projects/forks/cuda-python/cuda_core/cuda/core/_memory/_virtual_memory_resource.py", line 555, in allocate
    raise_if_driver_error(res)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^
  File "cuda/core/_utils/cuda_utils.pyx", line 134, in cuda.core._utils.cuda_utils._check_driver_error
  File "cuda/core/_utils/cuda_utils.pyx", line 145, in cuda.core._utils.cuda_utils._check_driver_error
cuda.core._utils.cuda_utils.CUDAError: CUDA_ERROR_INVALID_VALUE: This indicates that one or more of the parameters passed to the API call is not within an acceptable range of values.

Expected behavior

allocate should either return a buffer pinned to a NUMA node, the way location_type="host" and location_type="device" already do, or fail at construction with an error naming the missing node id. Driving cuMemCreate directly on this system, CU_MEM_LOCATION_TYPE_HOST_NUMA returns CUDA_SUCCESS at location.id = 0 and CUDA_ERROR_INVALID_VALUE at location.id = -1, so the location id is the only thing standing between the current behaviour and a working allocation.

Operating System

Ubuntu 26.04 LTS

nvidia-smi output

Tue Aug 25 11:22:23 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.84                 Driver Version: 595.84         CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3050 ...    Off |   00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8              3W /   30W |      11MiB /   4096MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

Metadata

Metadata

Assignees

No one assigned

    Labels

    triageNeeds the team's attention

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions