Skip to content

test(bindings): cap the pools in the memPool attribute tests #3140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
juenglin merged 3 commits into from
Oct 17, 2026
Merged

test(bindings): cap the pools in the memPool attribute tests #3140

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
30ff361
test(bindings): cap the pools in the memPool attribute tests
juenglin Oct 16, 2026
c215623
test(bindings): generalize the address-space wording in pool comments
juenglin Oct 16, 2026
c183f19
Merge remote-tracking branch 'upstream/main' into test-cap-mempool-at…
juenglin Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 26 additions & 1 deletion cuda_bindings/tests/test_cuda.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,11 @@
from cuda.bindings._internal.utils import FunctionNotFoundError
from cuda_python_test_helpers import driver_version_less_than

# Cap for memory pools created by tests. A pool created with maxSize 0 reserves
# virtual address space of about twice the device memory, and that reservation
# can fail on systems with a limited virtual address space.
POOL_SIZE = 2 * 1024 * 1024 # 2 MiB


def supportsMemoryPool():
err, isSupported = cudart.cudaDeviceGetAttribute(cudart.cudaDeviceAttr.cudaDevAttrMemoryPoolsSupported, 0)
Expand Down Expand Up @@ -265,6 +270,7 @@ def test_cuda_memPool_attr():
poolProps.allocType = cuda.CUmemAllocationType.CU_MEM_ALLOCATION_TYPE_PINNED
poolProps.location.id = 0
poolProps.location.type = cuda.CUmemLocationType.CU_MEM_LOCATION_TYPE_DEVICE
poolProps.maxSize = POOL_SIZE

attr_list = [None] * 8
err, pool = cuda.cuMemPoolCreate(poolProps)
Expand Down Expand Up @@ -460,11 +466,29 @@ def test_cuda_mem_range_attr(device):
assert err == cuda.CUresult.CUDA_SUCCESS


@pytest.fixture
def xfail_on_graph_mempool_oom():
"""Return a check that xfails the calling test if the graph memory pool cannot be created.

The graph memory pool reserves virtual address space of about twice the
device memory, and unlike pools from cuMemPoolCreate it cannot be capped.
On systems with a limited virtual address space this reservation can fail
with CUDA_ERROR_OUT_OF_MEMORY, especially when another uncapped pool
already exists in the process.
"""

def check(err):
if err == cuda.CUresult.CUDA_ERROR_OUT_OF_MEMORY:
pytest.xfail("graph memory pool creation failed with CUDA_ERROR_OUT_OF_MEMORY")

return check
Comment on lines +469 to +484

@coderabbitai coderabbitai Bot Oct 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🔎 Supported by static analysis

🏁 Script executed:

sed -n '450,530p' cuda_bindings/tests/test_cuda.py
rg -n 'xfail_on_graph_mempool_oom|cuGraphAddMemAllocNode|test_cuda_graphMem_attr|CUDA_ERROR_OUT_OF_MEMORY' cuda_bindings/tests/test_cuda.py

Repository: NVIDIA/cuda-python

Length of output: 4065

🏁 Script executed:

# Read the complete graph-memory test and locate the device fixture and API declaration/docs
sed -n '485,570p' cuda_bindings/tests/test_cuda.py
rg -n -F --glob '*.py' -e 'def device' -e 'deviceGetAttribute' -e 'cuGraphAddMemAllocNode' cuda_bindings/tests tests || test "$?" -eq 1
rg -n -F --glob '*.{py,pyx,pxd,c,h}' -e 'cuGraphAddMemAllocNode' cuda_bindings | head -40

Repository: NVIDIA/cuda-python

Length of output: 9972

🏁 Script executed:

sed -n '46384,46520p' cuda_bindings/cuda/bindings/driver.pyx
sed -n '145,195p' cuda_bindings/tests/conftest.py
rg -n -F --glob '*.py' -e 'cuDeviceTotalMem' -e 'CU_DEVICE_ATTRIBUTE_TOTAL_CONSTANT_MEMORY' -e 'cuDeviceGetAttribute' cuda_bindings/tests/conftest.py cuda_bindings/tests/test_cuda.py | head -80

Repository: NVIDIA/cuda-python

Length of output: 8642

🌐 Web query:

official CUDA Driver API cuGraphAddMemAllocNode CUDA_ERROR_OUT_OF_MEMORY virtual address space graph memory pool

💡 Result:

**`cuGraphAddMemAllocNode` does not list `CUDA_ERROR_OUT_OF_MEMORY` as a documented return value** in the CUDA Driver API 13.4 reference. Its listed results are `CUDA_SUCCESS`, `CUDA_ERROR_DEINITIALIZED`, `CUDA_ERROR_NOT_INITIALIZED`, and `CUDA_ERROR_INVALID_VALUE`. The reference does caution that the call may also surface errors from earlier asynchronous launches. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))

The API assigns the allocation’s virtual address when the node is created, and that address stays fixed across graph instantiations and launches. CUDA can reuse virtual address ranges for allocations whose graph-ordered lifetimes do not overlap, and can reuse physical memory across graphs that do not execute simultaneously. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))

**What the docs don’t establish:** they don’t say that `CUDA_ERROR_OUT_OF_MEMORY` from this call specifically means virtual-address-space exhaustion, nor do they document that this call allocates from a user-selected memory pool. So that diagnosis cannot be concluded from the return-code list alone. Check which CUDA operation actually reports the error and whether it’s a prior asynchronous error; separately inspect graph-memory usage with `cuDeviceGetGraphMemAttribute` and consider `cuDeviceGraphMemTrim` for unused cached graph memory. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))

Citations:

- 1: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html
- 2: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html
- 3: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html

suggestion: Gate this XFAIL on the known virtual-address-space limitation.

params.bytesize is 1, but cuGraphAddMemAllocNode returns only a CUresult; this branch does not identify why the driver returned CUDA_ERROR_OUT_OF_MEMORY. Any unrelated allocation or driver regression with the same result skips the success assertion and all graph-memory assertions. The API documentation does not define this error as a virtual-address-space-specific result. Use a predicate that identifies the known address-space configuration; otherwise let the error fail the test. pytest.skip and xfail(strict=False) would not narrow this condition.



@pytest.mark.skipif(
driver_version_less_than(11040) or not supportsMemoryPool(), reason="Mempool for graphs not supported"
)
@pytest.mark.thread_unsafe(reason="used high memory can be higher if threaded.")
def test_cuda_graphMem_attr(device):
def test_cuda_graphMem_attr(device, xfail_on_graph_mempool_oom):
err, stream = cuda.cuStreamCreate(0)
assert err == cuda.CUresult.CUDA_SUCCESS

Expand All @@ -485,6 +509,7 @@ def test_cuda_graphMem_attr(device):
assert destroy_err == cuda.CUresult.CUDA_SUCCESS
(destroy_err,) = cuda.cuStreamDestroy(stream)
assert destroy_err == cuda.CUresult.CUDA_SUCCESS
xfail_on_graph_mempool_oom(err)
assert err == cuda.CUresult.CUDA_SUCCESS
err, freeNode = cuda.cuGraphAddMemFreeNode(graph, [allocNode], 1, params.dptr)
assert err == cuda.CUresult.CUDA_SUCCESS
Expand Down
6 changes: 6 additions & 0 deletions cuda_bindings/tests/test_cudart.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,11 @@
from cuda.bindings import runtime
from cuda_python_test_helpers import driver_version_less_than

# Cap for memory pools created by tests. A pool created with maxSize 0 reserves
# virtual address space of about twice the device memory, and that reservation
# can fail on systems with a limited virtual address space.
POOL_SIZE = 2 * 1024 * 1024 # 2 MiB


def isSuccess(err):
return err == cudart.cudaError_t.cudaSuccess
Expand Down Expand Up @@ -520,6 +525,7 @@ def test_cudart_MemPool_attr():
poolProps.allocType = cudart.cudaMemAllocationType.cudaMemAllocationTypePinned
poolProps.location.id = 0
poolProps.location.type = cudart.cudaMemLocationType.cudaMemLocationTypeDevice
poolProps.maxSize = POOL_SIZE

attr_list = [None] * 8
err, pool = cudart.cudaMemPoolCreate(poolProps)
Expand Down
Loading