Repository navigation
test(bindings): cap the pools in the memPool attribute tests - #3140
Conversation
test_cuda_memPool_attr and test_cudart_MemPool_attr created pools with maxSize 0, which reserves virtual address space of about twice the device memory. In a 39-bit address space (riscv64 Sv39) this fails with CUDA_ERROR_OUT_OF_MEMORY on large-memory GPUs. Create 2 MiB pools instead; the tests only query and set pool attributes. The graph memory pool used by test_cuda_graphMem_attr cannot be capped the same way, so xfail that test when cuGraphAddMemAllocNode returns CUDA_ERROR_OUT_OF_MEMORY. Previously the OOM branch cleaned up and then failed the assertion.
juenglin
added
test
|
/ok to test c183f19 |
📝 Summary
Priority: ➖ Normal Change: Bug fix Merge Risk: 🔵 Low · up to
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
ℹ️ Review info
⚙️ Run configuration
- Configuration used: Repository: NVIDIA/cuda-python/.coderabbit.yaml
- Review profile: CHILL
- Plan: Enterprise
- Run ID:
08bf1385-72e9-418d-9e6c-efea69bae525
📒 Files selected for processing: 2
cuda_bindings/tests/test_cuda.pycuda_bindings/tests/test_cudart.py
Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 9 remain after this review.
| @pytest.fixture | ||
| def xfail_on_graph_mempool_oom(): | ||
| """Return a check that xfails the calling test if the graph memory pool cannot be created. | ||
|
|
||
| The graph memory pool reserves virtual address space of about twice the | ||
| device memory, and unlike pools from cuMemPoolCreate it cannot be capped. | ||
| On systems with a limited virtual address space this reservation can fail | ||
| with CUDA_ERROR_OUT_OF_MEMORY, especially when another uncapped pool | ||
| already exists in the process. | ||
| """ | ||
|
|
||
| def check(err): | ||
| if err == cuda.CUresult.CUDA_ERROR_OUT_OF_MEMORY: | ||
| pytest.xfail("graph memory pool creation failed with CUDA_ERROR_OUT_OF_MEMORY") | ||
|
|
||
| return check |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🔎 Supported by static analysis
🏁 Script executed:
sed -n '450,530p' cuda_bindings/tests/test_cuda.py
rg -n 'xfail_on_graph_mempool_oom|cuGraphAddMemAllocNode|test_cuda_graphMem_attr|CUDA_ERROR_OUT_OF_MEMORY' cuda_bindings/tests/test_cuda.pyRepository: NVIDIA/cuda-python
Length of output: 4065
🏁 Script executed:
# Read the complete graph-memory test and locate the device fixture and API declaration/docs
sed -n '485,570p' cuda_bindings/tests/test_cuda.py
rg -n -F --glob '*.py' -e 'def device' -e 'deviceGetAttribute' -e 'cuGraphAddMemAllocNode' cuda_bindings/tests tests || test "$?" -eq 1
rg -n -F --glob '*.{py,pyx,pxd,c,h}' -e 'cuGraphAddMemAllocNode' cuda_bindings | head -40Repository: NVIDIA/cuda-python
Length of output: 9972
🏁 Script executed:
sed -n '46384,46520p' cuda_bindings/cuda/bindings/driver.pyx
sed -n '145,195p' cuda_bindings/tests/conftest.py
rg -n -F --glob '*.py' -e 'cuDeviceTotalMem' -e 'CU_DEVICE_ATTRIBUTE_TOTAL_CONSTANT_MEMORY' -e 'cuDeviceGetAttribute' cuda_bindings/tests/conftest.py cuda_bindings/tests/test_cuda.py | head -80Repository: NVIDIA/cuda-python
Length of output: 8642
🌐 Web query:
official CUDA Driver API cuGraphAddMemAllocNode CUDA_ERROR_OUT_OF_MEMORY virtual address space graph memory pool
💡 Result:
**`cuGraphAddMemAllocNode` does not list `CUDA_ERROR_OUT_OF_MEMORY` as a documented return value** in the CUDA Driver API 13.4 reference. Its listed results are `CUDA_SUCCESS`, `CUDA_ERROR_DEINITIALIZED`, `CUDA_ERROR_NOT_INITIALIZED`, and `CUDA_ERROR_INVALID_VALUE`. The reference does caution that the call may also surface errors from earlier asynchronous launches. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))
The API assigns the allocation’s virtual address when the node is created, and that address stays fixed across graph instantiations and launches. CUDA can reuse virtual address ranges for allocations whose graph-ordered lifetimes do not overlap, and can reuse physical memory across graphs that do not execute simultaneously. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))
**What the docs don’t establish:** they don’t say that `CUDA_ERROR_OUT_OF_MEMORY` from this call specifically means virtual-address-space exhaustion, nor do they document that this call allocates from a user-selected memory pool. So that diagnosis cannot be concluded from the return-code list alone. Check which CUDA operation actually reports the error and whether it’s a prior asynchronous error; separately inspect graph-memory usage with `cuDeviceGetGraphMemAttribute` and consider `cuDeviceGraphMemTrim` for unused cached graph memory. ([docs.nvidia.com](https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html))
Citations:
- 1: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html
- 2: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html
- 3: https://docs.nvidia.com/cuda/cuda-driver-api/cuda_driver_api/group__CUDA__GRAPH.html
suggestion: Gate this XFAIL on the known virtual-address-space limitation.
params.bytesize is 1, but cuGraphAddMemAllocNode returns only a CUresult; this branch does not identify why the driver returned CUDA_ERROR_OUT_OF_MEMORY. Any unrelated allocation or driver regression with the same result skips the success assertion and all graph-memory assertions. The API documentation does not define this error as a virtual-address-space-specific result. Use a predicate that identifies the known address-space configuration; otherwise let the error fail the test. pytest.skip and xfail(strict=False) would not narrow this condition.
|
2 participants
Description
follow-on to #2381 and #3011
test_cuda_memPool_attrandtest_cudart_MemPool_attrcreate a memory pool withmaxSizeleft at 0. Such a pool reserves virtual address space of about twice the installed device memory. In a 39-bit address space (riscv64 Sv39, 256 GiB of user address space) that reservation fails withCUDA_ERROR_OUT_OF_MEMORYon large-memory GPUs, even though the GPU is nearly empty. Both tests only query and set pool attributes, so they now create a 2 MiB pool (maxSize = POOL_SIZE), the same captest_interop_memPooluses since #3011.test_cuda_graphMem_attrallocates from the device's graph memory pool. That pool is managed by the driver and, unlike pools fromcuMemPoolCreate, cannot be capped (cuda.hdocuments nomaxSizeforCUDA_MEM_ALLOC_NODE_PARAMS.poolProps). Its reservation can still fail in a 39-bit address space, especially when another uncapped pool already exists in the process. The test already had aCUDA_ERROR_OUT_OF_MEMORYbranch from #2000 that cleaned up the graph and stream, but since #3004 removed the xfail helper it then fell through to the failing assertion. This PR adds a localxfail_on_graph_mempool_oomfixture that the branch calls after cleanup, so the test reportsXFAILinstead of failing when graph memory pool creation runs out of address space.Validation:
cuGraphAddMemAllocNodepatched to returnCUDA_ERROR_OUT_OF_MEMORY,test_cuda_graphMem_attrreportsXFAIL.Checklist