You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
cuda.core: IPC pickling rebuilds memory resources with the sender's device indices #3097
DeviceMemoryResource.__reduce__ and VirtualMemoryResource.__reduce__ pickle a resource as (device index, options), and the receiver rebuilds it with the same index. Device indices are per process: under a different CUDA_VISIBLE_DEVICES, the same index names a different GPU.
Effects
Owner device: Buffer.from_ipc_descriptor checks that the imported memory lives on the resource's device, so a mismatch raises ValueError during unpickling instead of mapping the wrong GPU. The failure is correct, but late and unhelpful.
The receiver can only name devices it can see, and it can import the same descriptor with any peers through Buffer.from_ipc_descriptor once it holds the handle, so this is a correctness problem, not an escape from CUDA_VISIBLE_DEVICES.
Proposal
Pickle devices by UUID, the owner and the peers, and map them back to the receiver's indices on unpickling; raise a clear error when a UUID is not visible in the receiver. Alternative for the VMM path: drop peers from the pickled config and let receivers add peers through their own resource.
Activity