Skip to content

cuda.core: IPC pickling rebuilds memory resources with the sender's device indices #3097

Open

Description

@Andy-Jost

DeviceMemoryResource.__reduce__ and VirtualMemoryResource.__reduce__ pickle a resource as (device index, options), and the receiver rebuilds it with the same index. Device indices are per process: under a different CUDA_VISIBLE_DEVICES, the same index names a different GPU.

Effects

  • Owner device: Buffer.from_ipc_descriptor checks that the imported memory lives on the resource's device, so a mismatch raises ValueError during unpickling instead of mapping the wrong GPU. The failure is correct, but late and unhelpful.
  • VirtualMemoryResource peers: the pickled config's peers are interpreted in the receiver's numbering. A peer index that names a different GPU in the receiver grants access from that GPU, and the intended peer faults later. Reported by @brandon-b-miller and @juenglin in cuda.core: share VirtualMemoryResource buffers between processes #3028.

The receiver can only name devices it can see, and it can import the same descriptor with any peers through Buffer.from_ipc_descriptor once it holds the handle, so this is a correctness problem, not an escape from CUDA_VISIBLE_DEVICES.

Proposal

Pickle devices by UUID, the owner and the peers, and map them back to the receiver's indices on unpickling; raise a clear error when a UUID is not visible in the receiver. Alternative for the VMM path: drop peers from the pickled config and let receivers add peers through their own resource.

Affects

  • cuda_core/cuda/core/_memory/_device_memory_resource.pyx (__reduce__)
  • cuda_core/cuda/core/_memory/_virtual_memory_resource.pyx (__reduce__)

Follow-up to #3028; part of the VMM epic #2980.

Activity

  1. self-assigned this
    on Oct 13, 2026
  2. added
    P1Medium priority - Should do
    cuda.coreEverything related to the cuda.core module
    on Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P1Medium priority - Should dobugSomething isn't workingcuda.coreEverything related to the cuda.core module

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions