VRAM exhaustion during multi-GPU FSDP/RCCL workload hard-aborts via rocdevice.cpp:3671 queue callback instead of returning a catchable OOM (PyTorch cannot recover)

[ChangyiYang]

Summary

On ROCm, when device memory is exhausted while an RCCL collective is running, the process is terminated with SIGABRT and no catchable error is delivered to the application. PyTorch's normal OOM handling (release_cached_blocks() + retry, or raising torch.OutOfMemoryError) never runs.

In the reproducer below the process is killed while 252 GB of the 256 GB device is held as reclaimable cache inside PyTorch's caching allocator — memory that a catchable error would have allowed PyTorch to release.

:0:rocdevice.cpp :3671: Callback: Queue 0x... Aborting with error :
  HSA_STATUS_ERROR_OUT_OF_RESOURCES: The runtime failed to allocate the necessary resources...
  Code: 0x1008  Available Free mem : 4 MB

Environment

• AMD Instinct MI325X (gfx942), 256 GB, 4 GPUs, single node
• Reproduced identically on two ROCm versions:
  • ROCm 7.0.0 / HIP 7.0.51831, PyTorch 2.9.0a0+git7bcbafe
  • ROCm 7.2.0, PyTorch 2.9.1+rocm7.2.0

ROCm	SIGABRT	collectives completed	catchable error
7.0.0	yes	0 / 20	none
7.2.0	yes	0 / 20	none

Same error code, same failure point (the first collective), and the same ~252 GB
of reclaimable PyTorch cache in both cases.

Reproducer

Fill the device, then hand all of it to PyTorch's cache with del (on ROCm it is not returned to the device: expandable_segments is unsupported here and garbage_collection_threshold defaults to 0). After that the script issues only collectives — PyTorch performs no further allocation, so it cannot be the caller that hits the exhausted device; RCCL is.

# rccl_only_repro.py — torchrun --nproc_per_node=4 rccl_only_repro.py
#   EMPTY_CACHE=0 (default) -> abort on the first collective
#   EMPTY_CACHE=1           -> all collectives succeed (same memory, simply returned to the device)
import os, torch, torch.distributed as dist
GB = 1024**3

def state():
    free = torch.cuda.mem_get_info()[0]/GB
    resv, live = torch.cuda.memory_reserved()/GB, torch.cuda.memory_allocated()/GB
    return f"device_free={free:.3f}GB reserved={resv:.1f}GB live={live:.2f}GB reclaimable={resv-live:.1f}GB"

torch.cuda.set_device(int(os.environ["LOCAL_RANK"]))
dist.init_process_group(backend="nccl")
rank = dist.get_rank()

N = 512*1024*1024//2                                     # bf16 elements
send = torch.ones(N, dtype=torch.bfloat16, device="cuda")
recv = torch.empty(N*dist.get_world_size(), dtype=torch.bfloat16, device="cuda")
dist.all_reduce(send); torch.cuda.synchronize(); dist.barrier()   # warm the communicator

blocks, chunk = [], 8*GB                                  # 1) fill the device
while chunk >= 32*1024*1024:
    try: blocks.append(torch.empty(int(chunk), dtype=torch.uint8, device="cuda"))
    except RuntimeError: chunk //= 2
del blocks                                                # 2) -> PyTorch cache; device stays full
if rank == 0: print(f"[after del]      {state()}", flush=True)

if os.environ.get("EMPTY_CACHE", "0") == "1":             # 3) optional: give it back to the device
    torch.cuda.empty_cache()
    if rank == 0: print(f"[empty_cache()]  {state()}", flush=True)
dist.barrier()

for i in range(20):                                       # 4) collectives only, no torch allocation
    dist.all_gather_into_tensor(recv, send)
    torch.cuda.synchronize()
    if rank == 0: print(f"collective {i} OK", flush=True)
if rank == 0: print("ALL 20 COLLECTIVES COMPLETED", flush=True)

Running this exact script both ways isolates the failure to the collective, and shows the memory was reclaimable the whole time:

EMPTY_CACHE=0 (default) — aborts:

[after del]      device_free=0.020GB reserved=254.6GB live=2.50GB reclaimable=252.1GB
:0:rocdevice.cpp :3671: Callback: Queue ... Aborting with error : HSA_STATUS_ERROR_OUT_OF_RESOURCES ... Available Free mem : 4 MB

• SIGABRT on all 4 ranks
• zero collective N OK lines — it dies on the very first collective (i = 0)
• no Python exception, no application-level traceback

EMPTY_CACHE=1 — identical run, one torch.cuda.empty_cache() before the collectives:

[after del]      device_free=0.020GB   reserved=254.6GB live=2.50GB reclaimable=252.1GB
[empty_cache()]  device_free=252.082GB reserved=2.5GB   live=2.50GB reclaimable=0.0GB
collective 0 OK
...
collective 19 OK
ALL 20 COLLECTIVES COMPLETED

• no abort, all 20 collectives succeed, process exits normally

The only difference between the two runs is whether PyTorch's cached blocks were handed back to the device beforehand. The 252 GB was reclaimable in both cases — in the failing run nothing ever asked for it, because the process was terminated instead of receiving an error.

Backtrace at the abort

Captured with an LD_PRELOAD interposer on abort():

<interposer>(abort+0x3d)
/opt/rocm/lib/libamdhip64.so.7(+0x42f71)
/opt/rocm/lib/libhsa-runtime64.so.1(+0x5e35c)
/opt/rocm/lib/libhsa-runtime64.so.1(+0x9273b)
/opt/rocm/lib/libhsa-runtime64.so.1(+0x32991)
/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3)      <- start_thread
/lib/x86_64-linux-gnu/libc.so.6(+0x126850)     <- clone

The stack contains no PyTorch or Python frames: the abort happens on an HSA runtime thread, so no application-level try/catch can intercept it.

Also tried

HIP_SKIP_ABORT_ON_GPU_ERROR=1, and that combined with ulimit -c 0, did not prevent the abort — all ranks still died with SIGABRT. Verified from inside the worker processes that both settings were actually in effect (HIP_SKIP_ABORT_ON_GPU_ERROR='1', RLIMIT_CORE=(0, 0)).

Request

Could an out-of-memory condition on this path be reported as a recoverable status (for example surfaced as hipErrorOutOfMemory on the associated stream/queue) instead of terminating the process?

For long-running training jobs this is the difference between a recoverable OOM — drop the batch, empty the cache, retry — and losing a multi-hour run. The framework-side recovery already exists and works when PyTorch's own allocation is the one that fails; on this path it simply never gets the chance.

[ChangyiYang]

Cross-platform data point: the identical sequence completes cleanly on NVIDIA + NCCL

Ran the reproducer from this issue verbatim on NVIDIA hardware for comparison:

Environment: 4× H100 80GB (single node), CUDA 13.0 driver stack, PyTorch 2.x + NCCL, torchrun --nproc_per_node=4.

variant	device state before collectives	result
EMPTY_CACHE=0 (the abort path on ROCm)	device_free=0.026GB reserved=77.7GB live=2.50GB reclaimable=75.2GB	ALL 20 COLLECTIVES COMPLETED — no error, no abort
EMPTY_CACHE=1	device_free=75.2GB after empty_cache()	ALL 20 COLLECTIVES COMPLETED

So with the device driven to 26 MB free and 75.2 GB held as reclaimable cache inside PyTorch's caching allocator, NCCL runs the same 20 all_gather_into_tensor collectives without incident on the CUDA stack — the same sequence that hard-aborts via the rocdevice.cpp:3671 queue callback on ROCm.

This supports the issue's framing: the failure mode is not an inherent consequence of running collectives on an exhausted device, but a ROCm runtime behavior — and a catchable error (letting PyTorch release its cache and retry, as its allocator is designed to do) would make the platforms behaviorally equivalent.