Environment
- GPU: AMD Radeon RX 9060 XT 16GB (gfx1200)
- Host: Windows 11 Pro 10.0.26200, WSL2 Ubuntu 24.04
- ROCm 7.2.0 (
amdgpu-install_7.2.70200, --usecase=wsl,rocm --no-dkms)
- Workload: PaddlePaddle built from source for gfx1200 (training a 15M-param OCR model)
Symptom
Training freezes (or aborts) at a varying iteration (observed 80–280
across 13 identical runs with a fixed data order). Two failure modes,
same root:
- Hang/livelock — main thread stuck in
GpuMemcpySync →
libamdhip64 → libhsa-runtime64, thread wchan =
dxgkio_wait_sync_object_cpu. An ioctl trace shows thousands of
LX_DXWAITFORSYNCHRONIZATIONOBJECTFROMCPU calls on a fence that never
signals.
- SIGABRT — when the assert catches it first:
pid:380 [CreateContext] fail c000000d
python: ./sources/wsl/libhsakmt/src/wddm/queue.cpp:1078:
wsl::thunk::SDMAQueue::SDMAQueue(...): Assertion `ret' failed.
Analysis
An ioctl trace on /dev/dxg over a full run shows 4
LX_DXCREATECONTEXTVIRTUAL calls at startup, all succeeding — then one
more mid-run, which fails with c000000d (STATUS_INVALID_PARAMETER).
The mid-run context creation happens when the HSA runtime decides to
create an additional SDMA/hardware queue, which it only does when GPU
copies are issued concurrently from two threads (an async dataloader
thread + the main thread). The failing context's queue then never
signals its fence → livelock.
Controlled isolation:
- Same training loop, single-threaded feeding: survives 2000+ iterations.
- Async (two-thread) feeding: dies at iter ~70–280, every run.
HSA_ENABLE_SDMA=0: delays failure (~420) but a later blit-queue
creation dies the same way.- Workaround that fully fixes it: keep all GPU copies on one thread
(dataloader feeding CPU tensors). 3×500-iteration runs + a full
4-epoch training with zero hangs.
Relation to #33
Same failure signature (CreateContext → c000000d,
dxgkio_create_context_virtual: invalid host handle, gfx1200,
9070 XT reportedly unaffected) — but #33 fails at init on a dual-GPU
box, while here the initial contexts succeed and only a context created
after warmup fails. So the gfx1200 path in the closed thunk seems to
mishandle context creation in some state reached after initialization,
not merely at startup.
Happy to provide the full strace ioctl log, py-spy stacks, or run
diagnostics on request.
Environment
amdgpu-install_7.2.70200,--usecase=wsl,rocm --no-dkms)Symptom
Training freezes (or aborts) at a varying iteration (observed 80–280
across 13 identical runs with a fixed data order). Two failure modes,
same root:
GpuMemcpySync→libamdhip64→libhsa-runtime64, thread wchan =dxgkio_wait_sync_object_cpu. An ioctl trace shows thousands ofLX_DXWAITFORSYNCHRONIZATIONOBJECTFROMCPUcalls on a fence that neversignals.
Analysis
An ioctl trace on
/dev/dxgover a full run shows 4LX_DXCREATECONTEXTVIRTUALcalls at startup, all succeeding — then onemore mid-run, which fails with
c000000d(STATUS_INVALID_PARAMETER).The mid-run context creation happens when the HSA runtime decides to
create an additional SDMA/hardware queue, which it only does when GPU
copies are issued concurrently from two threads (an async dataloader
thread + the main thread). The failing context's queue then never
signals its fence → livelock.
Controlled isolation:
HSA_ENABLE_SDMA=0: delays failure (~420) but a later blit-queuecreation dies the same way.
(dataloader feeding CPU tensors). 3×500-iteration runs + a full
4-epoch training with zero hangs.
Relation to #33
Same failure signature (
CreateContext→c000000d,dxgkio_create_context_virtual: invalid host handle, gfx1200,9070 XT reportedly unaffected) — but #33 fails at init on a dual-GPU
box, while here the initial contexts succeed and only a context created
after warmup fails. So the gfx1200 path in the closed thunk seems to
mishandle context creation in some state reached after initialization,
not merely at startup.
Happy to provide the full strace ioctl log, py-spy stacks, or run
diagnostics on request.