failed to run on R6000 prof

[gaowayne]

hello author
I run the same setup that works even well on 1xL40s, but failed to run R6000 server with error below

(ds4redis1) root@salab-hpedl380g11-05:/mnt/tlc/wayne/ds4redis1/ds4# ./ds4-server     --ctx 1000000     --kv-disk-dir /mnt/qlc1/nfsrdma/dsv4-kv     --kv-disk-space-mb 2621440 --debug
ds4: Linux cuda backend set oom_score_adj=1000
ds4: CUDA backend initialized on NVIDIA RTX PRO 6000 Blackwell Server Edition (sm_120)
ds4: CUDA host registration skipped: operation not supported
ds4: CUDA preparing model tensor mappings: 80.24 GiB
ds4: CUDA q8 fp16 cache budget exhausted; using q8 kernels (request=64.00 MiB cached=5.66 GiB limit=12.00 GiB free=4.75 GiB reserve=4.75 GiB total=94.97 GiB)
ds4: CUDA startup model preparation covered 80.76 GiB of tensor spans in 7.835s
ds4: cuda backend initialized for graph diagnostics
0714 06:04:00 ds4-server: context buffers 21302.79 MiB (ctx=1000000, backend=cuda, prefill_chunk=4096, raw_kv_rows=4352, compressed_kv_rows=250002)
ds4: CUDA using managed KV cache for ctx=1000000 (kv cache 13.17 GiB, context buffers 20.80 GiB); this may degrade performance but is needed for very large contexts
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
ds4: CUDA tensor alloc failed: out of memory
0714 06:04:00 ds4-server: failed to create cuda session

(ds4redis1) root@salab-hpedl380g11-05:/mnt/tlc/wayne/ds4redis1/ds4# nvidia-smi
Tue Jul 14 06:04:13 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.159.03             Driver Version: 580.159.03     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX PRO 6000 Blac...    Off |   00000000:8A:00.0 Off |                    0 |
| N/A   42C    P8             35W /  450W |       0MiB /  97887MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+
(ds4redis1) root@salab-hpedl380g11-05:/mnt/tlc/wayne/ds4redis1/ds4#