[0z5a]

Summary

Four fixes to prefill–decode disaggregation, measured on one GH200 node with the prefill engine on GPU 0 and the decode engine on GPU 1:

1. Huginn coordinator config: superseded by [Bugfix][M9] Restore model-neutral PD coordinator metadata #140. This branch used to rebuild the config class from the model_type the worker reports, because the coordinator built every model with OuroConfig. main now resolves the model configuration at the entry point, passes it to the workers and rejects Huginn in PD before spawning them; the merge of main keeps that and drops this change.
2. Coalesced transfers. KV segments contiguous on both sides become one descriptor, and each progress pass fills the in-flight budget instead of sending one chunk per request. The overlap counter now also credits transfers issued during a prefill computation that drained before the next chunk was computed; the wavefront overlap GPU test counted 0 before.
3. UCX error handling off, without a switch. The UCX backend is created with ucx_error_handling_mode="none"; the VLLM_RLT_NIXL_UCX_ERROR_HANDLING variable and nixl_backend_params() helper are gone. NIXL's default (peer) mode requires a peer failure handler, which only the network transports provide, so UCX leaves the shared-memory lanes out of intra-node RMA. With none, get_backend_params("UCX") reports the mode and UCX keeps sysv/posix shared memory next to cuda_ipc. A lost worker is reported by the control channel, which owns worker liveness, so the transfer path needs no failure handler and the choice is not a user-facing option.
4. Default transfer budget: 256 MiB chunks, 2 GiB in flight (was 64 MiB / 256 MiB). Chunks are submitted, and their completion observed, only between engine steps, so a request whose KV exceeds the in-flight budget waits extra steps after its prefill (the commit tail). 2 GiB holds a 2048-token Ouro-1.4B prompt (1.6 GB of KV). In-flight bytes are not a buffer: NIXL writes straight from the registered prefill pool into the decode pool. pd_serve's CLI defaults now read the PDConfig defaults.

Measurements

Ouro-1.4B 1P1D, 2048-token prompts, 64 output tokens, 32 requests, packed prefill, error handling none, two runs per arm in alternating order:

chunk / in flight	tok/s	TTFT p50	commit tail p50
64 MiB / 256 MiB (old default)	170–279	3.4–3.9 s	0.96–1.05 s
256 MiB / 1 GiB	254–290	3.2–3.5 s	0.68–0.90 s
256 MiB / 2 GiB (new default)	463–466	2.0 s	0.017 s
peer (default), 256 MiB / 1 GiB	185–334	3.0–3.4 s	0.74–0.81 s

Switching the link alone (peer → none, old budget, one run each) changed a 64 MiB chunk from 20.0 to 16.9 ms and left the commit tail near 1 s: the budget rounds, not the transport, set the tail. Earlier, behind per-row prefill, chunks four times larger took four times longer under the default mode. Only 2048-token prompts were measured; much longer prompts still need several rounds with the 2 GiB default. The peer row uses NIXL's default error-handling mode, which drops the shared-memory lanes at startup; a 256 MiB single VRAM write measured 580–870 GB/s in the default, peer and none constructions on the GH200 build tested here, so the mode does not turn large writes into TCP on that build.

Platforms. GH200: torch 2.14.0+cu130, Triton 3.8.0, NIXL 1.4.1. RTX 5090: torch 2.13.0+cu130, Triton 3.7.1. Models: Ouro-1.4B (24 layers, 4 loops, fixed depth) and Huginn-0125 (2 prelude + 4 recurrent + 2 coda layers, 32 loops). Burst workloads issue C requests at once with random-token prompts; session workloads replay multi-turn agent traces with the prefix cache on.

Tests

• CPU suite at 38a0d6c on GH200: 867 passed, 342 skipped; ruff passes. New tests: coalescing preserves every byte and both limits, the UCX backend is created with ucx_error_handling_mode="none" while other backends are untouched, the control channel detects an exited worker and reaps every peer, and the default budget holds a 2048-token Ouro prompt.
• GH200 transport and PD run at 38a0d6c (NIXL 1.4.1, UCX 1.22, torch 2.14.1+cu130). The test nodes expose one GPU, so create_pd's two- and four-GPU cases skip. The same PD engine was run with both worker processes on GPU 0: outputs matched a local engine, KV travelled over NIXL (86,208 bytes in 24 transfers), and killing either worker with transfers outstanding raised through the control channel in about 0.5 s (the transfer timeout is 90 s), closed both channels, reaped both workers and left no child, /dev/shm or GPU process behind.
• Earlier GPU PD tests on the two-GPU setup (head d8fece2, one prefill and one decode GPU, pytest --run-gpu tests/test_pd.py tests/test_transfer_connector.py -k "not flash"): 56 passed; main (66920da) with the same command: 52 passed.

Notes

The GPU numbers come from a development branch that contains all changes of this series; most cells are single runs, and on the shared GH200 host two runs of main at 32 requests differed by up to 16%, so smaller differences are ties. This branch alone was checked with the CPU suite, ruff and the GPU PD tests above.

Series. Draft PRs from one study of serving looped models. They are independent except #152, which is stacked on #147; several touch the same files, so later ones may need a rebase after earlier ones merge.

• [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU*up for CUDA graph decode #146 [Kernels] Bitwise-exact fused RMSNorm, RoPE and SiLU·up for graph decode: bitwise, 1.14–1.44× on Ouro
• [KV] Pack recurrent depth planes for partially recurrent models #147 [KV] Pack recurrent depth planes: bitwise, 1.88× Huginn KV capacity
• [Scheduler] Fuse fixed-depth decode loops into one submission, keeping the per-loop prefill share #148 [Scheduler] Fused decode loops with the per-loop prefill share: per-row arithmetic unchanged
• [Engine] Packed prefill resolves automatically, for Huginn too #149 [Engine] Automatic packed prefill, Huginn included: accuracy-gated, 2.0–2.4× on 2048-token prompts
• [PD] Coalesced transfers, UCX error-handling mode, larger default budget #150 [PD] Huginn coordinator config, coalesced transfers, UCX error handling, 2 GiB default budget ← this PR
• [Perf] Host-path costs: prefix-key caching, O(1) free-block count, per-run KV bookkeeping, one argmax per batch #151 [Perf] Host-path costs in admission, KV bookkeeping and sampling
• [Huginn] Opt-in prefix caching with position-keyed recurrent-state noise (stacked on #147) #152 [Huginn] Opt-in prefix caching with position-keyed noise (stacked on [KV] Pack recurrent depth planes for partially recurrent models #147): 1.91× session throughput

[bjf-frz]

Make the UCX transport configuration explicit by default.

[bjf-frz]

Please set ucx_error_handling_mode="none" directly when creating the UCX backend, and remove the environment variable and nixl_backend_params() helper. Users should not need a hidden switch to enable the faster transport path. Verify that the existing P/D control channel handles peer termination without hanging or leaking resources.

[0z5a]

Done in 38a0d6c. NixlConnector now creates the UCX backend with ucx_error_handling_mode="none" directly, and nixl_backend_params() plus VLLM_RLT_NIXL_UCX_ERROR_HANDLING are gone; other backends keep the previous construction. Two CPU tests pin both paths, including that an obsolete value of the variable no longer changes the mode. On GH200, get_backend_params("UCX") reports ucx_error_handling_mode: none, and the UCX context keeps sysv/posix shared memory next to cuda_ipc.

On peer termination: the control channel already owns worker liveness, and it ends the deployment well before any transfer timeout. StageClient.poll() sees the dead worker (exited process or closed pipe) and raises, PDEngine.step() records the failure, and _terminate() stops and joins both roles and closes both channels. test_control_channel_detects_a_worker_exit_and_reaps_every_peer covers that path without a GPU.

GH200 run at 38a0d6c (NIXL 1.4.1, UCX 1.22, torch 2.14.1+cu130): the test nodes expose one GPU, so the two- and four-GPU cases skip, and I ran the same PD engine with both workers on GPU 0. Outputs matched a local engine (86,208 bytes of KV in 24 NIXL transfers). Killing either worker with transfers outstanding returned control from step() in about 0.5 s (the transfer timeout is 90 s), with closed=True, both channels closed, both processes reaped, and no child, /dev/shm or GPU process left behind. CPU suite on GH200: 867 passed, 342 skipped; ruff passes.

One caveat, also reflected in the description: on this build a 256 MiB single VRAM write ran at 580–870 GB/s in the default, peer and none constructions, so the mode's effect here is not a large-write TCP fallback, and the peer measurement row is now labelled as NIXL's default mode rather than TCP.

[bjf-frz]

Please keep the stats interface unchanged in this PR.

[bjf-frz]

Please remove the added num_blocks, prefix_hit_tokens, and prefix_queries fields from stats() and revert the corresponding test changes. These statistics are unrelated to the transfer optimizations; let's keep this PR focused.

[0z5a]

Done in 699b6ad. stats() is back to requests and used_blocks, and the two assertions in tests/test_stage_engine.py compare that payload verbatim again, so the PR no longer touches statistics. The prefix-cache counters themselves remain in KVCacheManager, where main already defines them.

GH200 at 699b6ad: ruff passes; CPU suite 867 passed, 342 skipped; the three affected test files pass on their own (118 passed, 9 skipped). The PR description no longer mentions the statistics.

[bjf-frz]

others lgtm

[bjf-frz]

lgtm