[RFC]: Support for current v0.2.0 features on different GPU architectures

Status: RFC draft for discussion. Date: 2026-11-05.

Feedback period: at least one week after publication. The issue stays open afterward for ownership and implementation tracking.

References: PR #171 (ROCm), PR #168 (Ascend), RFC #156 (FlashLoop), RFC #111 (PD/KV transfer).

1. Motivation, goal and scope

The v0.2.0 runtime has one validated target: NVIDIA CUDA. This RFC defines what "supporting" the same feature set on another GPU architecture means, records the ROCm implementation, and sets the Ascend TODO list. It is not a kernel-portability claim: a feature is supported only when its public configuration path runs and its acceptance level is recorded.

Feature scope:

• packed prefill and prefill_uva;
• wavefront chunked prefill (WCPB);
• token-level decode pipeline;
• preemption and recovery;
• prefix cache;
• asynchronous scheduling and execution;
• synchronous self-speculation and its asynchronous variant;
• CUDA Graph replay, static buffers, multi-stream;
• mapped host metadata (UVA);
• HTTP serving.

ROCm is implemented by PR #171. Ascend is an experimental contribution in PR #168 and is intentionally TODO for most features. The Stage/Loop architecture, scheduler policies, model code and numerical contracts are unchanged; only platform capability decisions and one Triton precision argument change.

2. Baseline

Pinned for this RFC:

• vllm-rlt 444638b plus PR [ROCm] v0.2.0 feature support on AMD Instinct #171.
• ROCm: 1× MI350X VF, gfx950, 288 GiB HBM, ROCm 7.2, PyTorch 2.14.1+rocm7.2, Triton 3.5.1.
• Ascend: PR [Feature] Add experimental Ascend support for Ouro #168 branch ouro-ascend-contribution, one NPU.
• Models: Ouro-1.4B, Ouro-2.6B, Ouro-1.4B/2.6B-Thinking, Nanbeige4.2-3B and Huginn-0125 (real-checkpoint generation; suite coverage follows the shared loaders).

3. Platform contract

1. Detection. One helper decides is_rocm() / is_cuda(); call sites never test torch.version.hip directly.
2. Attention. On CUDA, auto prefers FA4/FA3/FA2, then Triton, then Torch. On ROCm it considers Triton then Torch and never imports FlashAttention; an explicit FA request keeps its interface and raises an NVIDIA-only error.
3. Packed prefill. Triton's packed kernel is the ROCm path, opt-in through prefill_uva, matching the existing capability-based selection. CUDA keeps FA4 as the packed owner.
4. UVA. Triton receives host pointers directly, so mapped host metadata is enabled only after hipHostGetDevicePointer returns the host address. VLLM_RLT_UVA=0 forces H2D copies; =1 skips the probe.
5. Device namespace. ROCm uses torch.cuda and CUDA_VISIBLE_DEVICES; no new public device flag.

4. Changes delivered on ROCm

No model code or scheduler behavior changes.

5. Feature support

Status: ✅ validated (suite + real model), 🚧 smoke/in progress, ⬜ pending/TODO. Owners are the current PR coordinators; contributors can add themselves per row.

5.1 ROCm

5.2 Ascend

The Ascend PD/KV-transfer cell is claimed by @O1range on 4× Ascend 910B3. The Ouro-1.4B Ascend cell is claimed by @KrystalRay per the RFC #69 ownership comment: BF16, synchronous, fixed four-loop, single NPU, CANN attention through torch_npu, depth-aware LAST_EXITED KV, with declared numerical tolerances and bounded accuracy/performance evidence. Multi-device PD is a separate @O1range claim; asynchronous execution, adaptive exit, self-speculation and the Nanbeige/Huginn ports are outside the Ouro-1.4B cell.

6. Ascend status and TODO

PR #168 targets BF16, fixed four-loop, synchronous greedy inference with full prefill on one NPU. It does not claim async scheduling, graphs, prefix cache, preemption, speculation, packed prefill or WCPB. The Ascend TODO list:

1. capability detection and auto backend wiring;
2. async scheduling and multi-stream;
3. graph capture/replay equivalent;
4. prefix cache and preemption/recovery;
5. synchronous and asynchronous self-speculation;
6. packed prefill and WCPB;
7. PD transport and KV transfer;
8. KV quantization and sparse decode from RFC [RFC]: FlashLoop integration — token-sparse prefill, sparse decode attention, and KV residual quantization #156;
9. serving parity and a CI runner.

Each item needs its own acceptance level; PR #168's smoke run is R0 only.

7. Configuration and environment

8. Validation and acceptance

• R0 smoke: one real checkpoint generates non-degenerate text.
• R1 suite: CPU suite plus GPU suite, with only explicit architecture/environment skips.
• R2 performance: matched workload/config, reported with device, runtime and kernel versions.
• R3 numerics: greedy equality against the Torch reference on fixed prompts; approximated paths labelled.
• R4 distributed: multi-GPU PD with output equality and transfer accounting.

ROCm reaches R1 and R2 for Ouro-1.4B, Ouro-2.6B, both Thinking checkpoints, Nanbeige4.2-3B and Huginn-0125. R3 remains open for speculative decoding (30/32 prompt equality; 0/20 GSM8K both arms). Packed prefill is located: FP32 greedy matches per-row exactly while BF16 differs by <=0.0156, so it stays opt-in. Ascend is R0.

Baseline definition: same commit, model, device and wheel set, attention_backend="triton", ExecutionConfig() defaults, same exit mode, VLLM_RLT_UVA=0, no optional feature. There was no earlier ROCm baseline, so all numbers are ROCm-relative; CUDA numbers were not measured on this lease. One warm-up generation is discarded, then the median of three runs is reported:

all excludes packed prefill. The 32-token cells are prefill-heavy; the 256-token cell is decode-heavy (baseline 6.3 ms/step, all 3.4 ms/step). Environment micro-tuning: HIP_FORCE_DEV_KERNARG=1 2781, HSA_ENABLE_INTERRUPT=0 2843 vs default 2798. PyTorch already selects hipBLASLt; forcing it did not improve throughput.

Full-feature sweep

Short context (80-token prompt, 64 output), Ouro-1.4B:

Long context and checkpoints (1k/4k/8k). runtime = async + graphs + static/pad2 + UVA + WCPB + prefix; spec adds pipeline K4/draft1. Huginn's optimized column is graph replay; Nanbeige has no spec path:

Huginn is compute-bound by 32 recurrent steps (flat across batch), graph replay adds 3-9%, and 8k is rejected by its context limit. 8k Ouro prefill: 1247 -> 11544 -> 26800 tok/s.

Per-feature warmed A/B at batch 128, one warm-up discarded, median of three. Non-spec rows use 32 output tokens and the baseline_delayed reference (970 tok/s); spec rows use 256 output tokens and the ouro baseline (1527 tok/s). The 32-output spec numbers (0.51x/0.95x/0.89x) are in the workload-dependent paragraph below.

Speculation is workload-dependent. At 32 output tokens prefill dominates and it loses; at 256 output tokens it wins on Ouro-1.4B (baseline 1527): pipeline K4/draft1 3270 in the sweep and 2842 confirmed (1.82x), block K2/draft1 2148/2251, async K2/draft1 2314 (1.48x). On Ouro-2.6B pipeline K2/draft1 is 814 vs 500 (1.63x). Greedy match against baseline is 0.66-0.80 and GSM8K scored 0/20 for both arms, so it stays opt-in. WCPB is the largest single win for non-speculative workloads.

Priority scheduling is positive: 64 low- plus 8 high-priority requests at 16 slots drop high-priority median completion from 7.80s (fcfs) to 2.22s, while total time moves 7.81s to 8.15s. Incremental KV gives 817 -> 848 tok/s (+3.8%). Sparse decode from #157 passes its ROCm suites (137 passed, 3 skipped) after merging #171; it is tracked in #157, not shipped here.

Not done on ROCm: packed engine parity, PD/KV transfer (two GPUs + ROCm NIXL/UCX), KV quantization (not on current main; RFC #156), profiling flags, multi-node, FlashAttention (NVIDIA-only).

9. Open decisions

1. Enable packed prefill by default on ROCm, or keep the prefill_uva opt-in until multi-sequence numerics are qualified?
2. Probe UVA on every platform, or keep the CUDA fast path without a probe?
3. Does Ascend need the same capability object, or a separate backend adapter?
4. Which ROCm CI runner is acceptable (self-hosted MI300/MI350 vs none)?
5. Should auto warn when prefill_uva is requested without packed support?

10. Community directions

Entry points left to the community, not implied commitments:

• ROCm PD and KV transfer (NIXL/UCX/RCCL, RFC [RFC]: PD coordination and KV transfer contracts #111);
• KV cache quantization and cross-loop residuals on gfx950 (RFC [RFC]: FlashLoop integration — token-sparse prefill, sparse decode attention, and KV residual quantization #156);
• sparse decode and FlashLoop kernels (RFC [RFC]: FlashLoop integration — token-sparse prefill, sparse decode attention, and KV residual quantization #156, PR Sparse decode attention: per-token reference lifecycle for async self-speculation #157);
• AITER/CK attention alternatives to Triton;
• graph budget/eviction behavior on ROCm;
• MI300/gfx942 qualification;
• Ascend CI and the TODO list above.

11. Ownership

Core maintainers own the platform contract, backend selection and UVA probe. Each architecture port has an owner per row in Section 5. Community PRs must pin versions and state the acceptance level they reach.