[latentcollapse]

Summary

Adds quantized_qwen35.rs — a GGUF-quantized inference implementation for Qwen3.5, the first working Gated DeltaNet + partial RoPE model in Candle.

Qwen3.5 is architecturally distinct from Qwen3: it interleaves full-attention transformer layers with linear-attention SSM layers (Gated DeltaNet), uses grouped-query attention with partial RoPE (rope.dimension_count=64), and has paired per-head RMSNorm on K/V. Getting coherent output required finding and fixing seven independent bugs. Each is documented below.

Update (Apr 14): Added chunked parallel GatedDeltaNet scan for prefill, contributed by @co-mapis. Replaces the O(T) sequential recurrence with an O(T/64) chunked WY/Woodbury implementation. See new section below.

Chunked GatedDeltaNet Scan (prefill)

candle-transformers/src/models/qwen3_5_linear_attn_scan.rs — contributed by @co-mapis, adapted from the HuggingFace torch_chunk_gated_delta_rule reference.

What it does:

• gated_delta_rule_chunked — O(T/64) chunked parallel scan for prefill (seq_len > 1). Processes tokens in chunks of 64 using the WY/Woodbury representation with forward substitution to solve (I + A) per chunk. State is propagated sequentially between chunks (causality requires this), but intra-chunk work is parallelised via batched GEMMs.
• sequential_step — single-step decode (seq_len = 1), extracted from the original loop.
• Dispatch in SSMWeights::forward: prefill → chunked scan, decode → sequential step.

Float32 overflow fix (baked in):
For long sequences with Qwen3.5's decay values (~−6.72 per step), exp(g_cumsum[i] − g_cumsum[j]) in the upper triangle reaches exp(~275) → +inf for t ≥ 15 tokens. inf * 0 = NaN. Fix: zero the upper triangle in log-space before exp, then re-zero after — safe because lower-triangular entries are always ≤ 0 so exp ≤ 1.

Validation:

• 9/9 unit tests pass on CPU (chunked vs sequential reference, max diff < 1e-4)
• Includes overflow regression test with log_g ≈ −6.72 (real Qwen3.5-0.8B values)
• Smoke tested end-to-end: Qwen3.5-0.8B-BF16, prompt "What is 2+2?", output: "Four" ✓

Bug elimination log

All bugs were diagnosed by comparing against modeling_qwen3_5.py from Hugging Face as the ground-truth oracle. The canonical probe throughout was: "What is the capital of France?" — expected: Paris.

Bug 1 — SSM decay sign double-negation
Raw GGUF ssm_a values are negative. The original path negated them again, turning exponential decay into exponential growth. Fixed: load ssm_a as-is.

Bug 2 — Depthwise causal conv applied backwards
apply_conv1d() kernel orientation mismatched PyTorch Conv1d left-padding convention. Fixed: reverse the kernel before application.

Bug 3 — Full-attention gate dropped
attn_q.weight in GGUF encodes both query and gate concatenated. The original path used the full tensor as query, discarding the gate. Fixed: split q_proj into (query, gate) and apply sigmoid(gate) before o_proj.

Bug 4 — Partial RoPE not honored
rope.dimension_count=64 from GGUF metadata was read but not used — RoPE was applied to the full head dimension. Fixed: apply RoPE only to the first 64 dims per head.

Bug 5 — <think> preamble masking failure
An empty <think> block in the prompt triggered chain-of-thought analysis mode. Removed from the generation path.

Bug 6 — GQA paired RMSNorm hardcoded 8×8 instead of 2×8
head_count_kv=2 was not threaded through norm construction; norm was applied to the wrong grouping. This caused copy-attention dominance and consistent France→United States substitution. Fixed: read head_count_kv from GGUF metadata and thread through norm construction.

Bug 7 — SSM norm epsilon hardcoded 1e-5 instead of 1e-6
qwen35.attention.layer_norm_rms_epsilon=1e-6 from GGUF metadata was ignored; 1e-5 was used globally. This distorted 18/24 SSM layers and was the fix that restored Paris on raw factual probe. Also fixed here: KV-head count was being inferred from blk.0 (an SSM layer) instead of from qwen35.attention.head_count_kv metadata directly.

Key lesson: No hardcoded dimensions, grouping factors, or norm conventions. Pull everything from GGUF metadata/config, or treat the path as suspect.

Validation

Tested on RTX 5060 (8 GB VRAM), all Unsloth quantizations, temperature=0.0:

Model	Format	capital of France	2+2	hello
Qwen3.5-0.8B	BF16	Paris	4	coherent
Qwen3.5-2B	UD-Q8_0	Paris	4	coherent
Qwen3.5-4B	UD-Q4_K_XL	Paris	4	coherent
Qwen3.5-9B	Q4_K_M	Paris	4	coherent

9B was validated with CPU/GPU split loading (32 GPU layers, embeddings on CPU, ~6.5 GB VRAM). That split-loading API is a separate follow-up PR.

Solo dev, limited hardware — open to corrections on larger variants or alternative quantization families.

Files changed

• candle-transformers/src/models/quantized_qwen35.rs — new file, GGUF-quantized hybrid SSM/attention model
• candle-transformers/src/models/qwen3_5_linear_attn_scan.rs — new file, chunked GatedDeltaNet scan (contributed by @co-mapis)
• candle-transformers/src/models/mod.rs — register both new modules
• candle-examples/examples/quantized-qwen35/main.rs — runnable example (local GGUF + tokenizer paths)

[latentcollapse]

PyTorch comparison — RTX 5060 (8151 MiB), CUDA 12.8, bf16

Model	PyTorch 2.10	Candle (this PR)	Delta
Qwen3.5-0.8B	15.1 tok/s	89.3 tok/s	+5.9x
Qwen3.5-2B	15.4 tok/s	52.1 tok/s	+3.4x

Worth noting: PyTorch logs "The fast path is not available — falling back to torch implementation" for this architecture. Qwen3.5's GatedDeltaNet layers require flash-linear-attention and causal-conv1d for the optimized PyTorch path, neither of which are installed by default. The Candle numbers above require no additional CUDA kernel dependencies beyond the standard toolkit.

If someone has flash-linear-attention installed, happy to rerun for a complete picture. But for the vast majority of users, this is the relevant comparison.

[co-esquie]

If it helps, I made a working flash-linear-attention implementation for GatedDeltaNet layers which is considered as working and heavily tested on CUDA and CPU backends here https://github.com/ericcurtin/inferrs/blob/main/inferrs-models/src/models/qwen3_5_linear_attn_scan.rs using candle only

[latentcollapse]

Awesome work! Does your Flash-linear-attention impl handle the full hybrid architecture (both SSM layers and the attention layers) or just the SSM path?

Edit: I folded it into my Candle PR because it's clean and we just ran the smoke test. Worked great. Really nice job on that, btw

[co-esquie]

I've also implemented Multi-Token Prediction speculative decoding llmmanorg/llmman#198 llmmanorg/llmman#202 if it also helps

[lucasjinreal]

When will this merge?

[latentcollapse]

When will this merge?

Changes can be cleanly merged, but I think it's on the candle engineers to merge it when they're satisfied. Unless I haven't hit some part of my obligation yet

[davidlandais]

Hi, first off, thanks a lot for this. Really appreciate the work.

I wanted to try GPU-only inference on my machine and hit two things on current HEAD. Sharing a minimal repro + fixes in case they're useful.

Environment: RTX 4090 (Ada, sm_89), CUDA 13.0, cudarc 0.19.8, Windows + MSVC. Model: unsloth/Qwen3.5-0.8B-GGUF (tried both Q4_K_M and BF16).

1. Build blocker, two calls to methods that don't exist

ModelWeights::save_prefix_kv_cache / restore_prefix_kv_cache call LayerWeights::{save,restore}_prefix_kv_cache, which aren't defined, so the crate doesn't compile (E0599). Nothing else references them, and the (Option<Tensor>, Option<Tensor>) signature doesn't carry the SSM recurrent state, so they look like WIP. Removing them (or implementing the per-layer methods) unblocks the build.

2. GPU dtype mismatch at runtime

Once it built, GPU inference failed on the very first layer, while --cpu worked fine and produced correct output:

Error: layer 0 failed: ln1 failed: dtype mismatch in binary op
# and, once past ln1:
Error: dtype mismatch in matmul, lhs: F16, rhs: F32

The per-layer input upcast is gated on is_cpu():

let x = if self.device.is_cpu() && x.dtype() != DType::F32 {
    x.to_dtype(DType::F32)?
} else { x };

This guard has been there since the initial commit and was harmless while the SSM ran in the model dtype. Since f6a3eb9 (Apr 14, "chunked GatedDeltaNet parallel scan") the SSM runs in F32 for overflow stability. So on GPU, where this guard skips the upcast, F16 activations now collide with the F32 SSM / dequantized weights. Your Apr 13 GPU benchmark predates f6a3eb9, which is consistent with a regression landing there.

Applying the upcast on all devices fixes GPU and leaves CPU behavior unchanged (verified on both, correct output, ~54 tok/s decode on the 4090):

let x = if x.dtype() != DType::F32 {
    x.to_dtype(DType::F32)?
} else { x };

GPU throughput after the fix (RTX 4090)

All three run correctly on GPU with the one-line change above (global = all tokens ÷ total wall time, prefill + decode):

Model	Quant	prefill (tok/s)	global (tok/s)
Qwen3.5-0.8B	BF16	86.7	72.7
Qwen3.5-2B	Q4_K_M	81.5	59.2
Qwen3.5-4B	Q4_K_M	67.6	44.4

(Sidenote: the Unsloth UD-* "dynamic" quants fail to load with unknown dtype for tensor N | standard Q4_K_M/BF16 load fine.)

[latentcollapse]

davidlandais thank you so much. I'm gonna do a review and fix those issues. I appreciate the feedback. It's definitely not perfect yet, but it's a mostly completed base so far for anyone wanting to run Qwen3.5

[alytaphoenix]

Tested this branch on Metal (Apple Silicon, M4 Pro) since the thread so far only shows CUDA/CPU results — hit two backend gaps, both fixed, neither in the DeltaNet scan itself.

1. RoPE inv_freq built as f64, cast to F16 — no such Metal kernel

Error: Metal contiguous to_dtype F64 F16 not implemented

RotaryEmbedding::new builds inv_freq as Vec<f64> (via Tensor::from_vec) then calls .to_dtype(dtype). candle's Metal backend has no to_dtype(F64, F16) kernel. Fix: compute the frequencies as f32 instead — the f64 was only ever used for precision in the scalar powf computation, never needed at the tensor level.

let inv_freq: Vec<f32> = (0..dim)
    .step_by(2)
    .map(|i| (1f64 / rope_theta.powf(i as f64 / dim as f64)) as f32)
    .collect();

2. RMSNorm's pre-norm F32 cast was CPU-only

Error: rmsnorm is not implemented for F16 F32

In the decoder layer's forward, the cast to F32 before ln1 is gated on self.device.is_cpu() — true for CPU (which needs it), silently false for Metal (which also needs it, just wasn't exercised yet). Dropping the is_cpu() condition so it applies on every backend fixes it:

let x = if x.dtype() != DType::F32 {
    x.to_dtype(DType::F32)?
} else {
    x

Result after both fixes: Qwen3.5-0.8B-Q4_K_M.gguf runs correctly on Metal — same two probes already used in this thread ("What is 2+2?" → 4, "What is the capital of France?" → Paris), 63-133 t/s once shaders warm up.

Found these while testing with Claude Code — happy to share the full diff against this branch if useful, or just leave this here as a pointer. Really nice work on the chunked scan, by the way — matching it against the HF reference layer-by-layer must have been a slog.

[astorise]

Thanks for the Qwen3.5 work. PR #3396 already introduces the public dense and quantized Qwen3.5 implementation under qwen3_5 / quantized_qwen3_5, so we should avoid maintaining a second quantized_qwen35 API and example.
However, the chunked GatedDeltaNet prefill scan in this PR is a valuable improvement: it replaces the per-token O(T) recurrence with a chunked scan and includes validation for prefill-to-decode state handoff. The preferred integration is therefore to rebase on #3396 and port these algorithmic improvements into quantized_qwen3_5, while keeping a single public quantized implementation.
I applied this consolidation in my fork: astorise@dcc06bb

[latentcollapse]

Sorry for the delayed response. I missed this notification while working through another project. Thanks for the review and for pointing me at #3396. I’ll rebase onto it, narrow the PR to the incremental parallel-scan, testing, and selective dispatch changes, then rerun the CPU/CUDA checks and update the branch.

…

On Fri, Jul 31, 2026 at 9:47 AM Sébastien ASTORI ***@***.***> wrote: *astorise* left a comment (huggingface/candle#3461) <#3461 (comment)> Thanks for the Qwen3.5 improvements. PR #3396 <#3396> already introduces dense and quantized Qwen3.5 support, including quantized_qwen3_5. Could this PR be rebased on top of #3396 <#3396> and narrowed to its incremental improvements—such as the parallel scan, tests, and selective CPU/GPU loading—instead of introducing a second quantized_qwen35 implementation and example? This would avoid duplicate public APIs and two divergent implementations to maintain. I applied this cleanup in my fork while preserving the implementation from #3396 <#3396>: ***@***.*** <astorise@0b7adb29> — Reply to this email directly, view it on GitHub <#3461?email_source=notifications&email_token=B3NUFUMRX3WMFA6FUJXR64T5HSPO3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUGM2TMNZUG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KYZTPN52GK4S7MNWGSY3L#issuecomment-5143567475>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/B3NUFUIAYUM5SEECK4ZNHU35HSPO3AVCNFSNUABFKJSXA33TNF2G64TZHM3DKNJXHE3TQNBYHNEXG43VMU5TIMRUGU4DQOJQGI4KC5QC> . Triage notifications, keep track of coding agent tasks and review pull requests on the go with GitHub Mobile for iOS <https://github.com/notifications/mobile/ios/B3NUFUMTKR5VFTR2P4HJVJD5HSPO3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUGM2TMNZUG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2KUZTPN52GK4S7NFXXG> and Android <https://github.com/notifications/mobile/android/B3NUFUPFJZ5QWLUDYYIZJWD5HSPO3A5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTKMJUGM2TMNZUG422M4TFMFZW63VGMF2XI2DPOKSWK5TFNZ2K4ZTPN52GK4S7MFXGI4TPNFSA>. Download it today! You are receiving this because you authored the thread.Message ID: ***@***.***>

[lucasjinreal]

Qwen3.5 (including vision part), Qwen3.6, Qwen3.8 all supported in: https://github.com/lucasjinreal/Crane/ , also with GGUF support.

[AlpineVibrations]

Qwen3.5 (including vision part), Qwen3.6, Qwen3.8 all supported in: https://github.com/lucasjinreal/Crane/ , also with GGUF support.

this library looks awesome. and it uses candle. and wow its really that much faster. gotta try qwen 3.8 on my MacBook. thanks.

[lucasjinreal]

@AlpineVibrations we also supported many onnx model by extending onnx op support range, which is currently candle lack of. And we also provides many kernels with extremely crafted speed. What's more, Crane is very fast at support new models, just open an issue, someone would get your model supported.

[AlpineVibrations]

cool. I just opened an issue for responses api support. been trying out the chat completions. pretty cool so far.

[AlpineVibrations]

also cant find the right quantized qwen 3.8 to run...

[latentcollapse]

candle-transformers: add chunked GatedDeltaNet scan for Qwen3.5 prefill

Summary

Adds a pure-Candle chunked GatedDeltaNet scan for Qwen3.5-style linear-attention prefill.

This PR was originally a larger Qwen3.5 GGUF implementation. It has since been rebased on current main and deliberately narrowed to the reusable scan primitive plus its validation coverage, so the current diff is only:

• candle-transformers/src/models/qwen3_5_linear_attn_scan.rs
• candle-transformers/src/models/mod.rs

What this adds

• gated_delta_rule_chunked for multi-token prefill using a chunked WY/Woodbury formulation.
• sequential_step for single-token decode.
• Recurrent-state propagation across chunk boundaries.
• CUDA-friendly reshaping around batched GEMMs to avoid problematic non-contiguous high-rank matmul paths.
• Log-space masking before exp to prevent float32 overflow/NaN in the upper triangle for strong decay values.
• Registration of the new module in candle-transformers::models.

The implementation uses a chunk size of 64 and is adapted from the Qwen3.5 torch_chunk_gated_delta_rule reference. The original scan contribution was based on work shared by @co-mapis.

Validation

The module includes regression and parity tests covering:

• chunked output/state vs. a sequential reference on CPU;
• CUDA vs. CPU parity when CUDA is available;
• sequence lengths below, at, and above the 64-token chunk boundary;
• multi-chunk prefill;
• batch sizes greater than 1;
• realistic Qwen3.5 head dimensions;
• strong-decay values around log_g ≈ -6.72 that previously exposed exp overflow;
• chunked-prefill → sequential-decode state handoff.

The numerical comparisons use tight tolerances and explicitly check for non-finite CUDA outputs in the overflow-sensitive paths.

Scope

This PR intentionally no longer contains the earlier full GGUF Qwen3.5 model implementation, CLI example, or unrelated backend/model-loading changes. Those were removed during the rebase/scope reduction so this PR can be reviewed as a focused linear-attention primitive.

The branch is rebased on current upstream main and is intended to be independently reviewable and mergeable.