Repository navigation
candle-core: add IQ4_XS (GGUF type 23) quantized dtype support - #3462
Open
latentcollapse wants to merge 7 commits into
Open
latentcollapse wants to merge 7 commits into
latentcollapse wants to merge 7 commits into
Conversation
Adds quantized_qwen35.rs implementing Qwen3.5 hybrid Gated DeltaNet SSM plus full-attention architecture for GGUF inference. Fixes eight correctness bugs from the initial implementation that caused coherent but factually wrong output, including larger-variant SSM value-head repeat ordering. Validated on Qwen3.5-0.8B-BF16, Qwen3.5-2B-Q8, and Qwen3.5-9B-Q4_K_M GGUFs. All three probes (raw factual, chat arithmetic, chat greeting) return correct output with greedy decoding.
- quantized_qwen35: add from_gguf_with_layer_limit for CPU/GPU split loading - set_device() helper to switch target device mid-load - keep_embeddings_on_device flag for embedding placement control - layer-by-layer device assignment up to max_device_layers - candle-core/quantized: IQ* dtype registration groundwork (ggml_file, mod, k_quants, cuda, metal) - candle-core/tests: quantized dtype coverage additions - candle-nn/kv_cache: supporting changes for budget-aware inference path Validated: Qwen3.5-9B-Q4_K_M loads cleanly at --vram-budget 6GB on RTX 5060 (8GB) 32 GPU layers, embeddings on CPU. Paris/2+2/hello all correct. Co-Authored-By: Codex <noreply@openai.com>
…2S routing Q1_0_g128 (GGUF dtype 41) is a PrismML/Bonsai-specific format from a private llama.cpp fork. It does not belong in upstream Candle. Metal GPU matmul for I2S and IQ4_XS now returns an explicit error instead of silently routing to kernel_mul_mv_f32_f32 (which would produce garbage). Both types are handled correctly via QMatMul::forward CPU dequant path. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
GGML_TYPE_IQ4_NL_4_4 (type 36) was removed from gguf files upstream. BlockI2S was mapped to this tombstoned type and its implementation did not match the actual IQ2_S format (type 22). No real GGUF file will contain type-36 tensors. D-166 PR scope is now: IQ4_XS (type 23) dtype support + CUDA CPU-fallback dequant for unsupported types. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
latentcollapse
marked this pull request as ready for review
April 19, 2026 01:20
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds support for the IQ4_XS (GGUF type 23) quantized dtype across the Candle stack.
GgmlDType::IQ4_XSvariant +BlockIQ4XSstruct withGgmlTypeimplQMatMul::forwardvia CPU dequant path (no Metal/CUDA GPU kernel exists upstream for this type; routes todequantize_f16+ matmul on CPU)Q1_0_g128(GGUF type 41): PrismML/Bonsai proprietary format — does not belong upstreamI2S(GGUF type 36): was mapped toIQ4_NL_4_4, which has been removed from the gguf specNote: This PR is based on top of #3461 (quantized Qwen3.5 support). The first commit in this branch (
07c07d49) is the Qwen3.5 model; the IQ4_XS changes begin at the second commit. Please review/merge #3461 first and then rebase this branch — or I can rebase after #3461 lands.Test plan
cargo check -p candle-corepassescargo check -p candle-transformerspassescargo test -p candle-core iq4_xs_dequant_and_matmul_fallbackpassesQMatMul::forwardwith IQ4_XS correctly routes to CPU dequant pathcargo check -p candle-metal-kernelsskipped locally (Linux)