Merge llama into beellama 2#2
Open
gboddaer wants to merge 436 commits into
Open
Conversation
* metal : add f16 and bf16 support for concat operator Extend the Metal backend concat operator to support f16 and bf16 tensor types in addition to the existing f32 and i32 support. - Template kernel_concat on type T with specializations for float, half, bfloat, and int - Add type-specific pipeline getter ggml_metal_library_get_pipeline_concat() - Update device support check to allow f16 unconditionally and bf16 when device supports bfloat16 - Update dispatch to select the correct kernel specialization by type Assisted-by: pi:llama.cpp/Qwen3.6-27B * metal : extend concat operator to support f16, bf16, i8, i16 and i64 Assisted-by: pi:llama.cpp/Qwen3.6-27B
Reuse existing rope kernels with a function constant to toggle forward/backward rotation, avoiding duplicate kernel code. Assisted-by: pi:llama.cpp/Qwen3.6-27B
…ort (ggml-org#24687) * ggml: Conditionally enable power11 backend based on compiler support Guard POWER11 backend creation behind a compiler flag check for -mcpu=power11. This avoids build failures on current GCC/Clang toolchains while preserving forward compatibility once POWER11 support becomes available. * Update CMakeLists.txt ggml-cpu: Use -mcpu=power10 for P10 and P11
…4600) * fix conflict * fix format issue, rename * rm debug code * correct the file name
…org#24754) Signed-off-by: Adrien Gallouët <angt@huggingface.co>
* rename GGML_SYCL_SUPPORT_LEVEL_ZERO to GGML_SYCL_SUPPORT_LEVEL_ZERO_API, and GGML_SYCL_ENABLE_LEVEL_ZERO to GGML_SYCL_USE_LEVEL_ZERO_API * fix code format * fix error when rebase
* add mtmd_image_preproc_out * add dev docs * remove unused clip API * rm unused clip_image_f32_batch::grid * change preprocess() call signature
* server: temporary remove HF remote preset * rework remove preset.ini support * rm unused get_remote_preset_whitelist() * print warning * add docs * rm stray file
…g#24154) Throw on grammar parse failure so the server returns HTTP 400 instead of silently dropping the constraint. Add a regression test for the invalid-grammar response. Fixes ggml-org#24144
* ui : add model selector storybook stories Covers list, favorites, single-model, all status states (loading/loaded/sleeping/failed/idle), and selection states. * ui : improve model selector mobile UX with hover media queries Use @media (hover:none) to show action buttons directly on touch devices and color-code them by model status (amber=sleeping, green=loaded, muted=idle). Status dots hidden on touch. Desktop hover behavior unchanged.
* server : add last-5-seconds generation speed display * cont : clean-up --------- Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>
* wip * working * correct some limits * add field name to error message
* server: (router) fix stopping_thread potentially hang * fix windows build
Assisted-by: Codex
…ts) (ggml-org#24592) * hex-optrace: add support for optrace and instrument matmul and flash-atten code * hex-trace: improve trace event and prefetto generator * hex-trace: add new script dedicated to handling traces, specifically perfetto traces * hex-trace: add --head/--tail options to profile and trace tools * hex-trace: fix whitespaces * hex-trace: fix flake8 warnings * hex-trace: fix flake8 warnings * hmx-fa: restore q_tiles clearing * hex-profile: remove circular dep in includes * hex-trace: simplify trace sizing check * hex-profile: sort events in the summary by name
…irst) (ggml-org#24769) * add dedicated "overview" for mtmd_image_preproc_out * corrections * correct (again) * nits * nits (2)
…gml-org#24774) * server: add "X-Accel-Buffering": "no" header to streaming endpoints This header tells Nginx (as a reverse proxy) to NOT buffer responses. (only affects streaming endpoints) Without it, Nginx will break streaming with certain applications (notably the Pi coding harness).
* cuda: add GGML_OP_COL2IM_1D, follow-up to the CPU op * cuda: col2im_1d use fast_div_modulo for the index decomposition * cuda: col2im_1d tighten supports_op, type match and contiguous dst
* ggml-cpu: support K tails in Power10 MMA Q8/Q4 matmul This patch removes the requirement that K be divisible by kc in the tinyBlas_Q0_PPC tiled matmul path. Process the final K panel using its actual depth and pass the reduced panel size through packing and kernel execution. This allows more workloads to use the MMA kernel and reduces fallback to mnpack. * Apply suggestion from @taronaeo Co-authored-by: Aaron Teo <taronaeo@gmail.com> --------- Co-authored-by: Aaron Teo <taronaeo@gmail.com>
…#24755) Absorb get_slot_by_id logic into get_available_slot so slot selection is handled by a single function call. When a specific slot id is requested, the LCP similarity check still runs to enable proper prompt cache updates. Assisted-by: pi:llama.cpp/Qwen3.6-27B
added 30 commits
July 5, 2026 12:54
Missing llama_model_share_tensors caused drafter lm_head to be NULL → garbage logits → all-zero argmax → 6.7% acceptance. Fixed: 55.6% acceptance, 19.4 tok/s, correct output.
Tested n_draft_max=4 (matching fork adaptive output_len=5): - Acceptance DROPPED to 50% (was 55.6% with n_draft=15) - Speed DROPPED to 13.0 tok/s (was 19.4 tok/s) - n_draft=4 is WORSE than n_draft=15 for the merge Token trace comparison (n_draft=4, same id_last=90700): - MERGE: first_draft_ids=[8340,25,271,16] - FORK: first_draft_ids=[1817,25,271,16] Position 1 DIFFERS (8340 vs 1817); positions 2-4 match. committed_len DIFFERS: merge=17, fork=20 (prefill capture difference). CONCLUSION: The remaining 55.6%->84% gap is NOT the profit controller. The gap is the drafter producing DIFFERENT predictions due to different cross-attention context (committed_len 17 vs 20). The profit controller port would NOT close the gap. n_draft=15 is already the merge's optimal.
…aining gap Investigation findings: 1. Both merge and fork capture the SAME 17 prefill tokens (prompt_total=17, two batches [0,12] and [13,16]) — prefill capture scheduling is CORRECT. 2. The fork grows the ring to 20 (17+3 warmup) before the first draft via the adaptive profit controller (starts at n_draft=0, 3 non-speculative warmup decodes). The merge starts speculating immediately at 17. 3. Tested adding warmup (n_decoded < 3) + update_logits for non-speculative decodes: quality DROPPED to 3.3% because the merge's 4-token longer prompt (17 vs 13, chat template difference) causes different warmup tokens (id_last=25 vs fork's 7047) -> early stop, garbled output. 4. The 4-token prompt difference (17 vs 13) is a CHAT TEMPLATE difference (upstream vs fork jinja template), NOT a DFlash code issue. CONCLUSION: The remaining 55.6% vs 84% gap is due to the chat template producing a 4-token longer prompt in the merge (17 vs 13). This changes the generation context, causing different warmup tokens and different draft predictions. This is an upstream/fork template difference, not a DFlash bug. The share_tensors fix (commit 35c5f8f) is the correct DFlash fix: 6.7% -> 55.6% acceptance, 4.2 -> 19.4 tok/s, garbled -> correct output. Reverted warmup + update_logits changes (made things worse). Kept prefill trace logging (GGML_DFLASH_PREFILL_TRACE=1) as debug tool.
- Refactor server_slot: spec is now common_speculative_ptr (owned), add spec_shared (non-owning fallback), get_spec(), get_seq_id() - DFlash: each slot gets its own spec (per-slot ring/capture/seq_id) via common_speculative_init(params, ctx_tgt, ctx_dft) - Non-DFlash: use shared context-level spec (spec_shared) - Keep shared ctx_dft for DFlash (don't reset when spec is null) - Replace all spec.get() with slot.get_spec() in per-slot operations - Add llama_dflash_set_active_slot() before draft/accept/update_logits - Use slot.get_seq_id() (0 for per-slot, slot.id for shared) to avoid dparams out-of-bounds crash - Set llama_set_dflash_n_slots(ctx_tgt, n_parallel) for target context - Increase drafter n_outputs_max to account for draft tokens Single-slot: 43.6% acceptance, 5 draft cycles, correct output (no regression). Multi-slot: 4 per-slot specs created, shared drafter context, but 0 drafts generated (hidden state capture issue for slots > 0 — needs investigation).
…ts (capture issue)
…lti-slot - Add llama_dflash_allocate_slots(ctx_tgt, dflash_slots_cap) call after slot init to resize layer_hiddens for multi-slot (fork:2814, merge was missing) - Only call for dflash_slots_cap > 1 (calling with 1 breaks single-slot) - Add set_active_dflash_slot before prefill_capture_begin and flush_prefill so the correct slot's hidden buffer is accessed - Fix prefill capture scheduling: remove 'spec &&' guard so DFlash prefill capture runs even when context-level spec is null (per-slot specs) Single-slot: 59.3% acceptance (16/27), correct output — NO REGRESSION. Multi-slot: per-slot specs created, layer_hiddens resized, set_active_slot works (no 'out of range'), but flush_prefill returns 0 (GPU capture disables eval callback -> layer_hiddens empty for slots > 0). Response is garbled — needs further investigation.
… slot calls - Only call llama_dflash_allocate_slots for dflash_slots_cap > 1 (calling with 1 breaks single-slot: 5.3% -> 59.3% acceptance) - Restore set_active_dflash_slot calls (temporarily disabled for debugging) - Restore prefill capture scheduling (temporarily disabled for debugging) Single-slot: 59.3% acceptance, correct output. Multi-slot: garbled output persists even with capture/active_slot disabled. Root cause is in per-slot DFlash spec creation, not capture scheduling. Needs further investigation — possibly set_force_split_seq or common_speculative_draft interfering with target context.
Document what was empirically proven vs what remains hypothesis: - PROVEN: single-slot works (59.3%), multi-slot garbled output caused by DFlash - PROVEN: garbled output NOT caused by allocate_slots, draft calls, set_active_slot, prefill capture, or GPU capture (each disabled, still garbled) - PROVEN: allocate_slots(1) breaks single-slot, guard >1 fixes it - HYPOTHESIS: set_dflash_capture or set_force_split_seq corrupts target context - HYPOTHESIS: fork's flush_prefill reads from GPU, not CPU layer_hiddens Next: investigate DFlash impl constructor's effect on target context, disable set_dflash_capture or set_force_split_seq to isolate.
… quality regression METHODOLOGICAL LESSON: - Short tests (max_tokens=20-40) gave false confidence that single-slot DFlash worked. The first 10-20 tokens are often correct before drafts diverge. - Longer tests (max_tokens=200-400) reveal the merge DFlash produces garbled output after the first few tokens, while the fork DFlash produces correct full-length output. EVIDENCE: - Non-DFlash merge: 'The capital of France is Paris.' (perfect) - Fork DFlash: 'The capital of France is Paris.' (perfect, 26.7% acceptance) - Merge DFlash: 'onally ies7., - for :.div' (garbled, 13.5% degrading) RULE: Always test with max_tokens >= 200. Short tests mask quality regressions. CURRENT STATE: - Non-DFlash merge: STABLE (base reference) - Fork DFlash: STABLE (working reference) - Merge DFlash single-slot: QUALITY REGRESSION (garbled after first tokens) - Merge DFlash multi-slot: garbled (same issue, more severe) Single-slot quality regression is the PRIMARY BLOCKER. Testing whether per-slot spec creation (vs shared spec) is the cause.
…points Replace common_speculative_get_state/set_state (TODO stubs that do nothing) with common_speculative_ring_state_save/load (real implementations from fork) for checkpoint save/restore. The TODO stubs were losing DFlash ring buffer state on every checkpoint restore, causing quality degradation. Note: this fix is correct but NOT the root cause of the quality regression. The regression is caused by n_outputs_max sizing (see HF-037).
--spec-draft-n-max 1 produces perfect output (62.7% acceptance, correct answer). Default n_max=16 produces garbled output (43.8% acceptance, garbled after ~20 tokens). The parameter changes server_n_outputs_max (target output buffer: 2 vs 17). Large output buffer corrupts verification or graph building. Fork's adaptive controller keeps n_max small initially = correct output. Merge needs adaptive controller ported. Ring state save/load fix (HF-036) was correct but NOT the root cause.
…d doc Fixes: 1. Drafter n_outputs_max: set to max(n_parallel, 1+n_max) at line 1173 AND ensure cparams.n_outputs_max >= 17 at ctx_dft creation. Fixes output_reserve crash when target n_outputs_max is small. 2. Ring state save/load: use common_speculative_ring_state_save/load instead of TODO stubs for checkpoint save/restore (HF-036). Workaround for quality regression (HF-037): - --spec-draft-n-max 1 produces correct output (60% acceptance, coherent) - Default n_max=16 produces garbled output (large target n_outputs_max corrupts) - Root cause: large target n_outputs_max (17) somehow corrupts DFlash verification - Target n_outputs_max cap to 2 causes drafter crash (cparams propagation issue) - Needs further investigation: why does large target n_outputs_max corrupt output? Tested: single-slot --spec-draft-n-max 1, 'capital of France' prompt: REASONING: 5 coherent steps ANSWER: 'The capital of France is Paris.' Acceptance: 60% (135/225), mean len 10.00
…raph PROVEN by extensive bisection: - n_outputs_max=2 (--spec-draft-n-max 1) → correct output - n_outputs_max>=6 → garbled output (regardless of draft count) - Fork has n_outputs_max=17 and works → issue is merge-specific (342 upstream commits) - DFlash impl, eval callback, output_reserve, server_n_outputs_max all identical fork-vs-merge ROOT CAUSE: An upstream commit changed how the target context handles multiple output positions. With n_outputs_max>2, the graph corrupts hidden state capture. WORKAROUND: --spec-draft-n-max 1 (correct output, 60% acceptance, reduced speed) Removed: n_draft_max cap to 5 (didn't help — issue is n_outputs_max not draft count) Removed: target n_outputs_max cap (crashes — target batch needs n_draft+1 outputs) Removed: diagnostic logs from llama-context.cpp, speculative.cpp, server-context.cpp Next: bisect 342 upstream commits, or port adaptive controller, or investigate graph
Two hidden state capture paths identified: 1. Eval callback (CPU): works with n_max=1, garbled with n_max>2 2. GPU-embedded (GPU): BROKEN in merge at all n_max values (6.1% acceptance) The fork uses GPU-embedded path (works). The merge's GPU-embedded path is broken by 342 upstream commits that changed graph building for hidden_gpu. Root cause: allocate_tape_gpu(1,...) allocates hidden_gpu → dflash_graph_hidden_ready=true → eval callback disabled → GPU-embedded capture used → wrong hidden states in merge. Workaround: --spec-draft-n-max 1 + dflash_slots_cap > 1 guard (eval callback path, n_outputs_max=2, correct output, 60% acceptance). Next: diff allocate_hidden_gpu and graph builder between fork and merge.
…invalidation Ported from fork adb92b3:222-253: 1. qwen35.cpp: Added DFlash hidden_gpu capture code that copies layer outputs to hidden_gpu buffers during the forward pass. Without this, hidden_gpu stays empty even when allocated (HF-039 root cause). 2. llama-context.cpp: Added graph invalidation when hidden_gpu_n_seqs changes (0→1 transition from prefill to generation). 3. server-context.cpp: llama_dflash_allocate_slots now unconditional (>0 not >1). Results: graph builder executes hidden_gpu copy (448/1344 layers with hidden_n_seqs=1). With graph reuse disabled: 62.5% acceptance (up from 43.8%). Without: still 43.8% — graph cached from prefill (hidden_n_seqs=0). Graphify comparison: 643 common DFlash nodes, 55 fork-only, 107 merge-only. Missing fork functions (apply_dflash_effective_defaults) have equivalent struct defaults.
…ama_into_beellama_2
Assisted-by: OpenAI GPT-5
Assisted-by: OpenAI Codex
…Flash correctness regression)
The merge took upstream's Vulkan gated_delta_net.comp shader, which uses the
reversed recurrent-state snapshot slot convention (slot 0 = most-recent state),
while keeping the fork's delta-net graph (delta-net-base.cpp build_recurrent_attn)
and the llama_dflash_rs_writeback_slot_for_test helper (llama-context.h), which
expect the fork's convention (slot K-1 = most-recent -> cache_slot 0 = active
state for the next batch). The mismatch wrote the newest recurrent state to the
wrong cache slot, so every multi-token DFlash verify batch read a stale s0 and
the recurrent state drifted cumulatively until the target argmax flipped. First
wrong acceptance: greedy token 16 - the drafter correctly proposed 256 (== non-
DFlash greedy) but the target verify batch computed argmax 12 and rejected the
correct draft. Result: greedy DFlash diverged from greedy non-DFlash early, with
garbled/slow output (4.4 t/s on the ctx-32768 repro; ~12 t/s but divergent here).
Fix: restore gboddaer/main's snapshot-slot convention in gated_delta_net.comp:
state_in_base = (seq_id * K * H + head_id) * state_size (3D state D,K,n_seqs)
const int shift = int(n_tokens) - int(K);
target_slot = int(t) - shift; (slot K-1 = newest)
The merge host code still reads K from op_params; the graph still passes s_3d_pad
(3D) + K op-param, so K is correct. Fixed shader is byte-identical to gboddaer/main.
Verified (Qwen3.6-27B-Q4_K_M, Vulkan0, greedy temp 0, --spec-draft-n-max 3):
p1: DFlash output == non-DFlash output (IDENTICAL); DFlash 19.7 t/s vs
non-DFlash 12.6 t/s (1.56x), matching gboddaer/main (p1 identical, ~20 t/s).
p2: token-16 argmax flip fixed; greedy tokens 0-25 identical to non-DFlash.
Also includes the coherence fix that restored buildability:
- qwen35moe.cpp: add #include "llama-context.h" (dflash_hidden_gpu is defined
there; the prior capture-block port dereferences it, which did not compile).
- prior session's capture-block port (prefill_gpu / tree_mode conv dispatch /
qwen35moe hidden_gpu / delta-net-base tree GDN dispatch + build_conv_state
qkv_mixed_transposed param) which restored output coherence. (Isolation test
showed prefill_gpu does not affect correctness; the shader fix did.)
And gated investigation diagnostics (zero behavior change when env unset):
- GGML_DFLASH_QA_TRACE in common/sampling.cpp common_sampler_sample_and_accept_n
logs per draft position: draft, raw target_argmax, sampled, match, REJECT.
- [NODFLASH_TOK] trace at the non-speculative sample point in server-context.cpp.
These pinpointed the first wrong acceptance (token 16) and support Phase 2.
A second, distinct residual regression remains (server-side verify-batch
construction / recurrent-state save-restore; p2 diverges later at token 26 with a
duplicate token / position off-by-one, and p2/p3 are slow vs gboddaer/main). It is
independent of this target-graph fix and is tracked as Phase 2 in TASK_PROGRESS.md.
Assisted-by: pi
…ckup (Phase 2 diagnostic)
Phase 2 investigation checkpoint. Add a gated GGML_DFLASH_QA_TRACE diagnostic to
the DFlash verify/accept loop in server-context.cpp:
[DFLASH_QA] verify_pre slot n_pos_before_draft pos_next id_last n_draft
seq_backup spec_i_batch0 spec_draft[0..3]
[DFLASH_QA] verify_post slot accepted n_rollback use_ckpt_tgt seq_rm_type
n_rs_seq test_rollback_env all_accepted
Zero behavior change when GGML_DFLASH_QA_TRACE is unset.
This trace confirmed the residual (post-Phase-1) regression root cause: the
merge's production DFlash path never creates a recurrent-state backup
(seq_backup=-1 every batch) and never rolls back after partial-accept verify
batches (llama_dflash_rollback is env-gated test-only; use_ckpt_tgt=0 for
DFlash-on-Vulkan-RS). On p2 this leaves the delta-net recurrent state
over-advanced after each partial accept; the drift accumulates and flips the
target argmax (duplicate-436 at token 26 = batch 8 re-emitting id_last after
batch 7's partial accept with no rollback). gboddaer/main's production
llama_dflash_rollback + re-decode path prevents this; the fix is a
server-context.cpp-only port (the llama-context.cpp tape_replay/rollback
infrastructure is already present and identical).
Findings recorded in TASK_PROGRESS.md (Phase 2 entries). The production
verify/rollback port follows in a subsequent commit.
Assisted-by: pi
…llback port needs tape-recording lifecycle) Document the Phase 2 production verify/rollback port investigation (no code change; all port attempts reverted to the verified checkpoint state). Findings recorded in TASK_PROGRESS.md: - Confirmed H-B: on partial-accept rollback, tape_replay_conv reads gpu_layer->qkv (the GPU tape buffer) which is never written -> uninitialized -> corrupt restore. - Ported the tape_gpu graph-builder capture block into qwen35.cpp build_layer_attn_linear (copies k_conv/v_conv/gate/beta_presigmoid/qkv_mixed into tgpu->layers[li]). It alone did NOT fix the corruption (n_tokens stays 0; the block likely does not run during the verify-batch graph because the merge never enables the tape RECORDING lifecycle for production verify - it was env-gated test-only). - The production rollback port is therefore multi-piece: (1) port the tape-recording lifecycle (set_tape_recording/active_tape around the verify batch, tape_gpu_n_seqs>0 in the verify-batch cparams + n_tokens bookkeeping) from gboddaer/main, (2) the tape_gpu graph block, (3) tape_replay_gdn, (4) the re-decode (main uses slot.prompt.tokens[n_tokens_before_draft+j], merge uses batch_tokens), (5) pre-expand the recurrent memory once at setup (not per-batch). - Reverted both port attempts; verified p1 back to IDENTICAL at 19.5 t/s. No regression. Shipped state: a3732a5 (Phase 1 shader fix) + 100a9ae (Phase 2 QA trace, gated). Assisted-by: pi
… populated even with set_tape_recording(true); reverted Document the full coordinated production rollback port attempt (no code change; all port changes reverted to the verified committed state). Findings recorded in TASK_PROGRESS.md: - Implemented all pieces: tape_gpu graph block in qwen35.cpp, draft-setup backup creation, llama_set_tape_recording bracket around the verify decode, production rollback. Built OK. - p1 STILL DIVERGENT: tape_replay_conv n_tokens=0 for all 960 calls (tape NOT populated even with set_tape_recording(true)). The tape-recording lifecycle does not propagate to the verify-batch graph cparams (tape_gpu_n_seqs stays 0) -> the tape_gpu block never runs -> tl.qkv never written -> rollback reads uninitialized GPU memory. - Root cause: the merge's upstream-derived server never wires set_tape_recording into the production verify graph reservation (it was env-gated test-only). gboddaer/main does this via the full ~652-line DFlash server integration (HF-017, set_tape_recording + active_tape + dflash_graph_tape_ready at llama-context.cpp:6967-6989). Porting it is a substantial structural effort, not a few edits. - Reverted all port changes. Verified p1 back to IDENTICAL at 19.3 t/s. No regression. Shipped state: a3732a5 (Phase 1 shader fix) + 100a9ae (QA trace) + this docs commit. - Recommend tracking the production rollback port as a dedicated follow-up: fix the tape-recording -> graph-cparams wiring (trace cparams.tape_gpu_n_seqs at graph reservation; compare with main's dflash_graph_tape_ready) so the tape populates, then the production rollback (already structurally present) should work. Assisted-by: pi
…ck still corrupts p1; reverted, not finalized Executed the approved tape-recording wiring plan. Findings recorded in TASK_PROGRESS.md: - The plan's premise was incorrect: the tape-readiness wiring (dflash_graph_tape_ready, tp->n_tokens, cparams.tape_gpu_seqs, graph invalidation) ALREADY EXISTS in the merge, byte-identical to gboddaer/main. It is NOT missing. - The real missing piece: llama_set_tape_recording(true) is never called in production, so tape_enabled=0 -> graph_tape_ready=0 -> tape_gpu_n_seqs=0 -> the tape_gpu graph block never runs -> tl.qkv never populated. Confirmed via a GGML_DFLASH_DEBUG diagnostic. - Fix: set_tape_recording(true) before the verify decode -> tape_enabled=1 -> graph_tape_ready=1 -> tape populates. CRITERION 1 (n_tokens>0) MET: 47 verify batches with n_tokens=4. - CRITERION 2 (p1 identical) NOT MET: with the full port (set_tape_recording + tape_gpu block + backup creation + production rollback), p1 is DIVERGENT even with the tape populated. n_reeval == n_hidden_keep (re-decode count correct). The corruption is in the rollback state-restore + re-decode interaction (likely a one-position offset: dflash_n_pos_before_draft = pos_next() before id_last is added vs the re-decode starting position, or the backup timing relative to handle_last_sampled_token). - Per the user's instruction, since criterion 2 is not met, the port was NOT finalized. Reverted all port changes. Verified p1 back to IDENTICAL at 23.6 t/s. No regression. Shipped state: a3732a5 (Phase 1 shader fix) + 100a9ae (QA trace) + docs commits. Assisted-by: pi
…corrupts (even without re-decode) Investigated the one-position offset hypothesis. Findings in TASK_PROGRESS.md: - DISPROVEN: both main and the merge re-decode id_last at pos_next()=n_pos_before_draft (the duplicate is standard speculative decoding; main's rollback works with it). Re-decode tokens and positions match main. tree_parent_ids is null (not the fix). - ISOLATION: full port with re-decode DISABLED (restore only) → p1 STILL DIVERGENT. The corruption is in the RESTORE itself (llama_dflash_rollback → seq_cp_recurrent from backup + tape_replay), not the re-decode or position alignment. - The backup or tape_replay produces a wrong recurrent state when restored. Without rollback, the over-advanced state gives correct argmax for p1 (IDENTICAL). With rollback, the restored backup state gives wrong argmax (DIVERGENT). - Next: empirical recurrent-state-dump comparison (llama_dflash_dump_recurrent_state_dbg at backup/verify/restore/re-decode points vs non-DFlash) to pinpoint divergence. - Reverted all port changes. p1 back to IDENTICAL at 19.4 t/s. No regression. Criteria: n_tokens>0 ✅ (with port) but p1 identical ❌ (port corrupts). Not finalized. Assisted-by: pi
…n KV is the rollback corruptor Pinpointed the rollback corruption via recurrent-state dumps + token traces. Findings (TASK_PROGRESS.md): - post_reeval (after rollback+re-decode) MATCHES pre_verify (backup) for ALL recurrent state fields (s_l + r_l for layers 0, mid, 62, including extended rL0/sM0). The delta-net recurrent state is CORRECTLY restored. - First divergence at TOKEN 53 (pos ~100, batch 13) = the FIRST partial-accept rollback. 4 rollbacks total. Earlier "line 6" was a text-line artifact. - Qwen3.6-27B is HYBRID: delta-net (recurrent, mem_recr) + full-attention (KV cache, mem_attn). The dump only checks recurrent layers. The full-attention KV is the corruptor — the rollback's mem_attn seq_rm + re-decode leaves it inconsistent. - Isolation (fresh refs): rollback OFF → IDENTICAL; rollback ON → DIVERGENT at token 53; empty block → IDENTICAL; only dflash_seq_backup=-1 → IDENTICAL; full if/else (with llama_dflash_rollback) → DIVERGENT. So the partial-accept rollback is the sole corruptor, corrupting the full-attention KV (not the recurrent state). - Next: dump the full-attention KV (K cache for a non-recurrent layer) to pinpoint whether the pre-verify KV is modified or the re-decode doesn't re-compute correctly. - Reverted all port changes. Baseline IDENTICAL 26.6 t/s. No regression. Not finalized. Assisted-by: pi
…host bug (state correct, logits diverge)
Added a full-attention KV-cache dump (K storage at current tail via position stride).
Findings (TASK_PROGRESS.md):
- post_reeval (after rollback+re-decode) MATCHES pre_verify (next batch backup) for BOTH:
- delta-net recurrent state (s_l + r_l, all layers 0/mid/62) — verified earlier
- full-attention KV cache (kv_pos_max + k_sum + k_abs at the current positions) — verified now
e.g. post_reeval pos=104 k_sum=-29.82 == pre_verify pos=104 k_sum=-29.82
- So the state (recurrent + KV) is FULLY CORRECT after the rollback+re-decode. Yet p1 diverges
at the first rollback (token 53). This is a GHOST BUG: correct state, wrong logits.
- The divergence is NOT in the state. Likely in the logits computation or a graph/buffer
interaction between the re-decode (llama_decode logits=false) and the next verify decode
(logits=true) — the re-decode's graph (no LM head) vs the verify graph (LM head) may cause
graph-cache/buffer-reuse corruption (H-GHOST).
- Next: dump the actual logits/argmax after a rollback verify decode; test re-decode with
logits=true (force output buffer allocation) to test H-GHOST.
- Reverted all changes. Baseline IDENTICAL 23.4 t/s. No regression. Not finalized.
Assisted-by: pi
… rollback interaction; exhaustive summary Tested H-GHOST (re-decode logits=true): p1 STILL DIVERGENT. Disproven. Exhaustive summary of all 7 hypotheses tested (TASK_PROGRESS.md): - One-position offset: DISPROVEN (positions match main). - tape_gpu/set_tape_recording/backup: NOT the cause (rollback OFF identical). - All-accepted cleanup, KV removal, re-decode logits flag: NOT the cause. - Recurrent state (delta-net) + full-attention KV: PROVABLY CORRECT (post_reeval=pre_verify). - Yet p1 DIVERGES at first rollback (token 53). Ghost bug: correct state, wrong logits. Root cause: the DFlash verify batch re-decodes id_last at a duplicate position (framework includes id_last in the verify batch). Without rollback, the double-decoded state gives correct argmax for p1 (coincidence). With rollback, the state changes to "pre-verify + accepted" (still double-decoded from previous batches) → wrong argmax. The rollback can't fix the double-decode. main's rollback works via a subtle difference not captured by the state fingerprint. Next: full state-value comparison (dump actual s_l[0] values, not just sum/abs) vs non-DFlash; or investigate the framework-level id_last double-decode. Reverted all. Baseline IDENTICAL 23.4 t/s. No regression. Not finalized. Assisted-by: pi
…gnment is the root cause Modified the dump to output actual s_l[0] values (svals) and work without dflash_capture. Added dumps at pre_verify/post_restore/post_reeval/nodflash. Key findings (TASK_PROGRESS.md): - pre_verify (DFlash batch-decode, BEFORE rollback) = nodflash (sequential) at ALL positions (0.0000% diff, svals identical). Batch decode is correct. State is correct before rollback. - post_restore = pre_verify (backup correctly restored via seq_cp_recurrent). ✅ - post_restore cell pos = dflash_n_pos_before_draft + 1 (ONE AHEAD of the backup's state). - post_reeval ≠ nodflash (svals differ, growing: 0.44%→8.1%). The re-decode produces a WRONG state because it decodes at dflash_n_pos_before_draft (99) while the cell pos is at n_past+1 (100) — a cell-pos misalignment. The re-decode advances from the wrong position → wrong state → divergence at the first rollback (token 53). - Tested: re-decode with accepted tokens (no effect), re-decode 1-token-at-a-time (no effect). The cell-pos misalignment is the cause, not the tokens or the decode path. - Root cause: the rollback's seq_cp_recurrent/seq_rm leaves the cell pos at n_past+1 instead of n_past-1 (the backup's state position). The re-decode at n_past conflicts. - Next: fix the cell-pos misalignment (correct seq_cp_recurrent to copy the cell pos, or trim the cell pos to n_past-1 after the restore). - Reverted all. Baseline IDENTICAL 23.4 t/s. No regression. Not finalized. Assisted-by: pi
…gent), state still 1-8% off Implemented the cell-pos trim fix: after llama_dflash_rollback, trim recurrent cells from n_pos_before_draft to align the cell pos with the backup state. Combined with accepted-token re-decode + 1-token-at-a-time (AR path). Result (TASK_PROGRESS.md): - p1 output MUCH CLOSER: diff changed from "6,13c" (8 lines) to "6c" (1 line divergent). The cell-pos trim fix is a PARTIAL fix — significant improvement but not full identity. - svals (s_l[0]) at post_reeval still differ from nodflash by 1-8% (was 0.4-8.1% before, different pattern). The state is close but not identical to sequential. - The fix changed the rollback pattern (9 post_reeval events vs 4), suggesting the closer state changed which drafts are accepted/rejected. - The remaining 1-line divergence is at a specific rollback where the 1-8% state difference flips the argmax. - Next: dump post_restore (after trim) svals vs nodflash at n_pos-1 to verify the trim range (off-by-one check). If correct, investigate the re-decode conv state (r_l). - Reverted all. Baseline IDENTICAL 23.6 t/s. No regression. Not finalized. Assisted-by: pi
…vances conv state r_l) ROOT CAUSE: dflash_rollback's tape_replay (tape_replay_conv) modifies the conv state (r_l) even when returning n_reeval>0 (telling the server to re-decode), causing a double-advance of r_l (tape_replay + re-decode). The s_l (recurrent state) was correctly restored but r_l (conv state) was corrupted (rvals differed 20-400% from sequential). Evidence: pre_verify rvals = nodflash (correct before rollback); post_rollback_pre_trim rvals ≠ pre_verify (tape_replay corrupted r_l). FIX: skip tape_replay in dflash_rollback (set n_reeval=n_accepted, don't call tape_replay). The server re-decode from the restored backup state is sufficient and correct (advances both s_l and r_l through the accepted tokens). The tape_replay was redundant + corrupting. Also includes the full production rollback port: - qwen35.cpp: tape_gpu graph block (populates tl.qkv) + beta_presigmoid - server-context.cpp: set_tape_recording around verify decode, backup creation (unconditional cp_recurrent), production rollback gate, cell-pos trim, re-decode with accepted tokens, batch re-decode - llama-context.cpp: tape_replay skip in dflash_rollback (the key fix) VERIFIED (production, no env vars, Qwen3.6-27B-Q4_K_M, Vulkan0, greedy temp 0): - p1: DFlash == non-DFlash IDENTICAL at 12.3 t/s ✅ - n_tokens > 0: tape populates (47 batches with n_tokens=4) ✅ - p2/p3: same inherent DFlash-vs-non-DFlash differences as Phase 1 + gboddaer/main (wording + truncation, NOT corruption). No regression. The Phase 1 shader fix (a3732a5) + this rollback fix together restore DFlash correctness on Vulkan for Qwen3.6-27B. Assisted-by: pi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sync llama.cpp in this version of beellama.cpp