Repository navigation

feat: KV cache compaction via attention matching - #4

Open

fabiantax wants to merge 107 commits into

masterfrom

feat/kv-compaction

Open

feat: KV cache compaction via attention matching#4
fabiantax wants to merge 107 commits into
masterfrom
feat/kv-compaction

Commits on Mar 4, 2026
- Add concat op to webgpu. (#20068)
  yomaytk
  authored
  541bf37
- [WebGPU] Fix wait logic for inflight jobs (#20096)
  nikhilJain17
  authored
  24d2ee0
- hexagon: add llama-completion runner script (#20095)
  tboinovski1
  authored
  1a29907
Commits on Mar 5, 2026
- opencl: add `SET`, support i32 for `CPY`, minor refactor for cpy (#20101)
  lhez
  authored
  69fd345
- hexagon: Flash Attention optimizations (dma, mpyacc, multi-row) and MatMul updates (#20118)
  
  max-krasnyansky
  and
  chraac
  authored
  7a99dc8
- chore : correct typos [no ci] (#20041)
  
  authored
  92f7da0
- webui: Improvements for Models Selector UI (#20066)
  allozaur
  authored
  5e335ba
- convert : register Qwen 3.5 ForCausalLM for text only (#20119)
  CISC
  authored
  cf23251
- cli : add command and file auto-completion (#19985)
  CISC
  authored
  b5ed0e0
- model : update Qwen3.5 model type detection (#20126)
  
  EZForever
  and
  CISC
  authored
  872646b
- CUDA: Improve performance via less synchronizations between token (#17795)
  
  aendk
  and
  ggerganov
  authored
  2cd20b7
- models : kda chunk size = 16 (#19827)
  
  ymcki
  and
  ggerganov
  authored
  a0ed91a
Commits on Mar 6, 2026
- hexagon: add fp16 support for binary ops: add,sub,mul,div (#20139)
  
  YardenTal44
  and
  max-krasnyansky
  authored
  2b10b62
- opencl: add neg, exp and diag (#20127)
  lhez
  authored
  6c97bff
- cli : Don't clear system prompt when using '/clear' (#20067)
  roj234
  authored
  f7db3f3
- kv-cache : fix M-RoPE checkpoints (#20132)
  ggerganov
  authored
  17a4258
- ggml-cpu: fix data race for debug asserts (#20148)
  JohannesGaessler
  authored
  2850bc6
- webui: Agentic Loop + MCP Client with support for Tools, Resources and Prompts (#18655)
  allozaur
  authored
  f6235a4
- Checkpoint every n tokens: squash (#20087)
  pwilkin
  authored
  f5ddcd1
- context: ignore zero scale LoRAs when checking sameness (#20166)
  TimNN
  authored
  388baab
- CUDA: use shared mem for ssm_conv (#20128)
  
  am17an
  and
  JohannesGaessler
  authored
  1e38a7a
- ggml-cpu: Fix gcc 15 ICE on ppc64le (#20083) (#20130)
  shalinib-ibm
  authored
  c6980ff
- ggml: update comments for backends which have no memory to report (#20157)
  taronaeo
  authored
  ba2ff79
- ggml-cuda: add mem check for fusion (#19916)
  
  am17an
  and
  JohannesGaessler
  authored
  d48e876
- cpu: skip redudant ROPE cache updates (#20149)
  max-krasnyansky
  authored
  ba2fd11
- server : preserve anthropic thinking blocks in conversion (#20120)
  
  T0mSIlver
  and
  root
  authored
  e68f2fb
- hexagon: add f32 ssm_conv op (#20122)
  
  tboinovski1
  and
  max-krasnyansky
  authored
  34df42f
- Autoparser - complete refactoring of parser architecture (#18675)
  pwilkin
  authored
  566059a
- Add @pwilkin to CODEOWNERS for autoparser code (#20174)
  pwilkin
  authored
  7463687
- quants : Add memsets and other fixes for IQ quants (#19861)
  bartowski1182
  authored
  649f064
- Autoparser: add optional argument reshuffle capability (#20171)
  pwilkin
  authored
  2f2923f
Commits on Mar 7, 2026
- Autoparser: True streaming (#20177)
  pwilkin
  authored
  c024d85
- opencl: add l2_norm (#20160)
  lhez
  authored
  6fce5c6
- ggml: add GATED_DELTA_NET op (#19504)
  
  am17an
  and
  ggerganov
  authored
  c5a7788
Commits on Mar 8, 2026
- [SYCL] supprt Flash Attention for fp32/fp16/Q4/Q5/Q8 (#20190)
  arthw
  authored
  213c4a0
- server : correct index on finish in OAI completion streams (#20226)
  decahedron1
  authored
  ff52ee9
- Revert to OAI-compatible args (#20213)
  pwilkin
  authored
  b283f6d
- readme : update infra list (#20212)
  Defilan
  authored
  a950479
- llama: end-to-end tests (#19802)
  JohannesGaessler
  authored
  a976ff0
- vulkan: Fix data races in coopmat1 mul_mat(_id) (#20084)
  jeffbolznv
  authored
  cd18a50
- ggml-vulkan: Add ELU op support (#20183)
  GiantPrince
  authored
  d088d5b
- Fix structured outputs (#20223)
  
  pwilkin
  and
  aldehir
  authored
  62b8143
- Fix compile bug (#20203)
  
  pwilkin
  and
  CISC
  authored
  9b24886
- common : gracefully handle incomplete output (#20191)
  aldehir
  authored
  451ef08
- graph : remove redundant scale_w parameter (#20235)
  CISC
  authored
  35bee03
- server : do not create checkpoints right after mtmd chunks (#20232)
  ggerganov
  authored
  d417bc4
Commits on Mar 9, 2026
- PEG parser for LFM2 (#20251)
  pwilkin
  authored
  97c64fb
- llama-bench: introduce `-hf` and `-hff` flags & use `--mmap 1` by default (#20211)
  taronaeo
  authored
  ae87863
- cuda : display total and free VRAM capacity during device initialization (#20185)
  tehsiuhuang
  authored
  5f4cdac
- vulkan: skip zero size tensors in backend copies (#20233)
  0cc4m
  authored
  b2f460b
- ggml-vulkan: add SGN operator, auto-generate Vulkan.csv and ops.md (#20219)
  bertaye
  authored
  0beb8db
- contributing: limit open PRs for new contributors to 1 (#20036)
  am17an
  authored
  e2763a6
- llama-quant : left-align tensor names in output (#20117)
  ddh0
  authored
  b518195
- ggml-cuda: disable gdn for musa (#20278)
  am17an
  authored
  e8bbc73
- server : add kill switch when server is stuck (#20277)
  ggerganov
  authored
  107d599
- models : fix assert in mamba2 graph (#20270)
  ggerganov
  authored
  43e1cbd
- common: map developer role to system (#20215)
  pwilkin
  authored
  f76565d
- server : fix off-by-1 in server_tokens::size_up_to_pos() (#20279)
  ggerganov
  authored
  d6e1556
- server : warn swa-full is not supported for non-SWA models (#20291)
  ggerganov
  authored
  344ee2a
- metal : add upscale (#20284)
  ggerganov
  authored
  ed0007a
- server : fix checkpoints n_tokens calculation (#20287)
  ggerganov
  authored
  96cfc49
- metal : extend mul_mv_ext to BF16, Q2_K, Q3_K (#20250)
  
  arkavo-com
  and
  claude
  authored
  e22cd0a
- Add strix-halo-optimizer skill for AMD Ryzen AI Max+ 395
  fabiantax
  committed
  60a4387
- Add eval workspace and metadata for strix-halo-optimizer iteration 1
  fabiantax
  committed
  30dd1bc
- Add eval run outputs and timing for iteration 1
  fabiantax
  committed
  8ad6b83
- Add grading results and benchmark summary for iteration 1
  fabiantax
  committed
  959bc8f
- Rewrite AGENTS.md for AI-optimized fork development
  fabiantax
  committed
  384266b
- Move strix-halo-optimizer skill to .claude/skills/ and expand content
  fabiantax
  committed
  28020a0
- Add iteration 2 eval results for strix-halo-optimizer skill
  fabiantax
  committed
  938e033
- Update skill with research findings: gfx1151, Vulkan parity, MoE models
  fabiantax
  committed
  ea0497a
- Add Vulkan cooperative matrix and FA tuning details
  fabiantax
  committed
  73ff9e9
- Add iteration-3 eval metadata for strix-halo-optimizer skill
  fabiantax
  committed
  96ebc81
- Add timing data for eval-3 with_skill run
  fabiantax
  committed
  80f1655
- Add timing data for eval-1 with_skill run
  fabiantax
  committed
  10e3f20
- Add timing data for eval-2 with_skill run
  fabiantax
  committed
  ca89fec
- Add eval-3 without_skill outputs and timing
  fabiantax
  committed
  9b32d31
- Add eval-2 without_skill outputs and timing
  fabiantax
  committed
  ef72276
- Add eval-1 without_skill outputs and timing
  fabiantax
  committed
  8f85db6
- Add eval-4 with_skill outputs and timing
  fabiantax
  committed
  c25396f
- Add eval-5 with_skill outputs and timing
  fabiantax
  committed
  adc2b8c
- Add eval-5 without_skill outputs and timing
  fabiantax
  committed
  71db90a
- Add eval-4 without_skill outputs and timing
  fabiantax
  committed
  6ef221d
- Add eval-6 with_skill outputs and timing
  fabiantax
  committed
  c31cfa6
- Add iteration-3 eval results, grading, and benchmark viewer
  fabiantax
  committed
  4c9b6dd
- ggml-cuda: auto-detect AMD APU unified memory and improve iGPU support
  fabiantax
  committed
  4e5622c
- ggml-cuda: tune flash attention kernels for RDNA 3.5
  fabiantax
  committed
  9230d66
- ggml-cuda: add UMA weight prefetching for integrated GPUs
  fabiantax
  committed
  2034f53
- ggml-backend: selective backend sync and eliminate redundant sync on copy fallback
  fabiantax
  committed
  f229ccc
- ggml-cuda: add auto-tuning for flash attention parallel_blocks and GEMV threshold
  fabiantax
  committed
  4bdaa8f
- docs: add implementation plan for auto-tuning flash attention
  fabiantax
  committed
  66c2d8d
- docs: add flash attention auto-tuning documentation
  fabiantax
  committed
  e7fa9d0
- common: add UMA auto-configuration for iGPU unified memory systems
  fabiantax
  committed
  305fec9
- common: add APEX-inspired bandwidth-aware layer splitting for UMA
  fabiantax
  committed
  c9c1948
- docs: add user stories for APEX runtime scheduling
  fabiantax
  committed
  51cbe35
- docs: add APEX runtime scheduling implementation plan
  fabiantax
  committed
  428d70e
- implement APEX runtime scheduling for hybrid CPU-GPU inference
  fabiantax
  committed
  e0bb11e
- vulkan: add RDNA3 pipeline config, iGPU flash attention tuning, and UMA zero-copy buffers
  fabiantax
  committed
  7d21e43
- fix: AVX512 build errors with GGML_ZEN5 non-native builds
  fabiantax
  committed
  7435012
- Add Strix Halo benchmark script for validating build performance
  fabiantax
  committed
  298be3e
- docs: add MoE token generation optimization stories, benchmark script, and ADR
  fabiantax
  committed
  70bed2d
- Add MoE expert selection analyzer tool for prefetch optimization
  fabiantax
  committed
  9ce49f3
- Add software prefetch for MoE expert weights in CPU MUL_MAT_ID
  fabiantax
  committed
  a2f7157
- Make UMA prefetch expert-aware for MoE on Strix Halo
  fabiantax
  committed
  3f39d21
- fix: moe-analyzer arg parsing, expert auto-detection, and token generation
  
  fabiantax
  and
  claude
  committed
  5677806
- vulkan: fused SSM recurrence + batched elementwise + shared memory tiling
  
  fabiantax
  and
  claude
  committed
  e69b782
- chore: move unfinished work to .unfinished/, update .gitignore
  
  fabiantax
  and
  claude
  committed
  686f364
- feat: KV cache compaction via attention matching (arXiv:2602.16284)
  
  fabiantax
  and
  claude
  committed
  190e8d9

Commits

Commits on Mar 4, 2026

Commits on Mar 5, 2026

Commits on Mar 6, 2026

Commits on Mar 7, 2026

Commits on Mar 8, 2026

Commits on Mar 9, 2026