Repository navigation
fix: run a custom embedding model with the pooling its gguf declares - #20395
Conversation
|
Measurement behind this PR: the evidence #12945 asked for, a model whose declared pooling retrieves better than last-token pooling. What was compared. Two encoders, each loaded through
Neither side adds query/document prompts, and both use llama.cpp's own tokenizer, so the difference is this PR alone.
Datasets (nDCG@10, binary relevance):
Δ = after − before, paired bootstrap, 95% CI; ↑ = CI excludes 0.
Reading it:
The evaluation harness is a local study outside this PR; the numbers above are its output, not something this diff runs. |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that the extra memory is acceptable: custom decoders running mean or CLS pooling lose the 512-token micro-batch, about 1 GB more peak RSS by the spec's own Qwen measurement. What this change shipsInventory (8 items) — 8 justifiedIntent: a person using a custom GGUF embedding model gets the retrieval quality it was trained for, because it runs with the pooling its file declares. FIX: the base spec's "honouring a declared
[FIRST-PRINCIPLES-REVIEWED] 5fd91f0 Description read[DESCRIPTION-READ] 438b3ac0af41e3a7290090b6843bf4843315eaf1f48724d6459cb46dd277c69b That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check one thing before merge: users who already run a custom mean- or CLS-trained model still get no signal that re-applying it would roughly double their recall. The description's own complaint is that "nothing tells them", and that stays true for these users. [DESIGN-REVIEWED] 5fd91f0 Description read[DESCRIPTION-READ] 630a83cd21cc77119aa67df11b5883fad9032ff33c02bec8fc3d7f03dee1d590 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
5dee49b to
edd02f7
Compare
|
|
|
|
|
edd02f7 to
06c2b9b
Compare
|
@iamwhatever, could you help unblock this fork PR? I have read-only access, so I can't do these myself:
On the previous head, Design and First Principles passed and Opus had no findings; the per-finding dispositions are above. |
|
CI on
This PR touches neither the MCP tool policy nor its tests. A re-run of this head would test the same merge commit, so once #20746 and #20802 merge I'll rebase and push. That push needs one more workflow approval, and then the GPT F1 override above at the new head. |
d4a3b62 to
cacd153
Compare
|
|
Pushed This head needs a workflow approval. After the GPT 6.1 review on it posts, F1 (rebuttal here) still needs a writer's override at this head, if you agree: |
cacd153 to
c761670
Compare
|
|
Pushed
Locally on this head: the 62 test files that import the embedding modules (3,212 passed), the scoped backend run, determinism on the 37 new and changed tests, |
A local GGUF applied as the embedding model always ran with last-token pooling, the bundled Qwen3 model's. An encoder trained for mean or CLS pooling (EmbeddingGemma, mDenseOn, DenseOn, Nemotron, the BERT family) then loads and searches without any error, but its vectors rank poorly. Every current converter records the pooling in the GGUF header (<arch>.pooling_type), which the loader already reads for the context size. A custom model recorded for the first time now takes the declared pooling into its id, after the digest (custom:sha256:<digest>@mean or @cls), and the pooling a model runs with is a pure function of that id. The pooling is therefore part of what names the vector space, and nobody has to choose it. No earlier release computed or kept an id ending in such a suffix, so a pooled id never names vectors already stored, and a label an operator typed, even custom@mean, keeps last-token pooling. An id already recorded keeps its label while its file is unchanged, so an existing custom model keeps its vectors and nothing re-embeds on upgrade; reapplying it records the file's pooling and re-embeds. A file replaced in place under a custom id is re-embedded anyway, so it takes its own declared pooling. A model named only by KIROCREW_EMBED_MODEL_PATH is never recorded, so it keeps last-token pooling. Vectors inherited from before identities were stamped were made with last-token pooling, so they are inherited only while the model still runs it. The header is never read on the event loop. A model running mean or CLS pooling is loaded with one micro-batch the size of its whole batch: llama.cpp pools each micro-batch on its own and keeps only the last result, which last-token pooling survives and mean and CLS do not. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
c761670 to
5fd91f0
Compare
|
Pushed Locally on this head: all 148 whole-tree scan test files (10,134 passed), the 65 files that import the embedding modules or guard the facade (3,319 passed), determinism on the 37 new and changed tests, |
mbriones98
left a comment
There was a problem hiding this comment.
Checks green and readiness passed at 5fd91f0; no behavior change.
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: a custom embedding GGUF now loads with the pooling its header declares (mean or CLS), recorded in the model id so stored vectors never change pooling under the same id; tests cover the parse, the id and the legacy-vector rebuild. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/memory-skills-hooks.md.
Problem / Motivation
Goal: A custom embedding model runs with the pooling its GGUF declares, so a model trained for mean or CLS pooling retrieves as well in Kiro Crew as it was trained to.
A local GGUF set as the embedding model (Memory → Embedding Model, or
memory.embed_model_path) always runs with last-token pooling, the bundled Qwen3 model's. An encoder trained for mean or CLS pooling (EmbeddingGemma, mDenseOn, DenseOn, Nemotron, the BERT family) then loads and searches with no error, but its vectors rank poorly. The only trace is llama.cpp's own log line:model default pooling_type is [2], but [3] was specified.Why it matters
Anyone who picks a non-Qwen embedding model today gets worse memory and Knowledge Library recall than the bundled default, and nothing tells them. #12945 deferred this fix until a measurement showed a model whose declared pooling retrieves better than last-token pooling. On Kiro Crew's own llama.cpp runtime it does, by a wide margin:
nDCG@10. Bold changes are significant (paired bootstrap, 95% CI). Setup and per-set details are in a PR comment.
Not a goal
query:/document:): a separate change; the measurement above runs without them on both sides.What changed (motivation → approach → change)
A custom embedding model now runs with the pooling its file declares when it is first applied; a model already in use keeps its current vectors until the user re-applies it.
Symptom → cause.
LlamaCppEmbedderpassed_POOLING_TYPE_LASTfor every file and never read the<arch>.pooling_typekey that every current converter writes into the GGUF header.Approach. The pooling becomes part of a custom model's identity, so it is part of what names its vector space and can never change under vectors already stored:
custom:sha256:…@mean/custom:sha256:…@cls/custom:sha256:…custom:sha256:…)customidKIROCREW_EMBED_MODEL_PATH(never recorded)custom:sha256:…, as beforememory.embed_model_idcustom@meanThe pooling sits after the digest because that position is provenance: every id an earlier release computed ended in its 64-hex digest, and it kept a configured id as typed only when it did, re-hashing anything else. So no stored vector sits under an id ending in
@meanor@cls, a pooling other than last-token always arrives with a fresh vector space, and an id recorded before this change keeps last-token pooling whatever its label says.A missing key, an unreadable header, or a pooling Kiro Crew cannot use (none, rank, out of range) keeps last-token pooling and never aborts. Legacy vectors from before identities were stamped were made with last-token pooling, so only a model that still runs last-token records or adopts them; any other model rebuilds them without a "reapply" warning. The declared pooling is cached per file stamp beside the digest, and an event-loop caller that misses either cache reports the model unverified and leaves both to the off-loop verification worker, so the dashboard's status poll never reads the header on the loop.
A model running mean or CLS pooling is loaded with one micro-batch the size of its whole batch, as a non-causal model already is. llama.cpp pools each micro-batch on its own and keeps only the last one's result: on the vendored runtime, Qwen3 loaded under each pooling gave one 1,191-token input a cosine of 1.000000 between
n_ubatch512 and 2,048 under last-token pooling, 0.983715 under mean and 0.228116 under CLS. A last-token decoder keeps the 512.Departure from #12945's recorded design. The issue planned a
pooling-metadata-v1policy revision that re-embeds every existing custom model once, plus a legacy-alias restamp rule. Here pooling is a pure function of the id instead: existing ids keep their pooling, so nothing re-embeds on upgrade, and the user opts in by re-applying the model. The header reader the issue named is reused as planned (one more field). The bundled Qwen3 GGUF does declarepooling_type = 3(last), unlike the issue's note; it keeps its id either way.The code sits in the embedding runtime's owner modules: the pooling read in
src/kiro_crew/embeddings_gguf.pyandsrc/kiro_crew/embeddings_settings.py, the identity insrc/kiro_crew/embeddings_model.py, the load parameters insrc/kiro_crew/embeddings_llama.pyand the legacy-vector guard insrc/kiro_crew/embeddings_shared.py; the facadesrc/kiro_crew/embeddings.pylists the new names for type checkers. The memory spec (memory-skills-hooks.md) and thememory.embed_model_idrow ofconfiguration.mddescribe the new identity rule.Backwards compatibility
Compatible: every id recorded before this change keeps its label and therefore its last-token pooling, and a model set only by
KIROCREW_EMBED_MODEL_PATH, which is never recorded, keeps the id it had, so no stored vector changes meaning and nothing re-embeds on upgrade; the bundled model's id and vectors are byte-stable.Reader: src/kiro_crew/embeddings_shared.py:_align_store_embedding_space -- entry: release -- a store adopts pre-identity legacy vectors only while its model runs last-token; every model recorded before this change still does, so adoption is unchanged for them | test_memv2_audit_c_legacy.py::test_lazily_opened_store_keeps_vectors_after_restart and ::test_legacy_vectors_are_rebuilt_when_the_file_declares_another_pooling
Tests
test/test_embedding_pooling.py(new): the declared pooling is read from the header, cached per file stamp, and never on the event loop (a loop caller that misses the cached label is refused before the header is opened); an unreadable header keeps last-token; a model recorded for the first time takes its file's pooling into its id; an unchanged file keeps its recorded or operator label; a file replaced in place under acustomid takes the new file's pooling, while an operator label carries over; an unchanged file whose stamp moved keeps its pooled id; a model set only byKIROCREW_EMBED_MODEL_PATHkeepscustom:sha256:…and last-token, even when the config records another file under a loader label; onlycustom:sha256:<digest>@mean/@clsmap to llama.cpp's pooling values, and every other id keeps last-token (the bundled one, an old-shapecustom@meanlabel, an operator label with a suffix, a short digest, an unknown suffix); an id recorded under a hand-typedcustom@meanlabel keeps its id and last-token pooling, and the loader passes last-token for it; the loader passes the pooling the id names to llama.cpp, and a causal decoder running mean or CLS pooling gets one micro-batch the size of its batch while a last-token one keeps 512.test/test_memv2_audit_c_legacy.py: legacy last-token vectors are rebuilt, not adopted, when the file declares mean pooling, and no legacy ids are left behind to ask for a reapply; legacy ids hand-written beside a pooled id are not adopted either.test/test_embeddings.py: the header-metadata tuple carries the newpooling_typefield.Determinism, on the 37 new and changed node ids:
-n0: 20/20 green.-n 4 --dist loadgroup --max-worker-restart=2withtest/test_custom_embedding_model.pyandtest/test_embed_kv_clear.pyas co-tenants: 10/10 green.prove.py: PROVEN, an assertion fails with the production change reverted. Reverting onlymodel_pooling_typeto the label-based parser fails the hand-typed-label test and the table's mean, CLS and old-shape rows; reverting only the micro-batch line fails the loader test's mean and CLS cases.Manual verification
e3546c79) throughLlamaCppEmbedder, with only the pooling varied (table above; details in a PR comment).Related Issues
Closes #12945
Pattern harvest
Rule candidate: review-prompt
Pattern: a runtime parameter the model file itself declares (pooling, context length, causality) overridden by one constant for every model, so a model built differently from the default runs wrong without any error
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.
🤖 Generated with Claude Code