Skip to content

fix: run a custom embedding model with the pooling its gguf declares - #20395

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
ykharkov:fix/custom-embedding-pooling
Oct 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
ykharkov:fix/custom-embedding-pooling

Conversation

@ykharkov

@ykharkov ykharkov commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A custom embedding model runs with the pooling its GGUF declares, so a model trained for mean or CLS pooling retrieves as well in Kiro Crew as it was trained to.

A local GGUF set as the embedding model (Memory → Embedding Model, or memory.embed_model_path) always runs with last-token pooling, the bundled Qwen3 model's. An encoder trained for mean or CLS pooling (EmbeddingGemma, mDenseOn, DenseOn, Nemotron, the BERT family) then loads and searches with no error, but its vectors rank poorly. The only trace is llama.cpp's own log line: model default pooling_type is [2], but [3] was specified.

Why it matters

Anyone who picks a non-Qwen embedding model today gets worse memory and Knowledge Library recall than the bundled default, and nothing tells them. #12945 deferred this fix until a measurement showed a model whose declared pooling retrieves better than last-token pooling. On Kiro Crew's own llama.cpp runtime it does, by a wide margin:

model (GGUF Q8, no prompts) Knowledge Library real memories code 6 other languages
EmbeddingGemma-300M, last-token → declared mean 0.469 → 0.909 0.465 → 0.766 0.297 → 0.543 0.447 → 0.813
mDenseOn, last-token → declared CLS 0.843 → 0.863 0.440 → 0.773 0.304 → 0.564 0.613 → 0.808
bundled Qwen3-0.6B, unchanged 0.912 0.737 0.585 0.803

nDCG@10. Bold changes are significant (paired bootstrap, 95% CI). Setup and per-set details are in a PR comment.

Not a goal

  • Query/document prompts per model (query: / document: ): a separate change; the measurement above runs without them on both sides.
  • Any change to the bundled Qwen3 model: its id, pooling and stored vectors stay exactly as they are.
  • Re-embedding existing custom models on upgrade: they keep last-token pooling until the user re-applies them.
  • Tokenizer differences between llama.cpp and a model's own tokenizer.
  • A model catalog or new Settings UI.

What changed (motivation → approach → change)

A custom embedding model now runs with the pooling its file declares when it is first applied; a model already in use keeps its current vectors until the user re-applies it.

Symptom → cause. LlamaCppEmbedder passed _POOLING_TYPE_LAST for every file and never read the <arch>.pooling_type key that every current converter writes into the GGUF header.

Approach. The pooling becomes part of a custom model's identity, so it is part of what names its vector space and can never change under vectors already stored:

model identity recorded pooling it runs with
bundled Qwen3 unchanged last-token, as today
custom file recorded for the first time custom:sha256:…@mean / custom:sha256:…@cls / custom:sha256:… the pooling its GGUF declares
custom id already recorded before this change unchanged (custom:sha256:…) last-token, until re-applied
custom file replaced in place under a custom id its new digest, with the new file's suffix the pooling its GGUF declares (it re-embeds anyway)
model set only by KIROCREW_EMBED_MODEL_PATH (never recorded) custom:sha256:…, as before last-token
operator label in memory.embed_model_id unchanged last-token, even a hand-typed custom@mean

The pooling sits after the digest because that position is provenance: every id an earlier release computed ended in its 64-hex digest, and it kept a configured id as typed only when it did, re-hashing anything else. So no stored vector sits under an id ending in @mean or @cls, a pooling other than last-token always arrives with a fresh vector space, and an id recorded before this change keeps last-token pooling whatever its label says.

A missing key, an unreadable header, or a pooling Kiro Crew cannot use (none, rank, out of range) keeps last-token pooling and never aborts. Legacy vectors from before identities were stamped were made with last-token pooling, so only a model that still runs last-token records or adopts them; any other model rebuilds them without a "reapply" warning. The declared pooling is cached per file stamp beside the digest, and an event-loop caller that misses either cache reports the model unverified and leaves both to the off-loop verification worker, so the dashboard's status poll never reads the header on the loop.

A model running mean or CLS pooling is loaded with one micro-batch the size of its whole batch, as a non-causal model already is. llama.cpp pools each micro-batch on its own and keeps only the last one's result: on the vendored runtime, Qwen3 loaded under each pooling gave one 1,191-token input a cosine of 1.000000 between n_ubatch 512 and 2,048 under last-token pooling, 0.983715 under mean and 0.228116 under CLS. A last-token decoder keeps the 512.

Departure from #12945's recorded design. The issue planned a pooling-metadata-v1 policy revision that re-embeds every existing custom model once, plus a legacy-alias restamp rule. Here pooling is a pure function of the id instead: existing ids keep their pooling, so nothing re-embeds on upgrade, and the user opts in by re-applying the model. The header reader the issue named is reused as planned (one more field). The bundled Qwen3 GGUF does declare pooling_type = 3 (last), unlike the issue's note; it keeps its id either way.

The code sits in the embedding runtime's owner modules: the pooling read in src/kiro_crew/embeddings_gguf.py and src/kiro_crew/embeddings_settings.py, the identity in src/kiro_crew/embeddings_model.py, the load parameters in src/kiro_crew/embeddings_llama.py and the legacy-vector guard in src/kiro_crew/embeddings_shared.py; the facade src/kiro_crew/embeddings.py lists the new names for type checkers. The memory spec (memory-skills-hooks.md) and the memory.embed_model_id row of configuration.md describe the new identity rule.

Backwards compatibility

Compatible: every id recorded before this change keeps its label and therefore its last-token pooling, and a model set only by KIROCREW_EMBED_MODEL_PATH, which is never recorded, keeps the id it had, so no stored vector changes meaning and nothing re-embeds on upgrade; the bundled model's id and vectors are byte-stable.

Reader: src/kiro_crew/embeddings_shared.py:_align_store_embedding_space -- entry: release -- a store adopts pre-identity legacy vectors only while its model runs last-token; every model recorded before this change still does, so adoption is unchanged for them | test_memv2_audit_c_legacy.py::test_lazily_opened_store_keeps_vectors_after_restart and ::test_legacy_vectors_are_rebuilt_when_the_file_declares_another_pooling

Tests

  • test/test_embedding_pooling.py (new): the declared pooling is read from the header, cached per file stamp, and never on the event loop (a loop caller that misses the cached label is refused before the header is opened); an unreadable header keeps last-token; a model recorded for the first time takes its file's pooling into its id; an unchanged file keeps its recorded or operator label; a file replaced in place under a custom id takes the new file's pooling, while an operator label carries over; an unchanged file whose stamp moved keeps its pooled id; a model set only by KIROCREW_EMBED_MODEL_PATH keeps custom:sha256:… and last-token, even when the config records another file under a loader label; only custom:sha256:<digest>@mean / @cls map to llama.cpp's pooling values, and every other id keeps last-token (the bundled one, an old-shape custom@mean label, an operator label with a suffix, a short digest, an unknown suffix); an id recorded under a hand-typed custom@mean label keeps its id and last-token pooling, and the loader passes last-token for it; the loader passes the pooling the id names to llama.cpp, and a causal decoder running mean or CLS pooling gets one micro-batch the size of its batch while a last-token one keeps 512.
  • test/test_memv2_audit_c_legacy.py: legacy last-token vectors are rebuilt, not adopted, when the file declares mean pooling, and no legacy ids are left behind to ask for a reapply; legacy ids hand-written beside a pooled id are not adopted either.
  • test/test_embeddings.py: the header-metadata tuple carries the new pooling_type field.

Determinism, on the 37 new and changed node ids:

  • 20 runs at -n0: 20/20 green.
  • 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_custom_embedding_model.py and test/test_embed_kv_clear.py as co-tenants: 10/10 green.
  • Shuffled order (ids passed shuffled, per testing-conventions), seeds 31435, 7211, 90817, 4402, 61263: all green.
  • prove.py: PROVEN, an assertion fails with the production change reverted. Reverting only model_pooling_type to the label-based parser fails the hand-typed-label test and the table's mean, CLS and old-shape rows; reverting only the micro-batch line fails the loader test's mean and CLS cases.

Manual verification

  • Retrieval effect measured on the vendored llama.cpp (llama-cpp-python 0.3.34, llama.cpp e3546c79) through LlamaCppEmbedder, with only the pooling varied (table above; details in a PR comment).
  • Not exercised by applying a model in the desktop app's Memory → Embedding Model card; the unit tests cover the identity and loader paths that card calls.

Related Issues

Closes #12945

Pattern harvest

Rule candidate: review-prompt
Pattern: a runtime parameter the model file itself declares (pooling, context length, causality) overridden by one constant for every model, so a model built differently from the default runs wrong without any error

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

🤖 Generated with Claude Code

@ykharkov
ykharkov requested review from a team and KiroCrewCaptain as code owners October 14, 2026 17:32
@ykharkov
ykharkov requested a review from mbriones98 October 14, 2026 17:32
@ykharkov

Copy link
Copy Markdown
Contributor Author

Measurement behind this PR: the evidence #12945 asked for, a model whose declared pooling retrieves better than last-token pooling.

What was compared. Two encoders, each loaded through LlamaCppEmbedder on the vendored runtime (llama-cpp-python 0.3.34, llama.cpp e3546c79), with production's context policy and 4 threads. Each was run twice with only the pooling changed:

  • before = last-token pooling, exactly what main passes for every file today;
  • after = the pooling the GGUF declares, which this PR passes.

Neither side adds query/document prompts, and both use llama.cpp's own tokenizer, so the difference is this PR alone.

model GGUF
EmbeddingGemma-300M (declares mean) ggml-org's embeddinggemma-300M-Q8_0.gguf (keeps its two dense projections)
mDenseOn (declares CLS) Q8_0, converted with llama.cpp's converter at the vendored commit

Datasets (nDCG@10, binary relevance):

  • KB: kb_golden_v2 (this repo), 40 answerable queries, through the production KB fusion (FTS5 + graph + vector RRF, w_v=2).
  • Real memories: 83 agent memories from one real store (private, not published) with 83 queries worded so they do not reuse the memory's words, through the production memory fusion (0.6·cosine + 0.4·keyword).
  • Code: a 100-query subset of Agent Retrieval Bench (code→test, review comment→code, stack trace→code, edit→ripple) over a shared 200-file corpus, dense leg only.
  • 6 other languages: the KB and real-memory sets machine-translated into es, zh, ja, it, de and vi (docs and queries both), production fusion, mean over the six.

Δ = after − before, paired bootstrap, 95% CI; ↑ = CI excludes 0.

model dataset before (last-token) after (declared) Δ (95% CI)
EmbeddingGemma-300M KB 0.469 0.909 +0.441 [+0.317, +0.568] ↑
Real memories 0.465 0.766 +0.301 [+0.225, +0.376] ↑
Code 0.297 0.543 +0.245 [+0.178, +0.312] ↑
6 other languages 0.447 0.813 +0.335 [+0.308, +0.361] ↑
mDenseOn KB 0.843 0.863 +0.019 [−0.047, +0.085]
Real memories 0.440 0.773 +0.334 [+0.250, +0.415] ↑
Code 0.304 0.564 +0.260 [+0.194, +0.324] ↑
6 other languages 0.613 0.808 +0.219 [+0.192, +0.245] ↑
bundled Qwen3-0.6B (unchanged by this PR) KB / memories / code / 6 languages 0.912 / 0.737 / 0.585 / 0.803 same —

Reading it:

  • Last-token pooling is far worse than the declared pooling on retrieval, by 0.22–0.44 nDCG@10 on 7 of 8 cells.
  • On EmbeddingGemma's KB set, last-token vectors fall below the keyword legs alone (0.469 against 0.853 for keyword-only), because the broken vector leg drags the fused ranking down.
  • With the declared pooling, both encoders match or beat the bundled model on real memories.
  • The fix(embeddings): size the native context from GGUF metadata so encoders never abort #12860 measurement saw no difference because it compared nearest-neighbour order on six paraphrase pairs. Last-token vectors keep that coarse order, but they rank whole corpora much worse.

The evaluation harness is a local study outside this PR; the numbers above are its output, not something this diff runs.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 5fd91f09cd15787d0bba74f9a8ee2729e41942ab touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 5fd91f09cd15787d0bba74f9a8ee2729e41942ab via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that the extra memory is acceptable: custom decoders running mean or CLS pooling lose the 512-token micro-batch, about 1 GB more peak RSS by the spec's own Qwen measurement.

What this change ships

Inventory (8 items) — 8 justified

Intent: a person using a custom GGUF embedding model gets the retrieval quality it was trained for, because it runs with the pooling its file declares. FIX: the base spec's "honouring a declared pooling_type is tracked separately" and #12945, now backed by the nDCG table.

  1. A custom model applied for the first time runs with the mean or CLS pooling its GGUF declares — justified
  2. Its recorded id now ends in @mean or @cls after the digest, which gives it a new vector space and so re-embeds — justified
  3. Custom models recorded earlier keep last-token pooling and their vectors until re-applied — justified
  4. A model set only by KIROCREW_EMBED_MODEL_PATH, or under an operator label, keeps last-token pooling — justified
  5. Models running mean or CLS pooling are loaded with one micro-batch the size of the whole batch, so they use more memory — justified
  6. Pre-identity legacy vectors are adopted only while the model runs last-token; otherwise they are rebuilt with no "reapply" warning — justified
  7. The pooling read is cached per file stamp and refused on the event loop, reusing the digest's refusal through the extracted _refuse_on_event_loop — justified
  8. New names are listed in the embeddings.py type-checker mirror, as its "every owner-defined name in _EXPORTS" comment requires — justified

[FIRST-PRINCIPLES-REVIEWED] 5fd91f0

Description read

[DESCRIPTION-READ] 438b3ac0af41e3a7290090b6843bf4843315eaf1f48724d6459cb46dd277c69b

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 5fd91f09cd15787d0bba74f9a8ee2729e41942ab via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check one thing before merge: users who already run a custom mean- or CLS-trained model still get no signal that re-applying it would roughly double their recall. The description's own complaint is that "nothing tells them", and that stays true for these users.

[DESIGN-REVIEWED] 5fd91f0

Description read

[DESCRIPTION-READ] 630a83cd21cc77119aa67df11b5883fad9032ff33c02bec8fc3d7f03dee1d590

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 5fd91f09cd15787d0bba74f9a8ee2729e41942ab via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5fd91f0

@github-actions

github-actions Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 5fd91f09cd15787d0bba74f9a8ee2729e41942ab via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5fd91f0

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 14, 2026
@ykharkov
ykharkov force-pushed the fix/custom-embedding-pooling branch from 5dee49b to edd02f7 Compare October 14, 2026 20:11
@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=7d27de40008c The pooling sticks to the path, not the bytes

fixed in edd02f7: in _custom_model_id, when the recomputed digest differs from the recorded one and the label is one the loader writes (custom, custom@mean, custom@cls: _LOADER_LABELS), the label is rebuilt from _new_model_label(path, stamp). An operator label, and a loader label with no recorded digest, keep theirs.
Test: test_a_file_replaced_in_place_takes_its_own_pooling_under_a_loader_label[cls-label|plain-custom|operator-label] overwrites a CLS file in place with a mean header (same inode and size, mtime moved with os.utime) and checks the id _verify_custom_model returns and persists.
Spec: "read once per file stamp" is replaced by when the header is read (a model recorded with no id, or a loader-labelled file whose digest changed; never an operator label) and that the result is cached per stamp. configuration.md says a file replaced in place under a custom id takes the new file's pooling.

@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=ea860b9fa688 EmbeddingProfile wraps a single int

fixed in edd02f7: EmbeddingProfile and LlamaCppEmbedder.profile are deleted.
model_pooling_type(model_id) -> int replaces embedding_profile with the same parsing and returns llama.cpp's pooling value.
The embedder stores self._pooling_type and passes it to the loader; _verify_custom_model and _align_store_embedding_space compare model_pooling_type(...) with _POOLING_TYPE_LAST.

@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=704643a05811 a model set only through KIROCREW_EMBED_MODEL_PATH never gets its id written

fixed in edd02f7: resolve_custom_model treats a file _verify_custom_model can never record (memory.embed_model_path does not name it) as labelled custom when no id is recorded or the recorded label is a loader label written for another file.
So such a model keeps custom:sha256:<digest>, the id base computed, and last-token pooling on every start; nothing about persistence changes, since update() could not write for that path before either.
Test: test_a_model_set_only_by_the_env_var_keeps_last_token_pooling[env-only|env-over-another-recorded-file|also-configured]; the last case shows the same file named by memory.embed_model_path is recorded with its declared pooling.
Docs: the KIROCREW_EMBED_MODEL_PATH row of configuration.md and the spec say such a model keeps last-token pooling until memory.embed_model_path names the same file.

@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=364e8a65ecc7 Independently cached labels allow GGUF parsing on the event loop

fixed in edd02f7: the running-loop check moved into _refuse_on_event_loop(), called first in both _model_content_digest and _new_model_label.
A loop caller that hits the digest cache but misses the label cache now raises _ModelIdentityUnverified before the header is opened; resolve_custom_model reports custom:unverified and starts the off-loop verification worker, which fills both caches (lru_cache does not cache the exception).
Test: test_a_loop_caller_that_misses_the_label_cache_never_reads_the_header caches the digest through an operator-labelled id, then calls _custom_model_id(model, "") inside asyncio.run: it raises and gguf_declared_pooling is never called.

@ykharkov

ykharkov commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=364e8a65ecc7 Legacy operator labels can silently change pooling without changing vector identity

rebutted, not a defect: the precondition is a store embedded before this change under an id labelled exactly custom@mean or custom@cls.
No base code, doc or UI wrote or suggested that shape: base _custom_model_id records "custom" when no label is set (embeddings.py:1527 at 79edb16), the dashboard apply passes an empty label, and git grep "custom@\|@mean\|@cls" 79edb1616d -- src docs website/src finds nothing.
So it needs an operator to have hand-typed into config.json the one label shape this change defines; every other label (e5, e5@mean, custom@max) keeps last-token (test_every_other_identity_keeps_last_token_pooling).
Nothing on disk is lost or rewritten even then: that store ranks worse until the model is reapplied (Memory → Embedding Model), which re-embeds it under one pooling.
I have read-only access, so no override is recorded here. A maintainer who verifies this could post at the new head: /ai-review override gpt edd02f747f09ea8e4e993deef1e732246b6a6b58: agent: F1 needs memory.embed_model_id hand-typed as exactly custom@mean or custom@cls before this change gave that shape meaning; no data is lost and reapplying re-embeds.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 14, 2026
@ykharkov
ykharkov force-pushed the fix/custom-embedding-pooling branch from edd02f7 to 06c2b9b Compare October 15, 2026 13:04
@ykharkov

ykharkov commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

@iamwhatever, could you help unblock this fork PR? I have read-only access, so I can't do these myself:

  1. Approve and run workflows for 06c2b9b (every fork push needs it). This head is rebased on main and includes test(acp): settle lease releases before the skill-view relaunch census #20485, which fixes the shard-3 test_skill_view_relaunch_harness flake the previous head hit; the other red there, Frontend Tests (4), was an npm ci ECONNRESET before any test ran.
  2. GPT 6.1 F1, which is fenced, so only a writer can clear it. My rebuttal and its evidence are in this comment, and the lane's own adjudicator flagged it as an edge case. If you agree after checking, the override, posted once the GPT review on 06c2b9b is up, would be:
    /ai-review override gpt 06c2b9b943fb67a8ab3e36b85a5c1d2f4c73e7ff: agent: F1 needs memory.embed_model_id hand-typed as exactly custom@mean or custom@cls before this change gave that shape meaning; no data is lost and reapplying re-embeds.
    
  3. Labels bug and area: core, as on embeddings: honour a GGUF's declared pooling_type (mean/cls) for custom embed models #12945, and a review when you have time.

On the previous head, Design and First Principles passed and Opus had no findings; the per-finding dispositions are above.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 15, 2026
@ykharkov

Copy link
Copy Markdown
Contributor Author

CI on 06c2b9b: the one test failure is Backend Tests (Windows) (7), the known Windows red on main, not this change:

This PR touches neither the MCP tool policy nor its tests. A re-run of this head would test the same merge commit, so once #20746 and #20802 merge I'll rebase and push. That push needs one more workflow approval, and then the GPT F1 override above at the new head.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 18, 2026
@ykharkov
ykharkov force-pushed the fix/custom-embedding-pooling branch from d4a3b62 to cacd153 Compare October 18, 2026 23:00
@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=9dce30417a6a Mean or CLS pooling on a causal custom model keeps the 512-token micro-batch

fixed in cacd153: LlamaCppEmbedder._load_model passes n_ubatch=policy.n_batch whenever the model's pooling is not last-token, as the non-causal path already does; a last-token decoder keeps 512.
Confirmed first on the vendored runtime: the bundled Qwen3 Q8_0 loaded under each pooling, one 1,191-token input, n_ubatch 512 vs 2,048 gave cosine 1.000000 (last), 0.983715 (mean), 0.228116 (CLS).
Test: test_a_decoder_runs_mean_and_cls_pooling_in_one_micro_batch[mean|cls|last] on a causal qwen3 header (asserted read as causal first); with the production line reverted, [mean] and [cls] fail.
Spec: the "Metadata-sized llama.cpp context" bullet states the rule, the measurement, and that a custom decoder pays the larger micro-batch only under mean or CLS pooling.

@ykharkov

Copy link
Copy Markdown
Contributor Author

Pushed cacd153: it fixes the Opus 5.5 finding above (mean or CLS pooling now loads with one micro-batch the size of the batch; disposition above), rebased on current main. The related embedding and memory tests, the scoped backend run, the determinism runs and prove.py pass locally.

This head needs a workflow approval. After the GPT 6.1 review on it posts, F1 (rebuttal here) still needs a writer's override at this head, if you agree:

/ai-review override gpt cacd1537d21d88900fc2efe3c10ca10c8667dcad: agent: F1 needs memory.embed_model_id hand-typed as exactly custom@mean or custom@cls before this change gave that shape meaning; no data is lost and reapplying re-embeds.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 18, 2026
@ykharkov
ykharkov force-pushed the fix/custom-embedding-pooling branch from cacd153 to c761670 Compare October 19, 2026 19:05
@ykharkov

Copy link
Copy Markdown
Contributor Author
  • span=364e8a65ecc7 Existing operator labels silently change pooling on upgrade

fixed in c761670: the pooling now sits after the digest, custom:sha256:<digest>@mean / @cls (_custom_model_id, _new_model_pooling), and model_pooling_type honours only that shape under the custom label with a valid 64-hex digest.
That position is the persisted provenance the finding asks for: every id an earlier release computed ended in its 64-hex digest, and base _custom_model_id kept a configured id verbatim only when the text after its last :sha256: was such a digest, re-hashing anything else. So no stored vector can sit under an id ending in @mean/@cls; a pooling other than last-token always arrives with a new vector-space identity.
A pre-upgrade hand-typed custom@mean:sha256:<digest> is now an operator label: it keeps its id and last-token pooling, and the loader passes pooling 3 (test_an_id_recorded_under_a_hand_typed_pooling_label_keeps_last_token_pooling, which fails with the old label parser).
test_only_a_custom_id_suffixed_after_its_digest_names_a_pooling pins the table: old-shape label, operator label with a suffix, short digest, @last, unknown suffix and the bundled id all stay last-token.

@ykharkov

Copy link
Copy Markdown
Contributor Author

Pushed c761670, rebased on current main (it resolves the conflict with the embeddings.py owner split, #21946):

  • GPT 6.1 F1 is fixed in code (disposition above): a pooled id now carries its pooling after the digest, custom:sha256:<digest>@mean, a shape no earlier release could have stored vectors under, so a pre-upgrade hand-typed custom@mean label keeps last-token pooling. No override is needed for it any more.
  • Backend Tests (3.12, 2) on the previous head failed only on two test_facade_mirror_guards.py whole-tree scans hitting the 120 s per-test timeout. Both take about 57 s locally on this branch and on main alike.

Locally on this head: the 62 test files that import the embedding modules (3,212 passed), the scoped backend run, determinism on the 37 new and changed tests, prove.py and the Fast Gate checks all pass. It needs a workflow approval to run CI.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 19, 2026
A local GGUF applied as the embedding model always ran with last-token
pooling, the bundled Qwen3 model's. An encoder trained for mean or CLS
pooling (EmbeddingGemma, mDenseOn, DenseOn, Nemotron, the BERT family)
then loads and searches without any error, but its vectors rank poorly.

Every current converter records the pooling in the GGUF header
(<arch>.pooling_type), which the loader already reads for the context
size. A custom model recorded for the first time now takes the declared
pooling into its id, after the digest (custom:sha256:<digest>@mean or
@cls), and the pooling a model runs with is a pure function of that id.
The pooling is therefore part of what names the vector space, and nobody
has to choose it. No earlier release computed or kept an id ending in
such a suffix, so a pooled id never names vectors already stored, and a
label an operator typed, even custom@mean, keeps last-token pooling.

An id already recorded keeps its label while its file is unchanged, so
an existing custom model keeps its vectors and nothing re-embeds on
upgrade; reapplying it records the file's pooling and re-embeds. A file
replaced in place under a custom id is re-embedded anyway,
so it takes its own declared pooling. A model named only by
KIROCREW_EMBED_MODEL_PATH is never recorded, so it keeps last-token
pooling. Vectors inherited from before identities were stamped were
made with last-token pooling, so they are inherited only while the
model still runs it. The header is never read on the event loop.

A model running mean or CLS pooling is loaded with one micro-batch the
size of its whole batch: llama.cpp pools each micro-batch on its own
and keeps only the last result, which last-token pooling survives and
mean and CLS do not.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@ykharkov
ykharkov force-pushed the fix/custom-embedding-pooling branch from c761670 to 5fd91f0 Compare October 20, 2026 02:03
@ykharkov

Copy link
Copy Markdown
Contributor Author

Pushed 5fd91f0: on c761670 every AI lane passed (GPT 6.1 now reports no blocking findings), and the only red was Backend Tests (3.12, 2) on test_facade_mirror_guards.py's manifest check: the facade's TYPE_CHECKING block in src/kiro_crew/embeddings.py did not list the 10 names this change adds to the owner modules. They are listed now, and the commit is rebased on current main (the memory spec conflict with #22083's bounded-decode bullet is resolved by keeping both bullets).

Locally on this head: all 148 whole-tree scan test files (10,134 passed), the 65 files that import the embedding modules or guard the facade (3,319 passed), determinism on the 37 new and changed tests, prove.py and the Fast Gate checks pass. It needs a workflow approval to run CI.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 20, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at 5fd91f0; no behavior change.

@KiroCrewCaptain
KiroCrewCaptain merged commit cfbb019 into kirodotdev:main Oct 20, 2026
100 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 20, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: a custom embedding GGUF now loads with the pooling its header declares (mean or CLS), recorded in the model id so stored vectors never change pooling under the same id; tests cover the parse, the id and the legacy-vector rebuild. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/memory-skills-hooks.md.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

embeddings: honour a GGUF's declared pooling_type (mean/cls) for custom embed models

3 participants