Repository navigation
Injection quality: embedding-model upgrade (gated on #95 — full re-index + A/B) #96
Description
Split out of #54 (Injection quality Tier 3), item 1.
State, verified against main at e706eb3 (v0.16.3)
all-MiniLM-L6-v2 is still the embedding model default, untouched since #54 was filed:
src/config.rs:243— the defaultsrc/config_tests.rs:9— asserts itsrc/access.rs:528,src/cli/benchmark.rs:64— repeated
The work
New model integration, a full re-index across every indexed repo, and an A/B eval against the
current model.
⚠️ Gated on #95 (fresh calibrated eval) — do not start before it
This is the most expensive of the three Tier 3 items and the hardest to reverse: it invalidates every
existing index. The committed eval numbers predate all 14 shipped quality features, so there is
currently no evidence that the embedding model is the bottleneck. #95 produces that evidence.
Start this only if #95's numbers identify the model as the limiting factor.
Activity
scbrown commentedon Oct 12, 2026
Current-source check at
main94bfc9353d9d5f7a256e673c489aa691c3155b6a:EmbeddingConfig::default()still selectsall-MiniLM-L6-v2, and the default-config test asserts it. No default-model upgrade is established by this check.The migration work should account for plumbing that already exists:
ModelConfig::getregisters MiniLM (384 dimensions, 256-token window), BGE-small (384, 512), and GTE-small (384, 512).Embedder::from_configalso loads custom ONNX model/tokenizer paths, and dimension resolution requires explicit dimensions for custom models.That support makes isolated candidate evaluation possible; it does not supply evidence to switch the default. Keep the #95 gate: use a frozen current-source retrieval/ablation baseline, then compare candidate models on the same held-out tasks and queries with separate fully rebuilt indexes. Equal vector dimensions do not make embeddings from different models interchangeable.
Before any shared-index cutover, measure retrieval quality alongside indexing/query cost, preserve the original index/model configuration for rollback, and identify exactly which indexes must be rebuilt. This triage introduces no new benchmark numbers, model switch, or production reindex.
scbrown commentedon Oct 21, 2026
2026-10-21 disposition: retain the #95 evidence gate
Current main
b036433f55b75ddb8520d5bf4ad0b681464187ebstill selectsall-MiniLM-L6-v2by default. The new information since the prior model triage is the capped #95 report: only 10/40 tasks, with invalid/error and dedup-contract outcomes retained. It explicitly does not identify the model as the limiting factor.Owner disposition: keep open, gated. Resume when #95 supplies a usable held-out baseline and paired evidence supporting model selection; compare candidates with separately rebuilt private indexes and record quality, indexing/query cost and a model/index rollback before any shared migration. Existing alternate/custom ONNX plumbing is not evidence to switch defaults; equal vector dimensions do not make embeddings interchangeable.
This dated disposition is under the repository issue-triage task; the existing tracking record remains
aegis-gqn. No model change, reindex, benchmark campaign or shared cutover was performed or authorized by this comment.scbrown commentedon Oct 23, 2026
2026-10-23 triage — owner: Gennaro. Keep open, explicitly blocked on usable paired evidence from #95.
I checked frozen main
fddcdb7b1bdf885528dbf0b432958125afe9c29d: EmbeddingConfig::default still selectsall-MiniLM-L6-v2. The newly merged scoring repair #204 improves the measurement contract; it is not a model A/B result. The retained #95 report is still incomplete and does not establish a model bottleneck.Next step: Gennaro completes #95's header/dedup contract reconciliation and bounded evaluation plan first. Resume this issue only when usable held-out, paired results support testing a model change. Then compare candidates with separately rebuilt private indexes, recording quality, indexing/query cost and the original model/index rollback before proposing any shared cutover. No default switch, shared reindex, new benchmark or migration is performed by this disposition.
scbrown commentedon Oct 24, 2026
2026-10-24: Still gated on #95; its measurement-contract repairs (#204 and #210) are merged, but the bounded current-runtime evaluation and acceptance gates remain open.