Repository navigation
fix(memory): rank startup lessons by rare-word overlap, not a capped count - #15161
Conversation
|
Intent: Make the startup lesson block keep the stored rules most relevant to the first message, by ranking a no-vector query with rarity-weighted, length-normalised overlap instead of a count that saturates at ten shared tokens. |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsFINDING -- src/kiro_crew/vector_memory_runtime/lessons.py:1238 -- |
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] cbfe372 Description read[DESCRIPTION-READ] a5dd1ce14554852777f0395358f9e8b659df41488a9011d1158f891999b9d351 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One sibling stays capped: What this change shipsInventory (8 items) — 8 justifiedIntent: FIX — a session's startup rule block keeps the stored rules relevant to the first message instead of the newest ones (#14243, tests fail on base).
Checks run: [FIRST-PRINCIPLES-REVIEWED] cbfe372 Description read[DESCRIPTION-READ] 23591687716e1ca7389ef03dd4f64ba957ab8630aa725048a74d1365647622d9 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5 Review (fork) — ✅ no blocking findingsReviewed Review detailsUn-embedded lesson rows still outrank well-matching embedded ones on the recall path, which now provably always carries a query vector. FINDING — src/kiro_crew/vector_memory_runtime/lessons.py:1200 — after the new [OPUS-REVIEWED] cbfe372 |
|
589f802 to
d6ae5e1
Compare
d6ae5e1 to
63d1691
Compare
|
|
Pushed 63d1691, rebased onto main 9ac3e7f; main moved only in files this PR does not touch. Besides the docstring fix above, the no-vector ranking now matches each request word once, through its stem. An inflected request word used to count twice: for "rotate the certificates", a rule saying |
|
…count Startup renders lessons with no query vector, so the keyword half of the hybrid score is the whole ranking there. That half caps at ten shared tokens, so a long first message tied nearly every stored rule at 1.0 and the stable sort returned newest-first: once the rule tier overflows its 37,000-character budget, the rules kept were the newest, not the relevant. With no query vector, rank_lessons now scores each row by the IDF-weighted overlap of its tokens with the request (IDF over the rows being ranked), divided by the square root of the row's token count. Each request word is matched once, through its stem, so an inflected word does not count twice. A query with a vector still uses the hybrid score; _keyword_score and _hybrid_score are unchanged. Document frequency is counted only for the request's tokens over the already-memoised row token sets, so no row is stemmed again. Fixes kirodotdev#14243
63d1691 to
cbfe372
Compare
|
|
iamwhatever
left a comment
There was a problem hiding this comment.
Reviewed. Only the no-vector path changes to the IDF/sqrt(len) scorer; vector path, budget, and the authored-before-untagged split are untouched, so 'always' lessons can be dropped exactly as before and no more. O(N*|q|) with memoised token sets.
Non-blocking: lessons.py:1246 sums over a frozenset, so with 3+ shared tokens exact ties can differ in the last bit between processes and break the recency fallback differently; math.fsum over sorted(shared) would make it deterministic. Nothing tests the embed-returns-None recall path.
|
Problem / Motivation
Goal: A new session gets the stored rules most relevant to its first message, as the block header promises, not just the newest ones.
When a store holds more standing rules than the 37,000-character startup budget, the kept set should be the rules closest to the first message. For any first message longer than a short paragraph it is the newest rules instead: nearly every rule ties at the top score, and the stable sort falls back to write order. #14243 measured 493 of 495 rules tied on a ~2,000-character message.
Why it matters
Standing rules are corrections the user taught after a past mistake. On a large store, an old rule that is squarely on topic does not reach a session unless something rewrote it recently, so the agent repeats a mistake the user already corrected. Long first messages (task briefs, pasted errors, subagent spawns) are hit hardest, and the block header still says "relevant ones first".
Not a goal
order_by_request_relevance), semantic-memory retrieval,_keyword_scoreor_hybrid_score.What changed (motivation → approach → change)
log((N + 1) / (df + 0.5))over shared tokens, divided bysqrt(row tokens)Startup ranks lessons with no query vector, so the keyword half of the hybrid score is the whole ranking there. That half is
min(overlap / 10, 1.0)and counts function words, so any rule sharing ten tokens with the message scores the maximum. The cut only started to matter again when #12431 and #12657 bound the rule tier to a budget, after #11996 had removed the embedding that used to break these ties.rank_lessonsinvector_memory_runtime/lessons.pynow scores a no-vector query with rarity-weighted overlap, taking document frequency from the rows being ranked. It is the scorer #14243 proposes. Each request word is matched once, through its stem, so an inflected word does not count twice. Every weight is positive, so any overlap still beats none, a request matching nothing keeps the stored order, and the findings tier's admission test still agrees with the ranking. Document frequency is counted only for the request's tokens, over row token sets_row_stem_tokens_for_scanalready memoises, so no row is stemmed again. The owning spec,docs/system-specs/modules/memory-skills-hooks.md, describes the scorer.Backwards compatibility
Compatible: no API, config or stored data changes. Only the order of rows on the no-vector ranking path changes, and an empty or non-matching request still returns the stored newest-first order.
Tests
test/test_lesson_keyword_scoring.py, following #14243's acceptance criteria:test/test_lesson_ranking.pyandtest/test_lesson_startup_budget.pypass unchanged.Manual verification
Replayed the startup render on a copy of a real 659-rule store, with the 37,000-character budget and no embedder. Counts only:
For the task brief, kept rules naming
push-forkwent from 2 to 7, and ones about review dispositions from 3 to 12.Related Issues
Fixes #14243
Pattern harvest
Rule candidate: review-prompt
Pattern: a saturating or capped score used as the only sort key, so ties silently fall back to input order
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)