Skip to content

fix(memory): rank startup lessons by rare-word overlap, not a capped count - #15161

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/lesson-rank-idf-score
Sep 30, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/lesson-rank-idf-score

Conversation

@unstablebrainiac

@unstablebrainiac unstablebrainiac commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A new session gets the stored rules most relevant to its first message, as the block header promises, not just the newest ones.

When a store holds more standing rules than the 37,000-character startup budget, the kept set should be the rules closest to the first message. For any first message longer than a short paragraph it is the newest rules instead: nearly every rule ties at the top score, and the stable sort falls back to write order. #14243 measured 493 of 495 rules tied on a ~2,000-character message.

Why it matters

Standing rules are corrections the user taught after a past mistake. On a large store, an old rule that is squarely on topic does not reach a session unless something rewrote it recently, so the agent repeats a mistake the user already corrected. Long first messages (task briefs, pasted errors, subagent spawns) are hit hardest, and the block header still says "relevant ones first".

Not a goal

  • Embedding the first message at startup. This keeps feat(context): bound default context and recall memory on demand #11996's no-embedding startup.
  • Changing the startup budgets, tier precedence (authored rules before untagged ones), or the greedy fill.
  • The JSONL store's ordering (order_by_request_relevance), semantic-memory retrieval, _keyword_score or _hybrid_score.
  • Merging near-duplicate stored rules.

What changed (motivation → approach → change)

With no query vector Before After
Row score shared tokens / 10, capped at 1.0 sum of log((N + 1) / (df + 0.5)) over shared tokens, divided by sqrt(row tokens)
Long first message almost every rule ties, so newest-first wins one rare shared term decides
Long row with only common words in common outranks a short, focused row ranks below it
Recall whose embed returned no vector the same capped count, through the same branch the same rarity score as startup
Query that has a vector hybrid score unchanged

Startup ranks lessons with no query vector, so the keyword half of the hybrid score is the whole ranking there. That half is min(overlap / 10, 1.0) and counts function words, so any rule sharing ten tokens with the message scores the maximum. The cut only started to matter again when #12431 and #12657 bound the rule tier to a budget, after #11996 had removed the embedding that used to break these ties.

rank_lessons in vector_memory_runtime/lessons.py now scores a no-vector query with rarity-weighted overlap, taking document frequency from the rows being ranked. It is the scorer #14243 proposes. Each request word is matched once, through its stem, so an inflected word does not count twice. Every weight is positive, so any overlap still beats none, a request matching nothing keeps the stored order, and the findings tier's admission test still agrees with the ranking. Document frequency is counted only for the request's tokens, over row token sets _row_stem_tokens_for_scan already memoises, so no row is stemmed again. The owning spec, docs/system-specs/modules/memory-skills-hooks.md, describes the scorer.

Backwards compatibility

Compatible: no API, config or stored data changes. Only the order of rows on the no-vector ranking path changes, and an empty or non-matching request still returns the stored newest-first order.

Tests

test/test_lesson_keyword_scoring.py, following #14243's acceptance criteria:

  • An older rule carrying a rare term is kept over 40 newer rules when a 1,500+ character request shares ten-plus tokens with every row and the budget fits about ten. Fails on main.
  • A short rule with the rare term outranks a row about 3x longer that adds only common words. Fails on main.
  • A request matching no row keeps newest-first.
  • Two rules that differ only in the inflection of the shared word ("certificate" and "certificates") rank by recency, not by the inflection. Fails without stem-only matching.
  • The startup render with a non-empty request never calls the embedder.
  • A second render stems only the request's words: rows come from the memo, so no row is re-stemmed.

test/test_lesson_ranking.py and test/test_lesson_startup_budget.py pass unchanged.

Manual verification

Replayed the startup render on a copy of a real 659-rule store, with the 37,000-character budget and no embedder. Counts only:

First message Rules tied at the top, before → after Kept rules that newest-first also keeps, before → after
Task brief, 1,347 chars 645 → 1 58 of 59 → 21 of 51
Paragraph, 168 chars 17 → 1 20 of 53 → 17 of 67
One line, 29 chars 5 → 1 21 of 60 → 9 of 69

For the task brief, kept rules naming push-fork went from 2 to 7, and ones about review dispositions from 3 to 12.

Related Issues

Fixes #14243

Pattern harvest

Rule candidate: review-prompt
Pattern: a saturating or capped score used as the only sort key, so ties silently fall back to input order

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Intent: Make the startup lesson block keep the stored rules most relevant to the first message, by ranking a no-vector query with rarity-weighted, length-normalised overlap instead of a count that saturates at ten shared tokens.
Not a goal: Embedding the first message at startup, changing the startup budgets or tier precedence, or changing the JSONL store, semantic retrieval, _keyword_score or _hybrid_score.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 29, 2026
@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed cbfe372c1b8d54b92de8263bd130b4d645cb2c6f via the fork AI-review pipeline; updated in place on each push.

Review details

FINDING -- src/kiro_crew/vector_memory_runtime/lessons.py:1238 -- len(tokens) counts surface forms plus stems, so inflections can change rank instead of preserving recency -> Fix: count one normalized token per source word.
[GPT-REVIEWED] cbfe372

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of cbfe372c1b8d54b92de8263bd130b4d645cb2c6f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] cbfe372

Description read

[DESCRIPTION-READ] a5dd1ce14554852777f0395358f9e8b659df41488a9011d1158f891999b9d351

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of cbfe372c1b8d54b92de8263bd130b4d645cb2c6f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One sibling stays capped: get_semantic_context (semantic.py:797) still uses the saturating _keyword_score as its whole score on a no-vector query — declared out of scope; confirm it is tracked.

What this change ships

Inventory (8 items) — 8 justified

Intent: FIX — a session's startup rule block keeps the stored rules relevant to the first message instead of the newest ones (#14243, tests fail on base).

  1. Long first message now keeps rare-term-matching rules over newer unrelated ones — justified
  2. A short focused rule outranks a longer row sharing only common words — justified
  3. A recall whose embed is unavailable ranks by the same rarity score — justified
  4. Inflection variants of one word no longer buy rank; equal-stem matches stay newest-first — justified
  5. Empty or non-matching request still renders the stored newest-first order byte-identically — justified
  6. Admission/ranking "same tokenization" pin rewritten to "admission covers every token ranking matches" — justified
  7. Owning spec paragraph updated in the same commit — justified
  8. New test file pinning the scorer, per the issue's acceptance criteria — justified

Checks run: _keyword_score( has 3 hits — the changed lessons branch, the definition, and one unfixed sibling at src/kiro_crew/vector_memory_runtime/semantic.py:797, which ## Not a goal defers along with the JSONL order_by_request_relevance. Stem-only query set is a subset of the admission test's surface-plus-stem set, so the rewritten pin's invariant (no tier suppressed for a match its own ranking found) still holds in the direction that matters. Not product shape: the block header ("most relevant first") and #14243 already pin the restored behaviour.

[FIRST-PRINCIPLES-REVIEWED] cbfe372

Description read

[DESCRIPTION-READ] 23591687716e1ca7389ef03dd4f64ba957ab8630aa725048a74d1365647622d9

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed cbfe372c1b8d54b92de8263bd130b4d645cb2c6f via the fork AI-review pipeline; updated in place on each push.

Review details

Un-embedded lesson rows still outrank well-matching embedded ones on the recall path, which now provably always carries a query vector.

FINDING — src/kiro_crew/vector_memory_runtime/lessons.py:1200 — after the new if not query_emb: early return the hybrid loop is reached only when the query IS embedded, yet score = _hybrid_score(_keyword_score(overlap), similarity(row)) still leaves query_has_vector at its False default, so a lesson the backfill has not embedded (similarity → 0.0) scores its full keyword value k while an on-topic embedded sibling with the same overlap scores at most 0.6·cos + 0.4·k — the exact inversion _hybrid_score's own docstring and semantic.py:809 guard against → Fix: pass query_has_vector=True and clamp the cosine at this call site (max(0.0, similarity(row))), matching semantic.py's comment that an unclamped negative cosine would otherwise sink below the keyword-only floor.

[OPUS-REVIEWED] cbfe372

@unstablebrainiac

unstablebrainiac commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Semantic-memory no-vector fallback still uses the saturating _keyword_score span=373c378af8e2

rebutted (out of scope): this PR's "Not a goal" names semantic-memory retrieval, and the finding changes nothing this PR ships.
get_semantic_context ranks facts with its own key-term weighting and admission rules, so the same saturation there needs its own measurement on a real fact store before its scorer changes.
Changing it here would alter which facts every no-vector recall returns, a separate decision from which startup rules survive the lesson budget.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 29, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/lesson-rank-idf-score branch from 589f802 to d6ae5e1 Compare September 29, 2026 23:54
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/memory-skills-hooks.md and 31 other files this PR shares a directory or import with, since this head went green (5d4f173 -> 31cec9a); patch unchanged, CI re-running.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/lesson-rank-idf-score branch from d6ae5e1 to 63d1691 Compare September 30, 2026 01:59
@unstablebrainiac

unstablebrainiac commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author
  • _lexical_lesson_scores docstring and spec overclaim that one rare term outranks any number of common words span=5e963e401217

fixed in 63d1691: the docstring and the owning spec sentence now say a distinctive term counts for far more than a common one, and that a token nearly every row carries weighs about 0.5 / N.
Neither text claims one rare term beats any number of common overlaps any more; enough common overlaps can still add up past it.
The scorer's arithmetic was already as described, so only the prose changed for this finding.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Pushed 63d1691, rebased onto main 9ac3e7f; main moved only in files this PR does not touch.

Besides the docstring fix above, the no-vector ranking now matches each request word once, through its stem. An inflected request word used to count twice: for "rotate the certificates", a rule saying certificates outranked an equally relevant rule saying certificate. TestInflectionDoesNotBuyRank fails without the fix. The replay numbers in the description are re-measured on this head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@unstablebrainiac

unstablebrainiac commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Not justified as shipped: a recall with no embedder ranks by the same rarity score span=563d3836e4eb (rebutted: kept as shipped)

Both callers reach the same if not query_emb: branch of rank_lessons: every startup render, and a recall only when its embed returned no vector.
Keeping the capped _keyword_score count for that recall would need a caller flag on rank_lessons, so one store would rank two ways for the same missing vector.
The capped count is the #14243 defect itself. At ten or more shared tokens every rule scores 1.0 and the stable sort returns newest-first, and a vector-less recall hits the same tie.
No new surface: _lexical_lesson_scores has one caller, and a recall that has a vector still takes the hybrid score, unchanged.

…count

Startup renders lessons with no query vector, so the keyword half of the
hybrid score is the whole ranking there. That half caps at ten shared
tokens, so a long first message tied nearly every stored rule at 1.0 and
the stable sort returned newest-first: once the rule tier overflows its
37,000-character budget, the rules kept were the newest, not the relevant.

With no query vector, rank_lessons now scores each row by the IDF-weighted
overlap of its tokens with the request (IDF over the rows being ranked),
divided by the square root of the row's token count. Each request word is
matched once, through its stem, so an inflected word does not count twice.
A query with a vector still uses the hybrid score; _keyword_score and
_hybrid_score are unchanged. Document frequency is counted only for the
request's tokens over the already-memoised row token sets, so no row is
stemmed again.

Fixes kirodotdev#14243
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 30, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/lesson-rank-idf-score branch from 63d1691 to cbfe372 Compare September 30, 2026 03:41
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • The cost test counted stems at a seam the lexical branch does not read span=0d674d1d8d84 (fixed in cbfe372)

rank_lessons takes the request's distinct words once (request_words). The no-vector branch stems each distinct word once, and the hybrid branch builds query_words only after the if not query_emb: return.
test_ranking_stems_only_the_request_once_rows_are_memoised counts at both lessons._stem_one (the request) and text_scoring._stem_one (row tokens), so a per-occurrence request stem or a re-stemmed row fails it.
The corrected test fails on 63d1691 (per-occurrence stems) and passes on cbfe372.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Pushed cbfe372. It carries the fix for the Opus cost-test finding (disposition above) and a rebase from 9ac3e7f onto 4b66d1e, where main had moved in files this PR imports (src/kiro_crew/sandbox.py and its tests); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

The hybrid call is unchanged from main. Main's rank_lessons also scores every row of an embedded query with _hybrid_score(_keyword_score(overlap), similarity(row)) and the False default, so an unembedded row already scores keyword-only there.
This PR adds only the no-vector branch above that loop. It changes no hybrid score.
#15238 changes this exact call to pass query_has_vector and max(0.0, similarity(row)), pinned by test_a_dissimilar_row_does_not_keep_its_unweighted_keyword_score and test_a_negative_cosine_counts_as_no_similarity.
The same edit here would put one change on the same line in both PRs.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 30, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. Only the no-vector path changes to the IDF/sqrt(len) scorer; vector path, budget, and the authored-before-untagged split are untouched, so 'always' lessons can be dropped exactly as before and no more. O(N*|q|) with memoised token sets.

Non-blocking: lessons.py:1246 sums over a frozenset, so with 3+ shared tokens exact ties can differ in the last bit between processes and break the recency fallback differently; math.fsum over sorted(shared) would make it deterministic. Nothing tests the embed-returns-None recall path.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 30, 2026 05:28
@iamwhatever
iamwhatever merged commit 7a895f4 into kirodotdev:main Sep 30, 2026
100 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Legitimate. _lexical_lesson_scores sized a row by its token set, every word plus every stem that differs from its word, so an inflected rule read as longer and lost its recency.
#15397 divides by the row's number of distinct words instead. Its new test fails on this head and passes on the fix.
This PR merged before the fix landed, so the fix ships as that follow-up.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

memory: lesson score saturates, so long prompts inject newest-first

2 participants