Skip to content

fix(memory): count a lesson's length in distinct words - #15397

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/lesson-length-distinct-words
Sep 30, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:fix/lesson-length-distinct-words

Conversation

@unstablebrainiac

@unstablebrainiac unstablebrainiac commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A stored rule written with inflected words ranks as high as an equally long rule written without them.

Startup lesson ranking (#15161) divides each row's rarity score by the square root of the row's length. That length was the size of the row's token set, which holds every word plus every stem that differs from its word. A rule written with "tests" and "pushing" therefore read as longer than a rule of the same length written with "test" and "push". When a first message matched both, the newer inflected rule ranked below the older one instead of keeping its recency.

Why it matters

The divisor exists so a long row cannot win on incidental overlap. Counting stems as extra length turns it into a penalty on ordinary English: plurals and "-ing" or "-ed" forms. Rules written in plain prose rank below rules written in base forms, and on a store that overflows the startup budget the lower rank is the one that gets cut.

Not a goal

What changed (motivation → approach → change)

A lesson's length now counts each of its distinct words once, so a rule written with plurals or "-ing" forms no longer reads as longer than the same rule in base forms. _lexical_lesson_scores in vector_memory_runtime/lessons.py divides by the square root of that count, instead of the size of the row's word-and-stem token set. Matching still runs on the stemmed token set, so only the length term changes. The owning spec paragraph in docs/system-specs/modules/memory-skills-hooks.md names the new divisor, and no longer promises that rules which fit within both budgets keep their old order. The test module's docstring names the divisor too.

Backwards compatibility

Compatible: no API, config or stored data changes. Only the order of rows on the no-vector ranking path changes, and only between rows whose words differ in how many have a separate stem.

Tests

test/test_lesson_keyword_scoring.py: a newer rule written with inflections ("tests", "pushing") keeps its recency over an older rule with the same number of distinct words when the request matches both. The newer rule repeats one word, so the test fails both on main and with a divisor that counts every word. The existing ranking tests pass unchanged.

Manual verification

N/A: unit coverage is sufficient. The change is one term of a pure scoring function, and the new test pins it against main's behaviour.

Related Issues

no linked issue: follow-up to #15161, which added this scorer for #14243.

Pattern harvest

Rule candidate: review-prompt
Pattern: a length normaliser measured on an expanded token set (words plus stems), so the expansion itself is counted as length

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@unstablebrainiac
unstablebrainiac requested a review from a team as a code owner September 30, 2026 11:31
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Intent: Make the startup lesson scorer's length divisor count a row's distinct words, so inflected wording no longer reads as extra length and costs a rule its rank.
Not a goal: Changing how rows match the request, the rarity weights, or the hybrid score used when the query has a vector.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 7cd5a82519d36f01a6103ca54003954883ef7807 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] 7cd5a82

Description read

[DESCRIPTION-READ] 6f7606fc0cf2c9f65042d86667e1c0dd742246c755ca8486962e2f4993b83d4a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 7cd5a82519d36f01a6103ca54003954883ef7807 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7cd5a82

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 7cd5a82519d36f01a6103ca54003954883ef7807 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new length term inlines the exact body of _tokenize — a second spelling of an existing tokenizer; the fix itself is pinned by a test failing on base.

Not justified as shipped

  • Item 2 — duplicate of src/kiro_crew/vector_memory_runtime/text_scoring.py: len(set(re.findall(r"\w+", normalized_text))) is character-for-character the body of _tokenize (text_scoring.py:236, set(re.findall(r"\w+", text.lower()))); grep for re.findall(r"\w+" shows the helper plus this new inline copy in a module that already imports five siblings from text_scoring (lessons.py:39).
    Subtraction: replace the inline expression with len(_tokenize(text)) and drop normalized_text.
    Clears when: the inline regex is replaced by the existing helper, or a reason the helper cannot serve is stated.

What this change ships

Inventory (3 items) — 2 justified

Intent: FIX — a rule written in ordinary inflected English should not rank below an equally long base-form rule in startup lesson selection. The new test fails on base, so the defect's provenance is checkable; grep of math.sqrt|len(tokens) across vector_memory_runtime finds no unfixed sibling divisor (history_search normalizes by chars, the JSONL path never stems).

  1. Inflected rules keep rank and recency against equal-length plain-form rules on the no-vector ranking — justified
  2. The length divisor is computed by a new inline word tokenizer — duplicate of src/kiro_crew/vector_memory_runtime/text_scoring.py (_tokenize)
  3. Owning spec's divisor sentence and its below-budget byte-identical promise rewritten, declared in the description — justified

[FIRST-PRINCIPLES-REVIEWED] 7cd5a82

Description read

[DESCRIPTION-READ] c96d8c97f2d3554dd4dd45cfc23f2d3050495a39d1246f1b46e8c902c9b5ef74

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 7cd5a82519d36f01a6103ca54003954883ef7807 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7cd5a82

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@unstablebrainiac
unstablebrainiac force-pushed the fix/lesson-length-distinct-words branch from dac7b0d to ba60cfd Compare September 30, 2026 14:39
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • Length fixture did not separate distinct words from total words span=e7633c533c47 (fixed in ba60cfd)

newer now ends with a second "tests": nine distinct words and ten in total, against nine and nine in older.
A divisor that counts every word ranks older first and fails the test, and so does the pre-PR len(tokens).
The distinct-word divisor passes, and the ordering assertion is unchanged.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • Verify the dropped "byte-identical below both budgets" promise is an accepted cost (rebutted: accepted, it is the fix)

The old sentence promised that adding the budgets changed nothing below them. This PR changes the ranking itself, so no fix to it could keep that promise: a new divisor reorders any render where rows share words with the first message.
Below both budgets nothing is dropped. Every eligible entry is still kept (the spec's new sentence); only the order moves, and order is what the model reads first, which is the defect being fixed.
When no row shares a word with the message every score is zero and the stable sort keeps the previous newest-first order, so those renders are unchanged.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 30, 2026
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • Verify the rewritten spec promise: below-budget renders reorder too (rebutted: accepted, it is the fix; same ruling as on dac7b0d)

The retired "byte-identical below both budgets" sentence promised that adding the budgets changed nothing below them. This PR changes the ranking itself, so any divisor fix reorders renders where rows share words with the first message.
Below both budgets nothing is dropped: every eligible entry is still kept, only its order moves, and order is what the model reads first, which is the defect being fixed.
When no row shares a word with the message every score is zero and the stable sort keeps the previous newest-first order, so those renders are unchanged. The description declares the change.

The rarity score behind startup lesson ranking divides by the square root of a
row's length. It measured that length as the row's token set, which holds every
word plus every stem that differs from its word, so a rule written with
inflections ("tests", "pushing") read as longer and ranked below an older rule
of the same length. The divisor is now the row's number of distinct words.
@unstablebrainiac
unstablebrainiac force-pushed the fix/lesson-length-distinct-words branch from ba60cfd to 7cd5a82 Compare September 30, 2026 15:32
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/subagent.md, src/kiro_crew/acp/skill_projection.py, src/kiro_crew/dashboard/handlers/messaging.py, src/kiro_crew/mcp_tools/spawn.py, test/test_crew_routing_and_delegation.py and test/test_native_skill_projection.py since this head went green (aec8141 -> 2b91a6b); CI re-running.

The patch is unchanged (git patch-id identical), so the new head 7cd5a82 carries the same diff that was reviewed on ba60cfd. Locally, test/test_lesson_keyword_scoring.py plus the two main-changed test files pass on the new head (212 passed).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Sep 30, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed. The length divisor now counts distinct words of the row instead of the stemmed token set, so an inflected newer row no longer takes a bigger penalty than its plain twin; the new test pins the tie-then-recency case with equal distinct-word counts.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 30, 2026 17:31
@iamwhatever
iamwhatever merged commit acd219f into kirodotdev:main Sep 30, 2026
100 checks passed
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • The length term duplicates _tokenize span=e2276d2b2aad (accepted, not landed: this PR merged on this head first)

Accepted: len(set(re.findall(r"\w+", normalized_text))) is _tokenize(text) written inline (text_scoring.py:236-238), the same pattern over the same lowered text, so the two give the same count for every input.
The subtraction was made and reviewed locally, but the PR merged on 7cd5a82 before it was pushed. It changes no behaviour, so a follow-up can land it.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants