Skip to content

feat(memory): add matching lessons to follow-up messages - #15517

Merged
bolichen97 merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:feat/per-message-lessons
Oct 1, 2026
Merged

bolichen97 merged 1 commit into
kirodotdev:mainfrom
unstablebrainiac:feat/per-message-lessons

Conversation

@unstablebrainiac

@unstablebrainiac unstablebrainiac commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A session can receive a stored lesson when its conversation reaches the lesson's topic, not only when its first message did.

Lessons reach a session once, at session start, ranked against the first message and cut to a 37,000-character budget. A store holding more rules than that sends the rest to no session that did not open on their topic. When the conversation moves on, a rule it now needs never arrives, and a rule saved with learn_add mid-session is not seen until the next session.

Why it matters

The costliest mistakes happen after the first message: a session that opens on one task and moves to another runs the second task without the corrections the user taught for it. On a store past the budget, most rules are in this position on any given session.

Not a goal

  • Changing what any session gets by default: the behaviour is behind memory.inject_lessons_per_turn, off by default. Turning it on by default is a product decision for an RFC.
  • Embedding each message: selection is lexical and spends no inference.
  • The JSONL fallback store, or lessons on a session's first message (the session-start block is unchanged).
  • Re-sending the session-start block after a compaction.

What changed (motivation → approach → change)

With memory.inject_lessons_per_turn on, a follow-up message that shares at least two rare words with a stored lesson gets that lesson, once per session, in a short block before the message.

Follow-up message, flag on Behaviour
Shares two or more words with a stored lesson, each found in at most 1% of stored lessons (never fewer than one) up to 3 lessons, their lines at most 2,000 characters, in a [Learned corrections — relevant to this message …] block
Shares only one such word, or only words common in the store nothing
Only stop-list words ("yes", "do", "it", "this", "that"), or words of two letters or fewer nothing
Text the runtime puts before the message (a cancelled-turn notice, subagent failures, app context) not matched; only what the user typed counts
Lesson already in the session-start block or an earlier per-message block skipped
After a compaction re-injection the record resets, so a lesson can return
Temporary session, minimal-context turn, lessons group withheld, inject_lessons off nothing

The match reads the user's own text (the user_text_range slice, or a transform hook's output), because every lesson it picks is recorded as shown. VectorMemoryStore.turn_lessons (a vector_memory.py facade over vector_memory_runtime/lessons.py) drops message words on the lesson stop list and words of two letters or fewer, then matches each remaining word once, through its stem. A lesson is admitted only when it shares at least two distinct stems with the message that each appear in at most 1% of the stored lessons, and never fewer than one, so a small store still admits through words no other lesson carries. The rule counts rare words rather than summing their rarity weights, because a sum lets one word found in a single lesson admit that lesson alone, and in a store of a few hundred lessons it counts ordinary words as rare. Admitted lessons are ordered by summed rarity weight over the square root of their number of distinct words (the #15161 scorer, split into _shared_lesson_stems and _rarity_weights); rules and findings are both eligible.

context.py keeps, per session and keyed by a fixed-size digest of the session key, the session-start block as sent and a hash of each lesson sent since (_ShownLessons: 256 sessions of 256 lessons, oldest dropped first). It reads the same store the session-start block reads under the same gates, checks its choice again against the record once the store has been read, and resets the record when a compaction re-injection runs. A stored lesson can come from an inferred or imported rule, so each line is scrubbed of boundary, member-authority, lesson-frame and skill-frame markers (_scrub_turn_lesson). The record is in memory and bounded: a gateway restart, turning the flag on mid-session, 256 newer lessons in the same session, or the record dropping the whole session can send a lesson once more. The 2,000-character cap counts the scrubbed lesson lines the block shows, not its header and footer, and a lesson left out for size is not counted as shown. A lesson counts as shown when its block is built, so a cancelled or failed turn withholds it until the next compaction.

The flag is declared in config/memory_sections.py and parsed in config/section_builders.py; config-baseline.json carries the new key. The memory spec, the config spec, docs/configuration.md and docs/architecture/context-management.md document it.

Backwards compatibility

Compatible: the flag defaults to off, and with it off no code path changes what a session receives. _lexical_lesson_weights, built from _shared_lesson_stems and _rarity_weights, and the shared _renderable_entries helper return exactly what the code they came from returned, so the session-start ranking is unchanged.

Tests

test/test_lesson_turn_injection.py:

  • Two rare shared words earn a lesson and one does not; one rare word beside four words carried by half the store earns nothing; common words and "do it" earn nothing, in a large store and in a store of two lessons, where shared content words still earn the lesson; a six-lesson store admits through words no other lesson carries; a shown lesson is skipped; the row and character caps hold.
  • With the flag on, a matching follow-up gets the lesson once, never a lesson already in the session-start block, and the lesson can return after a compaction re-injection.
  • A notice put before the typed text neither picks nor uses up a lesson, the typed text behind it still gets its lesson, and a transform hook's output is what is matched.
  • Off by default, and absent when inject_lessons is off, for a temporary session and for a minimal-context turn.
  • A choice is checked again when the session's record is replaced or dropped during the store read, and the record keeps at most 256 lessons per session. Both retention limits, 256 sessions and 256 lessons each, are pinned to the literal the docs state.
  • A lesson line loses every frame, member-authority and boundary marker, a fullwidth forgery included, and the character cap holds after that scrub and after the punctuation fold the turn applies: a lesson too long once scrubbed or folded is skipped for a later one that fits, and can still arrive on a later message.
  • The documented limits are measured through the production paths, not read from the constants: four matching lessons give exactly three lines, the lines fit in 2,000 characters, and in a 20-lesson store two message words that each appear in one lesson admit it while one such word, or two words that each appear in two lessons, do not. Every operator-facing copy is checked for those values, and changing any of the four limits fails its test.

test/test_vector_memory_composition_contract.py pins the facade's signature, with the two limits keyword-only and no default. test/test_config_baseline.py checks the regenerated config-baseline.json. Each gate was checked by removing it, and the test written for it failed every time; the one-rare-word and six-lesson tests both fail against a rule that sums rarity weights instead of counting rare words.

Manual verification

N/A. Unit coverage exercises the builder end to end (build_message against a real vector store); the flag is off by default, so no existing session changes.

Screenshots / video

Why no screenshot: this PR changes no file under website/. The Screenshot Evidence check diffs against the base GitHub recorded before the rebase, so it would count frontend files that main changed in between.

Related Issues

no linked issue: this adds an opt-in capability; #14243 (fixed by #15161) is about the session-start ranking it builds on.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Intent: Let a session receive a stored lesson when its conversation reaches that lesson's topic, not only when its first message did, behind memory.inject_lessons_per_turn (off by default).
Not a goal: Changing what any session gets by default, embedding each message, the JSONL fallback store, lessons on a session's first message, or re-sending the session-start block after a compaction.

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Sep 30, 2026
@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of fe7069d1dc68d08211e9d62d251055fe93ac5188 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Admission rule may rarely fire on the motivating case: topic words shared by more than max(1, 1%·N) lessons can never count as rare.

Watch

  • The PR's motivation is a large store ("most rules are in this position"), but lessons cluster by topic, and a stem carried by even a handful of lessons in a few-hundred-lesson store exceeds the 1% rarity cap — so a follow-up that "reaches the lesson's topic" through the topic's own vocabulary admits nothing unless it also shares two per-lesson-unique words. Every test seeds synthetic unique words (flywheel, zephyr quartz), so recall on a realistic clustered store is unmeasured; the flag ships, an operator turns it on, and the feature silently delivers nothing for exactly the rules the budget cut. Reversible (constants, off-by-default flag), so advisory.
    Clears when: a match-rate measurement against a realistic lesson store (or a test with clustered-topic lessons) shows the two-rare-stems rule still surfaces them, or the author states per-lesson-unique terms are the intended trigger and adjusts the description's "reaches the lesson's topic" claim accordingly.

[DESIGN-REVIEWED] fe7069d

Description read

[DESCRIPTION-READ] c0d862c1945f56906196aac61026a01b7c7395a516c691c3abb6c3087316cb2e

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of fe7069d1dc68d08211e9d62d251055fe93ac5188 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

With the flag on, every follow-up message scans and stems the whole lesson store lexically — verify that cost stays negligible on a store of thousands of lessons.

What this change ships

Inventory (9 items) — 9 justified

Intent: let a session receive a stored lesson when the conversation reaches its topic, not only at session start — an ADDITION, gated off by default (explicitly not product shape: a new flag nobody gets without choosing, with the default change deferred to an RFC).

  1. New config key memory.inject_lessons_per_turn, off by default — justified
  2. Flag on: a follow-up sharing two rare words with a stored lesson gets up to 3 lessons / 2,000 chars in a labelled block — justified
  3. A lesson arrives at most once per session and may return after a compaction — justified
  4. Each lesson line is scrubbed of frame and authority markers before injection — justified
  5. New callable store method VectorMemoryStore.turn_lessons — justified
  6. Session-start lesson ranking split into shared helpers, output unchanged — justified
  7. Runtime-prefixed notices neither pick nor spend a lesson; only the user's own text matches — justified
  8. The key documented in the config/memory specs and user docs — justified
  9. config-baseline.json regenerated for the new key — justified

Counts run: turn_lessons has 1 production consumer (_turn_lessons_block) behind the facade pattern vector_memory.py already uses for every _lessons delegate; the extracted helpers (_renderable_entries, _shared_lesson_stems, _rarity_weights) each have 2 consumers (the pre-existing session-start path and the new per-turn path), so the extraction is reuse, not surface. _scrub_turn_lesson composes the existing _scrub_member_payload / _apply_marker_spans / _MULTIBYTE_TABLE scrubs rather than re-implementing them, and its harm has a named cause (lessons are imported/inferred external content). No existing mechanism pushes a lesson mid-session — memory_recall is agent-initiated pull, and the base spec confirms a warm follow-up receives no memory content.

[FIRST-PRINCIPLES-REVIEWED] fe7069d

Description read

[DESCRIPTION-READ] 6fa4a8cae915b1e33772fd6c26be4b36c3db2ac24523f2ea930d9db97cb336f8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed fe7069d1dc68d08211e9d62d251055fe93ac5188 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fe7069d

@github-actions

github-actions Bot commented Sep 30, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed fe7069d1dc68d08211e9d62d251055fe93ac5188 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fe7069d

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 30, 2026
@unstablebrainiac
unstablebrainiac force-pushed the feat/per-message-lessons branch from 0fdfb51 to 5cb3704 Compare October 1, 2026 01:18
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Design Review on 0fdfb51 asked: "Verify the 2-term/5% admission bar against a real lesson store -- the precision thresholds are pinned only on synthetic fixtures." Verified, and the bar was too loose; 5cb3704 replaces it.

  • On a copy of a real store (747 lessons), over 300 real follow-up messages, the summed bar admitted lessons for 84% of messages, a median of 12 candidates each. In a hand-read sample, about one delivered lesson in eight fit its message.
  • The sum is the cause: one word found in a single lesson clears the bar alone, and at 5% an ordinary word counts as rare in a store of this size.
  • The rule now counts rare words. A lesson needs at least two distinct shared stems that each appear in at most 1% of the stored lessons, and never fewer than one. On the same 300 messages it adds lessons to 14% of them, and for a hand-read sample of 12 of those messages, 11 of the 17 delivered lessons fit.
  • Two tests pin it: one rare word beside four words carried by half the store admits nothing, and a six-lesson store still admits through words no other lesson carries. Both fail against the summed rule, and the second also fails without the floor of one.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

self-added: no

  • turn_lessons limit defaults restate context.py's constants with no production user span=8b043e4457a4 (fixed in 5cb3704)

Both signatures drop the max_rows=3 / max_chars=2_000 defaults: vector_memory_runtime/lessons.py::turn_lessons and the VectorMemoryStore.turn_lessons facade. Both limits are now keyword-only with no default.
The one production caller, context.py::_turn_lessons_block, already passes _TURN_LESSONS_MAX and _TURN_LESSONS_CHARS. The tests pass them through one helper that imports those constants, and the composition-contract test pins the new signature.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

self-added: no

  • Spec says the block is at most 2,000 characters, but only the lesson lines are capped span=5f43642af427 (fixed in 5cb3704)

turn_lessons counts len(rendered) + 3 per chosen line against max_chars, and _turn_lessons_block adds the header and footer after that count.
The memory-skills-hooks.md bullet now says the block holds at most _TURN_LESSONS_MAX (3) lessons whose lines total at most _TURN_LESSONS_CHARS (2,000) characters, with the header and footer outside that count.
The copies in docs/configuration.md, docs/architecture/context-management.md and the config spec describe the lessons, not the block, so they stay as they are.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@unstablebrainiac
unstablebrainiac force-pushed the feat/per-message-lessons branch from 5cb3704 to 1713e2e Compare October 1, 2026 02:33
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Pushed 1713e2e: it fixes the three advisory GPT findings on 5cb3704 (dispositions below) and rebases onto main 7460284, which carries the Windows lock-file sweep fix (#15633). Main touched none of this PR's files. CI re-running.

The one red check on 5cb3704, Windows shard 2, fails the same two test/test_native_skill_projection.py lock-file tests on main's own CI at 321fd99, so it is not this PR's.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • In-method member_context_identity import lacks the circular-import explanation span=451b2d4903f8: fixed in 1713e2e

Fixed by moving the import rather than commenting it. context.py already imports kiro_crew.member_essential_context at module level, and that module imports kiro_crew.context only inside a function, so no cycle exists for a comment to explain.
member_context_identity now joins the module-level import from that module, and the in-method import in _turn_lessons_block is gone.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • Lesson budget counted before the final punctuation fold span=451b2d4903f8: fixed in 1713e2e

Fixed: _scrub_turn_lesson returns its scrubbed text folded through _MULTIBYTE_TABLE, so turn_lessons budgets the characters the turn actually sends, and the final whole-turn fold is a no-op on these lines.
The fold runs after marker neutralization, as the whole-turn fold already did, and every marker matcher reads a folded view, so the fold cannot create a marker that survives.
New test test_final_punctuation_fold_is_included_in_character_budget: three lessons that fit the budget raw but not folded admit two lines within 2,000 characters; without the fix it admits three.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author
  • Retention-limit assertions derived from the production constants span=051d95b6d20d: fixed in 1713e2e

Fixed: test_operator_facing_copies_state_the_measured_bounds asserts _LESSONS_SHOWN_SESSIONS == 256 and _LESSONS_SHOWN_PER_SESSION == 256, so either constant drifting from the documented 256 fails the test.

@unstablebrainiac
unstablebrainiac force-pushed the feat/per-message-lessons branch from 1713e2e to 8b66512 Compare October 1, 2026 03:10
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in files this PR touches or sits beside (src/kiro_crew/context.py, src/kiro_crew/context_blocks.py, src/kiro_crew/dashboard/, src/kiro_crew/acp/session_mcp.py, docs/system-specs/modules/) since 1713e2e's run (7460284 -> 309d005). The rebase was clean. In context.py, #15643 (the interrupted-turn restore) added an import next to one of this PR's import lines; this PR's own lines are unchanged (zero-context patch-id equal). The PR's tests and the context, dashboard chat and turn-restart tests pass locally on the rebased commit. CI re-running.

The red checks on 1713e2e were outside this PR:

  • The GPT, Opus, Security Scope and Internal Content Scan lanes could not run, and Fork PR Description and Resolve desktop matrix failed with them. Each job hit GitHub's API rate limit exceeded for installation between 02:27 and 02:38 UTC.
  • Linux shard 3: test/test_session_export.py::test_a_sent_export_streams_its_staged_file_and_removes_it, a staged-file removal timing check that failed the same way on fix(memory): rank startup lessons with the first message's vector #15238. This PR does not touch export code.
  • Coverage Gate failed only because shard 3 did.

…y.inject_lessons_per_turn

With the flag on (off by default), each follow-up message can carry up to
three stored lessons that match it on words rare among stored lessons and
that the session has not been shown: the session-start block holds only
what fits its budget. A per-session record of the startup block and a
bounded hash of each lesson sent since keeps a lesson from repeating; it
is checked again after the store read, reset on a compaction
re-injection, and each lesson line is scrubbed of boundary,
member-authority, lesson-frame and skill-frame markers.
@unstablebrainiac
unstablebrainiac force-pushed the feat/per-message-lessons branch from 8b66512 to fe7069d Compare October 1, 2026 03:54
@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

Rebased onto main ecdc59a (from 309d005) to re-run the review lanes. This PR's diff is unchanged (patch-id equal), and main touched none of its files. CI re-running.

On 8b66512 every other check passed, but no lane verdict reached the PR:

  • GPT, Design and First Principles finished, and none of their comments was updated. The GPT job log shows it withheld its verdict after GitHub's API rate limit exceeded for installation at 03:21 UTC.
  • Opus 5 Review ended "review incomplete" in the same window.

@unstablebrainiac

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • Watch: topic-clustered lessons may miss the 1% rarity cap span=464a33e3667f: rebutted

Words that set one lesson apart are the intended trigger. The What changed table states it: a follow-up that shares only words common in the store gets nothing.
Measured on a copy of a real 747-lesson store against 4,385 human follow-up messages from session transcripts: the shipped rule adds a lesson to 561 messages (12.8%). In a hand read of 15 delivered pairs, 7 fit the message.
Letting words carried by up to 5% of lessons count (37 lessons here, enough for a topic cluster) adds a lesson to 2,473 messages (56.4%). In a hand read of 15 of those added lessons, 3 fit.
So the lessons the 1% cap rejects through topic-wide words are mostly off-topic. A lesson inside a cluster still arrives when the message uses the words that set it apart, such as a tool, analyzer, dataset or account name.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved: PR Readiness passed on this head, no open change requests.

@bolichen97
bolichen97 merged commit 600d4b9 into kirodotdev:main Oct 1, 2026
100 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 1, 2026
chenmingwei23 pushed a commit to chenmingwei23/KiroCrew that referenced this pull request Oct 1, 2026
kirodotdev#15688 replaced lessons.py's by-name imports from text_scoring with reads through the owner module. kirodotdev#15517, which merged after it, added turn_lessons with one bare _row_stem_tokens_for_scan call, so main raises NameError: name '_row_stem_tokens_for_scan' is not defined on every turn_lessons call (flake8 F821). Read it through _text_scoring like the other callers in the file.
bolichen97 pushed a commit that referenced this pull request Oct 1, 2026
…15744)

* fix(crew-log): regenerate the fold catalogue for the subagents fold

#15078 added a `subagents` projection fold but did not add its answer
sentence to the catalogue generator or regenerate the committed
catalogue, so test_fold_catalogue.py reddened on main d5bdff6 (shard
4, Linux + Windows): the generator refused to build (no `_ANSWERS`
sentence for `subagents`) and the committed folds.json / FOLDS.md no
longer matched the registry.

Add the `subagents` sentence to `_ANSWERS` and regenerate both
artifacts with `python3 scripts/fold_catalogue.py --write`.

* fix(memory): read turn_lessons' row-token helper through text_scoring

#15688 replaced lessons.py's by-name imports from text_scoring with reads through the owner module. #15517, which merged after it, added turn_lessons with one bare _row_stem_tokens_for_scan call, so main raises NameError: name '_row_stem_tokens_for_scan' is not defined on every turn_lessons call (flake8 F821). Read it through _text_scoring like the other callers in the file.

---------

Co-authored-by: kirocrew-agent <agent@kirocrew.local>
Co-authored-by: Sajal Narang <sajalnarang@gmail.com>
chenmingwei23 pushed a commit to chenmingwei23/KiroCrew that referenced this pull request Oct 1, 2026
kirodotdev#15688 replaced lessons.py's by-name imports from text_scoring with reads through the owner module. kirodotdev#15517, which merged after it, added turn_lessons with one bare _row_stem_tokens_for_scan call, so main raises NameError: name '_row_stem_tokens_for_scan' is not defined on every turn_lessons call (flake8 F821). Read it through _text_scoring like the other callers in the file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants