Skip to content

fix(memory): consolidation states the lesson applies tier - #12875

Merged
bolichen97 merged 2 commits into
mainfrom
fix/consolidation-lesson-applies-12495
Sep 23, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
fix/consolidation-lesson-applies-12495

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

What

History consolidation now states the startup lesson tier (applies) it extracts, on both lesson write paths.

  • src/kiro_crew/history_consolidation.py
    • The lessons key of the consolidation prompt asks for "applies": "always|on_topic" per item, with the same instruction learn_add carries: decide it from what the user actually said; always is a standing rule, on_topic a past finding; omit the key when you cannot tell.
    • New _lesson_tier(item) calls extracted_lesson_applies (in lesson_validation.py, shared with the member-store path) once per item: it runs normalize_lesson_applies; absent / blank -> None (unstated); a value that is not one of the two literals -> warning log (reason only, never the untrusted value) and the row is still written unstated, never dropped.
    • _save_lessons passes the tier to vector_store.write_lesson(applies=...) and to Lesson(applies=...).
  • docs/system-specs/modules/memory-skills-hooks.md: the sentence saying consolidation states no tier, and the compatibility note, now describe the new behaviour.
  • src/kiro_crew/vector_memory.py: apply_consolidation (member-store path) persists the tier through the same extracted_lesson_applies helper, so the two paths cannot drift.
  • src/kiro_crew/lesson_validation.py + src/kiro_crew/mcp_tools/learn.py: the tier instruction lives in one constant, LESSON_APPLIES_INSTRUCTION, read by both learn_add's schema and the consolidation prompt.
  • test/test_history.py: TestConsolidationLessonApplies (7 tests); test/test_member_memory_database.py: test_consolidation_persists_the_authored_lesson_tier.

Why

Closes #12495. #12431 introduced the authored applies tier and the two startup budgets, but consolidation, the highest-volume lesson writer, never reached it: its prompt asked only for rule / negative / category / repo_scope, and _save_lessons wrote source="consolidation" rows with no tier. Every consolidated row therefore landed unstated and was served as a standing rule, so the budget split protected nothing for the rows that make up most of a store.

The tier is authored, never derived (spec: applies is AUTHORED, never derived), so the only correct place to state it is the extracting model, and the instruction text is lifted from mcp_tools/learn.py so both writers ask the same question.

How

Writers of consolidation lessons, enumerated (every write_lesson / Lesson(...) construction reachable from consolidation):

Site Verdict
_save_lessons vector path -> VectorMemoryStore.write_lesson(source="consolidation", ...) fixed: applies=self._lesson_tier(item)
_save_lessons JSONL fallback -> Lesson(...) -> LessonStore.save fixed: applies=self._lesson_tier(item)
VectorMemoryStore.apply_consolidation (member-store / V2 path, builds the lesson value itself) fixed: normalizes applies with the same policy, key absent when unstated
taskrunner.py LLM lesson extraction (source="task_runner") and cli_commands.py direct write_lesson out of scope for #12495 (not consolidation); named in the spec as writers that still state no tier
skills.py:4783 source="consolidation" auto-skill provenance, not a lesson; out of scope
vector_memory.py:3869 source="consolidation" semantic/episodic default, not a lesson; out of scope

Misspelling policy: the write path's normalize_lesson_applies raises by design so a buggy writer is audible. Here the writer is a model, so the raise is caught per item, logged at warning, and the row lands unstated. Dropping it would lose a correction the user actually made over a one-word slip; unstated is the documented safe direction (a demoted standing rule is the costlier mistake). The value is normalized before write_lesson, so its own raising check cannot fire on it.

Tests

test/test_history.py::TestConsolidationLessonApplies:

  • always and on_topic round-trip through the vector store (value_json["applies"]).
  • omitted key -> applies key absent on the vector row (byte-identical to a pre-field row).
  • misspelled value ("Directive") -> warning logged, row lands unstated, clean sibling keeps its tier, the untrusted value never appears in the log.
  • JSONL fallback: both tiers round-trip via LessonStore.load_all(); the unstated row carries no applies key on disk.
  • JSONL fallback: "ALWAYS " canonicalises without a warning; a non-string (7) warns and lands unstated.
  • The prompt the code actually builds (via consolidate_session with _call_llm patched) names "applies": "always|on_topic", "YOU decide it from what the user actually said", and "OMIT the key when you genuinely cannot tell".

Red on base: 6 of 7 fail on e46967b5d (the omitted-key test passes there by construction, as the invariant guard). Mutation: removing the warning log fails both misspelled tests; dropping applies= from the JSONL path fails the JSONL round-trip. Run locally by path with -n0; suites run in CI.

test/test_member_memory_database.py::test_consolidation_persists_the_authored_lesson_tier: on the member-store path both tiers round-trip, an omitted key is absent, a misspelling warns and lands unstated (red on base: KeyError: 'applies').

Static gates run locally on the changed files: black (ratchet), isort, flake8, mypy, comment-history.

Pattern harvest

Rule candidate: when a new authored field is added to a write API, grep every writer of that row type by source= and list each with a verdict in the PR; the highest-volume writer is the one most likely to be missed because it does not call the new surface by name.

Not generalizable: the catch-and-write-unstated policy is specific to a model-authored value where the alternative (drop) loses user data; a human-facing surface should keep the raise.

Other

No product decision remains: the spec fixes that the tier is authored by the extracting model and that an undecided row stays unstated. No existing row is rewritten.

@CrysisDeu
CrysisDeu requested review from a team and patrigao September 22, 2026 17:51
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 22, 2026
@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 10a9b6b9ae1192bab62bda6de2348ba2e0995efc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify one live consolidation run emits applies — tests pin only the prompt text, and a model that ignores it degrades silently to the pre-fix unstated rows.

[DESIGN-REVIEWED] 10a9b6b

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed 10a9b6b9ae1192bab62bda6de2348ba2e0995efc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 10a9b6b

Verdict parsed from the review's SHA-scoped output markers for commit 10a9b6b9ae1192bab62bda6de2348ba2e0995efc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 10a9b6b9ae1192bab62bda6de2348ba2e0995efc: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 10a9b6b9ae1192bab62bda6de2348ba2e0995efc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 10a9b6b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 10a9b6b9ae1192bab62bda6de2348ba2e0995efc: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 10a9b6b9ae1192bab62bda6de2348ba2e0995efc — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify issue #12495's scope covers the member-store path too — the second commit extends the fix there beyond the first commit's two paths.

What this change ships

Inventory (8 items) — 8 justified

Intent: Consolidated lessons should carry the authored startup tier so the two-budget split actually governs the highest-volume lesson writer — a FIX (issue #12495, 6 of 7 new tests red on base).

  1. Consolidation's extraction prompt now asks the model to state each lesson's tier — justified
  2. Consolidated lessons keep their stated tier on the vector-store path — justified
  3. Same tier kept on the JSONL fallback path — justified
  4. Member-store consolidation path (apply_consolidation) keeps the tier — justified
  5. A misspelled tier warns and the lesson still lands unstated, never dropped — justified
  6. learn_add's tier instruction deduplicated into one constant both writers read — justified
  7. New shared helper extracted_lesson_applies (2 counted consumers) — justified
  8. Spec sentence describing the gap rewritten in the same commit — justified

[FIRST-PRINCIPLES-REVIEWED] 10a9b6b

@CrysisDeu
CrysisDeu force-pushed the fix/consolidation-lesson-applies-12495 branch from e3d5d89 to 6e2c860 Compare September 22, 2026 18:15
@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for e3d5d89e2cc266f886354e9586f346b8da7a624e; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt e3d5d89e2cc266f886354e9586f346b8da7a624e: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ⚠️ review incomplete

GPT 5.6 did not produce a complete verdict for 6e2c860e62d1e47b04a520324248b459c7fd4c8d; inspect the workflow logs and re-run it.

This comment is updated in place on each push.

See the GPT 5.6 Review job logs; this commit has no completed GPT verdict.

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6e2c860e62d1e47b04a520324248b459c7fd4c8d: <one-sentence reason>

…2495)

The extraction prompt now asks the model for "applies": "always|on_topic"
per lesson with the same decide-from-what-the-user-said instruction that
learn_add carries, and _save_lessons forwards the normalized tier to both
stores. An omitted or blank tier lands the row unstated; a misspelled one
is logged at warning and still lands unstated rather than dropping the
correction.

Closes #12495
VectorMemoryStore.apply_consolidation builds the lesson value itself and
dropped the model-stated applies tier. It now normalizes it with the same
policy as _save_lessons, via one shared extracted_lesson_applies helper:
absent stays absent, a misspelling is logged and lands unstated. The tier
instruction moves to one shared constant read by both learn_add's schema
and the consolidation prompt, and the spec names the writers that still
state no tier.
@CrysisDeu
CrysisDeu force-pushed the fix/consolidation-lesson-applies-12495 branch from 6e2c860 to 10a9b6b Compare September 22, 2026 19:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 22, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 23, 2026 01:28
auto-merge was automatically disabled September 23, 2026 01:29

Base branch was modified

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after verified readiness: passed, PR Readiness success on this head, mergeable, no open change requests. [operator: bolichen97 batch review 2026-09-23]

@bolichen97
bolichen97 merged commit 83a413b into main Sep 23, 2026
93 checks passed
@bolichen97
bolichen97 deleted the fix/consolidation-lesson-applies-12495 branch September 23, 2026 04:36
wknight1 pushed a commit to wknight1/KiroCrew that referenced this pull request Oct 5, 2026
…#12875)

* fix(memory): history consolidation states the lesson applies tier (kirodotdev#12495)

The extraction prompt now asks the model for "applies": "always|on_topic"
per lesson with the same decide-from-what-the-user-said instruction that
learn_add carries, and _save_lessons forwards the normalized tier to both
stores. An omitted or blank tier lands the row unstated; a misspelled one
is logged at warning and still lands unstated rather than dropping the
correction.

Closes kirodotdev#12495

* fix(memory): member-store consolidation persists the lesson tier too

VectorMemoryStore.apply_consolidation builds the lesson value itself and
dropped the model-stated applies tier. It now normalizes it with the same
policy as _save_lessons, via one shared extracted_lesson_applies helper:
absent stays absent, a misspelling is logged and lands unstated. The tier
instruction moves to one shared constant read by both learn_add's schema
and the consolidation prompt, and the spec names the writers that still
state no tier.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

memory: history consolidation states no startup lesson tier, so its rows all land unstated

3 participants