Skip to content

fix: keep a user's lesson when a consolidation rewording overlaps it - #30833

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/9163-lesson-dedup-authority
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/9163-lesson-dedup-authority

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A lesson the user saved is never deleted by a lower-authority lesson write, such as a consolidation rewording, at any of the three dedup delete sites.

A user's store showed three of their own user_explicit lessons replaced by consolidation rewordings. Each replacement lost a detail the user had written: the named pipelines in one, the mechanism clause in another, the trigger condition in the third. The replacements carry source=consolidation at confidence 0.9, so the user's corrections were demoted with no visible event.

Why it matters

Lessons are how a user corrects the agent. When background consolidation silently rewrites them, the correction the user made is gone, and the agent falls back to a vaguer rule. Nothing in the UI shows that it happened, so the user cannot notice or restore it.

Not a goal

  • Changing what the semantic (cosine) branch does. It already declines a write that a stored near-duplicate outranks.
  • Changing the applies tier rule (tier_permissions), the exact-rule pass, or the substring_covered decline.
  • The JSONL fallback store, whose rows record no source or confidence.
  • Cleaning up lessons already lost on existing installs.
  • Revisiting rfc-memory-hygiene.md, whose sweep holds off retiring lessons until this ordering is settled.

What changed (motivation → approach → change)

The dedup scan in write_lesson has three rules that can delete a stored lesson: substring containment, topic overlap (half the keywords shared), and cosine similarity. Only the cosine rule checked who wrote the stored row. The two text rules run first and are the likelier match for a rewording, so they retired user_explicit lessons on text alone.

The decision this PR records, for the maintainer to approve with the code:

Question from the issue Answer here
Which delete sites All three (substring, topic overlap, cosine)
Which comparison The cosine branch's strict rule, one shared helper: the stored row outranks when it is user_explicit and the write is not, or when its stored confidence is strictly higher
What an outranked text match does Keeps the stored row and stores the submission beside it, as the applies tier rule already does; the cosine branch still declines the write

A user_explicit write is outranked by nothing, so a user correcting their own lesson still replaces it, and equal-authority writes keep newest-wins. The cost: a consolidation correction of a user rule ("use light mode" over a user's "use dark mode") leaves both rules in effect until the user replaces or deletes the old one. The owning spec (memory-skills-hooks.md) records the rule and that cost, and the onboarding_import.py comment and its spec (onboarding-import.md) no longer say write_lesson can delete a lesson the user taught.

Backwards compatibility

Compatible: every write that was accepted is still accepted with the same outcome (inserted); only the set of rows it deletes shrinks.

Removes nothing: the change deletes fewer stored lessons and removes no field, route, key or function.

Tests

  • test_the_authority_pre_pass_skips_rows_the_lexical_branches_claim: a consolidation write that contains a user_explicit rule is stored, reports no supersede, and both rows stay. It pinned the old deletion before.
  • test_a_topic_overlap_write_keeps_an_outranking_row (user_explicit and import at 1.0): a consolidation write sharing half its keywords keeps the higher row, by source and by confidence.
  • test_a_semantic_match_the_pre_pass_skips_keeps_an_outranking_row: the scan's cosine branch keeps a user's on_topic row that the pre-pass left to the scan.
  • Existing tests keep pinning equal-authority newest-wins (test_a_committed_write_still_executes_its_lexical_supersedes) and the cosine branch's decline.
  • Determinism: the four changed node ids 20/20 green at -n0, 10/10 at -n 4 --dist loadgroup --max-worker-restart=2 beside test_lesson_write_outcome.py and test_lesson_write_outcome_supersede.py, and the whole file green in a shuffled order (seed 12183).

Proof: PROVEN test/test_vector_memory_coverage_vector_space.py::TestLessonDedupPaths::test_the_authority_pre_pass_skips_rows_the_lexical_branches_claim, test/test_vector_memory_coverage_vector_space.py::TestLessonDedupPaths::test_a_topic_overlap_write_keeps_an_outranking_row[user_explicit], test/test_vector_memory_coverage_vector_space.py::TestLessonDedupPaths::test_a_topic_overlap_write_keeps_an_outranking_row[import], test/test_vector_memory_coverage_vector_space.py::TestLessonDedupPaths::test_a_semantic_match_the_pre_pass_skips_keeps_an_outranking_row (head 81b7a1d)

Manual verification

N/A — unit coverage sufficient: the change is one store method's delete decision, exercised end to end on a real SQLite store in the tests above.

Related Issues

Closes #9163

Sibling sweep

Swept: git grep -nE "lesson_keywords(|tier_permissions(|delete_semantic(" -- src/
Sibling: src/kiro_crew/vector_memory_runtime/lesson_write.py -- changed here
Sibling: src/kiro_crew/learn.py:256 -- not affected -- a JSONL row records no source or confidence, so there is no authority to compare
Sibling: src/kiro_crew/dashboard/handlers/cron_lesson_supersede.py -- not affected -- the contradiction sweep retires only on_topic findings
Sibling: src/kiro_crew/history_consolidation_memory.py -- not affected -- a consolidation delete is refused unless the row is on_topic

Pattern harvest

Not generalizable: the authority rule existed and was applied at one of three delete sites; the sites now share one gate, so there is no second shape left to catch.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9427adfc5f109a181549c441aea9c3ed8a091aa8 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9427adfc5f109a181549c441aea9c3ed8a091aa8 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9427adf

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9427adfc5f109a181549c441aea9c3ed8a091aa8 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9427adfc5f109a181549c441aea9c3ed8a091aa8 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix applies the cosine branch's existing authority rule to all three delete sites through one shared helper, and the old test pin was recorded as a pending decision, not a settled one.

Notes

  • Two lessons can now stay in effect when they contradict each other, for example a consolidation "use light mode" stored next to a user's "use dark mode". The PR states and accepts this cost. Nothing tells the user both rules are live, so the agent's behaviour can become unclear for as long as both stay. A small follow-up that flags these kept pairs would let the user resolve them.
  • The confidence comparison is strict, so a higher-confidence automated row now blocks a lower-confidence automated rewording on the text rules too. Duplicates can build up among automated lessons, not only beside user_explicit ones.

[DESIGN-REVIEWED] 9427adf

Description read

[DESCRIPTION-READ] 180e0b5863484133d2d50babb92d42402cd65c2151b8763c5777883cfe696c16

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9427adfc5f109a181549c441aea9c3ed8a091aa8 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Fixes a reported defect at its cause. Issue #9163 is linked, and the comment this PR deletes had already marked scan-wide authority ordering as a "tracked follow-up decision".

What this change ships

Inventory (6 items) — 6 justified

Intent: stop background consolidation from silently deleting lessons the user saved (FIX).

  1. A consolidation write that contains a user's lesson is now stored beside that lesson; it no longer replaces it. — justified
  2. A topic-overlap write no longer retires a user_explicit row or a row with higher confidence (such as an import at 1.0). — justified
  3. The scan's cosine branch now keeps a higher-authority on_topic row that the pre-pass leaves to the scan. — justified
  4. When consolidation corrects a user rule, both rules stay in effect until the user replaces or deletes the old one (cost stated in the description). — justified
  5. All three delete sites now use one authority check (_outranks_write, 2 call sites); the cosine rule's own copy of the check is removed. — justified
  6. The onboarding-import comment and its spec stop saying write_lesson can delete a lesson the user taught. — justified

[FIRST-PRINCIPLES-REVIEWED] 9427adf

Description read

[DESCRIPTION-READ] f7fc7b183ff31ddcd1d114ad5afc63863a5318a0d4861378ab99717b28bf3104

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@bolichen97
bolichen97 enabled auto-merge (squash) November 5, 2026 23:13
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 5, 2026
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from b95424c to 153f595 Compare November 6, 2026 01:20
@bolichen97

bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/memory-skills-hooks.md (and 27 same-directory files) since this head went green (bf9b611 -> 821f8d5); CI re-running. The patch is unchanged; the earlier red lanes were API rate-limit failures, not findings.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 6, 2026
@github-actions

github-actions Bot commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9427adfc5f109a181549c441aea9c3ed8a091aa8 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9427adf

Verdict parsed from the review's SHA-scoped output markers for commit 9427adfc5f109a181549c441aea9c3ed8a091aa8.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9427adfc5f109a181549c441aea9c3ed8a091aa8 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from 153f595 to 630e47e Compare November 6, 2026 03:18
@bolichen97

bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/ (same-directory specs) and src/kiro_crew/webhooks.py (same directory as onboarding_import.py) since this head went green (821f8d5 -> 1120304); CI re-running. The patch is unchanged. The last round's only red was a test worker that wedged in an unrelated store-open test, logged in #31024.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from 630e47e to 81b7a1d Compare November 6, 2026 06:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 6, 2026
@bolichen97

bolichen97 commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_dashboard_server_composition_contract.py (the lazy-import count pin that failed Backend Tests shard 6 on Linux and Windows here was red on main and is fixed there by #30984) and same-directory specs since this head's CI ran (1120304 -> 71f6de0); patch unchanged, CI re-running.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 6, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from 81b7a1d to be92931 Compare November 6, 2026 18:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

The red Backend Tests (3.12, 2) / Backend Tests (Windows) (2) (and the Coverage Gate after it) is main's, not this PR's: test/test_security_gate_liveness.py::test_the_package_stays_within_its_line_budget fails on main at 34,753 lines against a budget of 34,733 since #30774. #31625 raises the budget; this branch will be rebased onto it once it lands.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from be92931 to 1bee9da Compare November 6, 2026 20:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 6, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

The red Backend Tests (3.12, 7) / Backend Tests (Windows) (7) is main's, not this PR's: test/test_pull_request_target_allowlist.py::test_only_the_allowed_workflows_use_pull_request_target fails on main since #31396 moved sensitive-change-review.yml to pull_request_target. #31803 adds it to the allowlist; this branch will be rebased onto it once it lands.

The substring and topic-overlap dedup branches of write_lesson retired
a stored lesson on text alone, so a consolidation rewording that shared
half its keywords with a user_explicit lesson replaced it and dropped
the user's specifics. Only the semantic branch weighed authority.

Every dedup delete site now applies the same authority rule the
semantic branch uses (user_explicit over another source, or strictly
higher stored confidence): an outranking row is kept and the
submission is stored beside it. A user_explicit write is unaffected.

Closes #9163
@bolichen97
bolichen97 force-pushed the fix/9163-lesson-dedup-authority branch from 1bee9da to 9427adf Compare November 6, 2026 21:47
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 9427adf, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit f54bb35 into main Nov 7, 2026
152 of 159 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/9163-lesson-dedup-authority branch November 7, 2026 00:46
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 7, 2026
buluoray added a commit that referenced this pull request Nov 8, 2026
…30833)

The substring and topic-overlap dedup branches of write_lesson retired
a stored lesson on text alone, so a consolidation rewording that shared
half its keywords with a user_explicit lesson replaced it and dropped
the user's specifics. Only the semantic branch weighed authority.

Every dedup delete site now applies the same authority rule the
semantic branch uses (user_explicit over another source, or strictly
higher stored confidence): an outranking row is kept and the
submission is stored beside it. A user_explicit write is unaffected.

Closes #9163
(cherry picked from commit f54bb35)

Originally authored on main by Bolin Chen (bolichen97); authored here with the public identity because the commit guard refuses the original address.
buluoray added a commit that referenced this pull request Nov 8, 2026
…30833)

The substring and topic-overlap dedup branches of write_lesson retired
a stored lesson on text alone, so a consolidation rewording that shared
half its keywords with a user_explicit lesson replaced it and dropped
the user's specifics. Only the semantic branch weighed authority.

Every dedup delete site now applies the same authority rule the
semantic branch uses (user_explicit over another source, or strictly
higher stored confidence): an outranking row is kept and the
submission is stored beside it. A user_explicit write is unaffected.

Closes #9163
(cherry picked from commit f54bb35)

Originally authored on main by Bolin Chen (bolichen97); authored here with the public identity because the commit guard refuses the original address.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decide: authority ordering at all three lesson-dedup delete sites (follow-up to #6175)

2 participants