Repository navigation
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change adds one nullable column, and rows without it (NULL) release their claim by their own hash exactly as before; no user-visible surface changes. Notes
[DESIGN-REVIEWED] d88028c Description read[DESCRIPTION-READ] 216c403cee977e906b0cc99a38505d881267a95ed655375da3ae097bb34c7330 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS This fixes the cause: the row now records the hash its claim moved to. One rider changes exact collapses, which the Goal doesn't cover. Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: when a file the knowledge base fuzzy-merged into another document is deleted or edited, its claim goes too, and no source keeps searchable text it never had. FIX (#21954).
[FIRST-PRINCIPLES-REVIEWED] d88028c Description read[DESCRIPTION-READ] 0bc69ea0850e14fb721a3a1ce90d86bfc6ae34dc384e749ceb7744f72b49ce6a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — 🔴 changes requested (blocking)Reviewed Blocking: deleting a fuzzy winner's source removes the claim of a holder that the pre-ingest gate attached without a marker, and never revives that holder. Its file stays on disk while the Library stops tracking it. BLOCKING — src/kiro_crew/knowledge/store_parts/item_rows.py:548 Trigger: The pre-ingest gate attaches folder file T to loser L's items. [BLOCK-MERGE] d88028c Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — 🔴 changes requested (blocking)GPT 6.1 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- Dropping a fuzzy artifact claim also destroys a surviving exact HTML claim
BLOCKING -- src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- A row claiming different content incorrectly qualifies its source as an heir
[BLOCK-MERGE] d88028c Adjudication (Opus 5.5) — is blocking on each finding proportionate?F1 — unbounded (fenced). F2 — unbounded (fenced). The genuine-holder check at False positive or not applicable? A repository writer can comment: |
cb81138 to
71d0626
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 findings on
|
71d0626 to
230bc55
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on
The GPT 6.1 red on |
230bc55 to
89b4073
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on
mechanism: the cascade's revive now detaches only when Reproduced before the fix: an agent document and a folder file exact-lost to one winner; deleting the winner handed the item to the folder and dropped the agent's location. Test Spec row in |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on
mechanism: the cascade's revive detaches only when Test |
89b4073 to
54af2b6
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on
mechanism: Test |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on
mechanism: Test Not changed, stated in the PR body under Not a goal: every ingest path releases a stale claim before the ingest, as main does for changed content. So a FUZZY claim on an unchanged re-add that the budget refuses is still released early. The next re-add retries and ingests the document's own text. Moving every release into each path's success hop is the full fix; it touches all three ingest paths. |
54af2b6 to
4ec06b1
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on
mechanism: the root cause was one inference -- exact vs fuzzy was guessed by comparing a row's own hash with the claim hash, and for an artifact those are in two domains (raw text vs extracted prose). The guess is gone. Tests:
Spec row updated in the same commit. Knowledge and store_parts suites: 1894 passed. |
…1954) A fuzzy collapse merges two documents whose text differs. It moved the loser's location claims onto the winner's items but left the loser's row with only its own hash, which names no item afterwards. Every later release and adoption resolves the claim by hash, so deleting or editing the loser's file released nothing, and deleting the winner then handed the loser's source searchable items no state row names. A fuzzy collapse now writes the winner's items.content_hash to a new nullable claimed_hash column on the loser's row in all three state tables, and on the rows of other holders whose claim moved with it. Folder deletion and re-read, include retirement, artifact and agent removal and re-ingest, and bundle import release claimed_hash when set. The claimant count reads COALESCE(claimed_hash, own hash). A fuzzy loser never inherits the winner's text: when the winner's copy goes, locations held only through claimed_hash are dropped before an heir is chosen, and the loser returns to pending, as on main. Adoption matches only a row's own hash. A chained holder's deferral marker moves to the winner with its claim, except in the winner's own source. An exact collapse writes no claimed_hash, so no reader guesses exact vs fuzzy by comparing an artifact's raw-text hash with an item hash, and never erases an existing one. The cascade's revive clears it, first releasing the claim when the claimed copy went to another holder. Rows written before the column read NULL and behave exactly as before. Closes #21954
4ec06b1 to
d88028c
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on
mechanism: owner ruling. A fuzzy loser is never handed the winner's text; it is rebuilt from its own bytes, as on main. Tests:
Push-driven losers keep main's behaviour ( |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on
mechanism: both loser-row writes in Test |
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Parked. This PR stays a draft at Claim-ownership findings, in order. There are 12 distinct ones. Rounds 1 and 7 each raised more than one, and round 3's GPT and Opus findings were the same bug.
The lane comments for 10 to 12 are updated in place. Their content is current as of head Root cause. Claim ownership is spread over three state tables ( Next step. Design one owner record that every path reads and writes, before any more code. The record should say which row holds which content, and by which claim. Also red on this head, caused by this diff: the Comment History Gate, for the wording "no longer" at |
…ing filenames
## Why is the change being made
On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.
## What has changed
`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.
## Why was this solution chosen?
`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.
## What are the component relationships
```
folder_watcher.py --stores backslash file_path--> folder_file_state rows
|
v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
|
v
filename_near_match (Tier-2 fuzzy gate)
```
Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.
## API changes
None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.
## How was this change tested
`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.
## Backward compatibility
Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.
## Additional I/O and performance impact
None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
…ing filenames
## Why is the change being made
On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.
## What has changed
`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.
## Why was this solution chosen?
`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.
## What are the component relationships
```
folder_watcher.py --stores backslash file_path--> folder_file_state rows
|
v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
|
v
filename_near_match (Tier-2 fuzzy gate)
```
Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.
## API changes
None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.
## How was this change tested
`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.
## Backward compatibility
Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.
## Additional I/O and performance impact
None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
…ing filenames (#24834) ## Why is the change being made On a Windows `local_folder` knowledge source, `folder_watcher` stores each row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`). `knowledge/dedup.py` derives the compare/display filename with `rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole folder prefix survives into the normalized stem. Two genuine copies of one note in differently-named folders then normalise to `work\meeting notes` vs `archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9 floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently fails to collapse them. POSIX collapses the same two notes correctly. The result is a silent correctness gap: the knowledge base keeps duplicate copies of the same document on Windows only. ## What has changed `normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the basename, so the folder prefix is stripped on Windows exactly as on POSIX. The change is two one-line edits plus comments; no imports added. A new test `test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the absolute-path case (red before the fix) with a POSIX control and a `_file_ext` control. ## Why was this solution chosen? `str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps the diff to the two functions the defect lives in. `ntpath.basename` was the alternative but would add an import next to the hunk open draft PR #22859 edits, and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both. ## What are the component relationships ``` folder_watcher.py --stores backslash file_path--> folder_file_state rows | v dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract) | v filename_near_match (Tier-2 fuzzy gate) ``` Sequence: a Windows `local_folder` walk persists `Work\notes.md` and `Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate collapses the duplicate. ## API changes None. `normalize_filename` and `_file_ext` keep the same signature and, for any input without a backslash (every POSIX path and URL/vault-relative string), the same output. ## How was this change tested `test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and absolute-path cases fail without the fix and pass with it; the full existing `test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo venv with an isolated temp `HOME`/`KIROCREW_HOME`. ## Backward compatibility Compatible: output is unchanged for every input that contains no backslash, so all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave exactly as before; only a backslash OS path changes, and only to the basename it should already have reduced to. ## Additional I/O and performance impact None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
Problem / Motivation
Goal: Deleting or editing a file that the knowledge base fuzzy-merged into another document releases its claim, so no source is left holding searchable text it never had.
The fuzzy dedup tier collapses two near-identical documents whose text differs. It moves the loser's location claims onto the winner's items and marks the loser's row
deduped, but the row keeps only its own hash, which names no item afterwards. Deleting the loser's file (or editing it) then releases nothing. When the winner's source is later deleted, its items are handed to the loser's source, and no state row there names them: the text stays searchable with no file behind it and nothing that can delete it.Why it matters
Any folder file that a fuzzy collapse merged and that is later deleted or edited hits this. The leftover items show up in search, are cited as coming from a source that never held that text, and cannot be removed through that source's normal delete path.
Not a goal
deduped, marker cleared, its own text not stored again until it is re-added (artifacts also on gateway-start reconcile or a manual Sync). No scanner is added here.What changed (motivation → approach → change)
Symptom: a fuzzy-merged file's claim survives its deletion or edit, and the winner's deletion leaves untracked items. Root cause: every release (
_handle_deleted,release_stale_claim, artifact and agent removal) and adoption (_adopt_reassigned_item) resolves the claim by the row's own hash, and after a fuzzy collapse that hash names no item. Change: record the hash the claim moved to on a fuzzy collapse, and use it in each release path. Owner ruling: a fuzzy loser is never handed the winner's text; when the winner goes it is rebuilt from its own bytes, as on main.store_parts/schema_migrations.pyclaimed_hashcolumn onfolder_file_state,artifact_item_state,agent_item_state; existing rows read NULLdedup.py_fuzzy_claim_hash: loser and winner item hashes differ),_collapse_docwrites the winner'sitems.content_hashto the loser's row; an exact collapse writes noclaimed_hash, so no reader has to guess exact vs fuzzy by comparing hashes from two domains (an artifact row hashes its raw text, its items the extracted prose). Likewise and_move_claims_to_winnerre-points the rows of other holders that owned nothing and claimed the loser's hash (a chained collapse): theirclaimed_hash(fuzzy only), and theirmerged_into_source_idwhen it named the loser's source, on an exact collapse too. Rows in the winner's own source are skipped, so they keep deferring to the loser's source and are revived when it is deletedstore_parts/item_ownership.py_CLAIM_HASH_COL=COALESCE(NULLIF(claimed_hash, ''), <own hash>), used by the claimant count indetach_source_location_by_hash;release_stale_claim(..., prev_claimed_hash=None)releases a recorded claim even if the bytes did not change, because the re-ingest rewrites the row and drops the column; an exact claim has none and follows the own-hash rule. New_drop_fuzzy_claimants: when a winner's copy goes, every location held only through aclaimed_hashis dropped BEFORE an heir is chosen, so a fuzzy loser never inherits different text. New_revive_fuzzy_claimants: those rows return topending(folder) or lose their marker (push-driven), as on main._adopt_reassigned_itemmatches only a row's own hash, as on main. Newclaimed_hash_ofreaderfolder_watcher.pyclaimed_hashfirst, then the existing text/bytes hash fallbackartifact_ingest.py,agent_source.pyclaimed_hashwhen setstore_parts/item_rows.pyclaimed_hashwith the marker. It first releases the claim when the claimed copy went to another holder, An exact claim has noclaimed_hash, so its location stays and the own-hash fallback keeps addressing it. When the copy went into the row's own source, the location stays, so the source never detaches from an item it now owns.store_parts/bundle_ownership.pydocs/system-specs/modules/knowledge.mdThe winner's item ids are still never written into
item_ids, for the reason the existing docstrings give.Backwards compatibility
Compatible: one nullable column is added; a row without it (every row written before this change, and every row a collapse never touched) resolves its claim by its own hash exactly as on main.
Removes nothing: the change adds a column and reads it; no path, row, key, validator or function is removed or narrowed.
Tests
New
test/test_knowledge_fuzzy_claim_hash.py(18 tests). Each guard was reverted on its own and the named test went red (16 of 16 mutations):test_deleting_the_loser_file_then_the_winner_leaves_nothing_untracked: the reporter's case through real folder scans and a realdedup_sweep. Red onorigin/main(the claim is not released).test_collapse_records_the_winner_hash_on_the_loser_rowtest_rereading_the_loser_file_releases_the_moved_claim[content|touch]: the edit path, with changed bytes and with a touch only.test_deleting_the_winner_rebuilds_the_loser_from_its_own_bytes: deleting the winner's source destroys the winner's text; the loser goes topendingand the next scan stores its own text. Red if the fuzzy claimant is not dropped before the heir is chosen, alone or with adoption via the claim hash restored.test_deleting_the_winner_file_rebuilds_the_loser: the same through the item path (the winner's file deleted). Red without the drop and, separately, without the revive.test_an_exact_recollapse_keeps_an_earlier_fuzzy_claim: an exact collapse keeps an existingclaimed_hash. Red withoutCOALESCE. Hand-built row: with adoption gone the code no longer produces it, so this pins the write.test_a_claim_two_rows_share_survives_one_deletion: the claimant count reads the claim hash.test_a_chained_holder_claim_moves_with_the_collapsetest_removing_a_fuzzy_merged_aggregate_document_releases_its_claim[agent|artifact]test_an_aggregate_reingest_releases_the_moved_claim[agent|artifact]test_a_revived_row_drops_the_claim_hash_it_no_longer_holdstest_every_chunk_of_a_multi_chunk_winner_leaves_its_losertest_no_fuzzy_loser_inherits_the_winnertest_an_unchanged_reingest_keeps_an_exact_claim[agent|artifact]: an unchanged re-add keeps an exact claim, so an ingest refused on budget loses nothing. Red if an exact collapse records a claim hash.test_an_exact_html_artifact_claim_survives_revive: an artifact row whose own hash (raw HTML) differs from its items' (prose) exact-loses to a folder copy also held by another folder; deleting the winner then that folder hands the document to the artifact. Red if an exact collapse records a claim hash, including with the round-4 own-hash guard restored.test_a_winner_source_row_keeps_deferring_to_the_loser: a row in the winner's source that deferred to the loser is still revived when the loser's source is deleted. Red without the winner-source skip.test_an_exact_claim_survives_the_heir_being_another_holder: an agent document and a folder file exact-lost to one winner; deleting the winner then the folder keeps the agent's copy. Red if an exact collapse records a claim hash.test_a_chained_deferral_follows_the_claim_to_the_new_winner[fuzzy|exact]test_a_legacy_row_without_a_claimed_hash_keeps_todays_releasetest_a_legacy_fuzzy_row_still_releases_nothing: pins the documented residual for pre-change rows.Migration:
test/test_store_parts_owner_paths.pynow expectsclaimed_hash= NULL on a legacy library.test/test_store_parts_bindings.pyinventory updated forclaimed_hash_ofand the newrelease_stale_claimsignature.Run locally by path with
-n0: the new file plustest_knowledge_dedup,test_folder_watcher,test_knowledge_asciidoc_include,test_knowledge_artifact_ingest,test_knowledge_agent_source,test_knowledge_agent_sync_off_loop,test_knowledge_bundle_ownership,test_knowledge_bundle_ownership_withholding,test_knowledge_delete_off_loop,test_folder_watcher_off_loop_guard,test_agent_source_residue_sweep,test_store_parts_*: all pass. isort, flake8 andmypy src/kiro_crew/knowledge/are clean.The include-retirement and bundle-import release sites have no dedicated mutation test; they follow the same rule as the deletion path and are covered by their existing suites only.
Manual verification
N/A — the store and scanner paths are exercised end to end by the tests above against a real SQLite store. Linux only; nothing here is platform-specific.
Related Issues
Closes #21954. Follows #21778.
Pattern harvest
Not generalizable: the defect is one missing piece of bookkeeping in one collapse path; the claim hash is now read through one shared expression (
_CLAIM_HASH_COL) by the claimant count, bundle import and include retirement. The crash-marker sweep's shield inmaintenance_sweeps.pystill reads_OWNERSHIP_HASH_COL; it covers only healthy rows and is left as on main.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Decision for the owner
Whether a fuzzy collapse should record the hash its claim moved to, so later deletes and edits can release it. This draft adds a nullable
claimed_hashcolumn, written only on a fuzzy collapse, that every release reads first; a fuzzy loser is never handed the winner's text (owner ruling), and leaves pre-change rows on today's behaviour rather than guessing a backfill.Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.