Skip to content

fix(knowledge): record the hash a dedup collapse moves a claim to (#21954) - #22859

Draft
buluoray wants to merge 1 commit into
mainfrom
fix/knowledge-fuzzy-dedup-claimed-hash-21954
Draft

buluoray wants to merge 1 commit into
mainfrom
fix/knowledge-fuzzy-dedup-claimed-hash-21954

Conversation

@buluoray

@buluoray buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Deleting or editing a file that the knowledge base fuzzy-merged into another document releases its claim, so no source is left holding searchable text it never had.

The fuzzy dedup tier collapses two near-identical documents whose text differs. It moves the loser's location claims onto the winner's items and marks the loser's row deduped, but the row keeps only its own hash, which names no item afterwards. Deleting the loser's file (or editing it) then releases nothing. When the winner's source is later deleted, its items are handed to the loser's source, and no state row there names them: the text stays searchable with no file behind it and nothing that can delete it.

Why it matters

Any folder file that a fuzzy collapse merged and that is later deleted or edited hits this. The leftover items show up in search, are cited as coming from a source that never held that text, and cannot be removed through that source's normal delete path.

Not a goal

  • Repairing rows written by a fuzzy collapse before this change. They have no record of the winner's hash, so they keep today's behaviour; nothing is guessed or backfilled.
  • Changing which pairs the fuzzy tier merges, or the AsciiDoc includer exclusion from feat(knowledge): expand local AsciiDoc include:: directives at ingest #21778.
  • The pre-existing transformed-file adoption gap documented in the knowledge spec (pre-ingest gate rows for PDF/DOCX/HTML).
  • Follow-up: push-driven losers. An agent or artifact document that fuzzy-lost keeps main's behaviour when its winner goes: deduped, marker cleared, its own text not stored again until it is re-added (artifacts also on gateway-start reconcile or a manual Sync). No scanner is added here.
  • Releasing claims inside each path's success hop. All three ingest paths release a stale claim before the ingest, as on main for changed content. So a fuzzy claim on an unchanged re-add that the budget refuses is released early; the next re-add, which retries, ingests the document's own text. An exact claim is not affected.

What changed (motivation → approach → change)

Symptom: a fuzzy-merged file's claim survives its deletion or edit, and the winner's deletion leaves untracked items. Root cause: every release (_handle_deleted, release_stale_claim, artifact and agent removal) and adoption (_adopt_reassigned_item) resolves the claim by the row's own hash, and after a fuzzy collapse that hash names no item. Change: record the hash the claim moved to on a fuzzy collapse, and use it in each release path. Owner ruling: a fuzzy loser is never handed the winner's text; when the winner goes it is rebuilt from its own bytes, as on main.

Part Change
store_parts/schema_migrations.py new nullable claimed_hash column on folder_file_state, artifact_item_state, agent_item_state; existing rows read NULL
dedup.py on a FUZZY collapse only (_fuzzy_claim_hash: loser and winner item hashes differ), _collapse_doc writes the winner's items.content_hash to the loser's row; an exact collapse writes no claimed_hash, so no reader has to guess exact vs fuzzy by comparing hashes from two domains (an artifact row hashes its raw text, its items the extracted prose). Likewise and _move_claims_to_winner re-points the rows of other holders that owned nothing and claimed the loser's hash (a chained collapse): their claimed_hash (fuzzy only), and their merged_into_source_id when it named the loser's source, on an exact collapse too. Rows in the winner's own source are skipped, so they keep deferring to the loser's source and are revived when it is deleted
store_parts/item_ownership.py _CLAIM_HASH_COL = COALESCE(NULLIF(claimed_hash, ''), <own hash>), used by the claimant count in detach_source_location_by_hash; release_stale_claim(..., prev_claimed_hash=None) releases a recorded claim even if the bytes did not change, because the re-ingest rewrites the row and drops the column; an exact claim has none and follows the own-hash rule. New _drop_fuzzy_claimants: when a winner's copy goes, every location held only through a claimed_hash is dropped BEFORE an heir is chosen, so a fuzzy loser never inherits different text. New _revive_fuzzy_claimants: those rows return to pending (folder) or lose their marker (push-driven), as on main. _adopt_reassigned_item matches only a row's own hash, as on main. New claimed_hash_of reader
folder_watcher.py deletion, re-read and include retirement release claimed_hash first, then the existing text/bytes hash fallback
artifact_ingest.py, agent_source.py removal and re-ingest release claimed_hash when set
store_parts/item_rows.py the cascade's revive clears claimed_hash with the marker. It first releases the claim when the claimed copy went to another holder, An exact claim has no claimed_hash, so its location stays and the own-hash fallback keeps addressing it. When the copy went into the row's own source, the location stays, so the source never detaches from an item it now owns.
store_parts/bundle_ownership.py bundle import releases a replaced marker's claim by its claim hash
docs/system-specs/modules/knowledge.md documents the column and the legacy-NULL rule

The winner's item ids are still never written into item_ids, for the reason the existing docstrings give.

Backwards compatibility

Compatible: one nullable column is added; a row without it (every row written before this change, and every row a collapse never touched) resolves its claim by its own hash exactly as on main.

Removes nothing: the change adds a column and reads it; no path, row, key, validator or function is removed or narrowed.

Tests

New test/test_knowledge_fuzzy_claim_hash.py (18 tests). Each guard was reverted on its own and the named test went red (16 of 16 mutations):

  • test_deleting_the_loser_file_then_the_winner_leaves_nothing_untracked: the reporter's case through real folder scans and a real dedup_sweep. Red on origin/main (the claim is not released).
  • test_collapse_records_the_winner_hash_on_the_loser_row
  • test_rereading_the_loser_file_releases_the_moved_claim[content|touch]: the edit path, with changed bytes and with a touch only.
  • test_deleting_the_winner_rebuilds_the_loser_from_its_own_bytes: deleting the winner's source destroys the winner's text; the loser goes to pending and the next scan stores its own text. Red if the fuzzy claimant is not dropped before the heir is chosen, alone or with adoption via the claim hash restored.
  • test_deleting_the_winner_file_rebuilds_the_loser: the same through the item path (the winner's file deleted). Red without the drop and, separately, without the revive.
  • test_an_exact_recollapse_keeps_an_earlier_fuzzy_claim: an exact collapse keeps an existing claimed_hash. Red without COALESCE. Hand-built row: with adoption gone the code no longer produces it, so this pins the write.
  • test_a_claim_two_rows_share_survives_one_deletion: the claimant count reads the claim hash.
  • test_a_chained_holder_claim_moves_with_the_collapse
  • test_removing_a_fuzzy_merged_aggregate_document_releases_its_claim[agent|artifact]
  • test_an_aggregate_reingest_releases_the_moved_claim[agent|artifact]
  • test_a_revived_row_drops_the_claim_hash_it_no_longer_holds
  • test_every_chunk_of_a_multi_chunk_winner_leaves_its_loser
  • test_no_fuzzy_loser_inherits_the_winner
  • test_an_unchanged_reingest_keeps_an_exact_claim[agent|artifact]: an unchanged re-add keeps an exact claim, so an ingest refused on budget loses nothing. Red if an exact collapse records a claim hash.
  • test_an_exact_html_artifact_claim_survives_revive: an artifact row whose own hash (raw HTML) differs from its items' (prose) exact-loses to a folder copy also held by another folder; deleting the winner then that folder hands the document to the artifact. Red if an exact collapse records a claim hash, including with the round-4 own-hash guard restored.
  • test_a_winner_source_row_keeps_deferring_to_the_loser: a row in the winner's source that deferred to the loser is still revived when the loser's source is deleted. Red without the winner-source skip.
  • test_an_exact_claim_survives_the_heir_being_another_holder: an agent document and a folder file exact-lost to one winner; deleting the winner then the folder keeps the agent's copy. Red if an exact collapse records a claim hash.
  • test_a_chained_deferral_follows_the_claim_to_the_new_winner[fuzzy|exact]
  • test_a_legacy_row_without_a_claimed_hash_keeps_todays_release
  • test_a_legacy_fuzzy_row_still_releases_nothing: pins the documented residual for pre-change rows.

Migration: test/test_store_parts_owner_paths.py now expects claimed_hash = NULL on a legacy library. test/test_store_parts_bindings.py inventory updated for claimed_hash_of and the new release_stale_claim signature.

Run locally by path with -n0: the new file plus test_knowledge_dedup, test_folder_watcher, test_knowledge_asciidoc_include, test_knowledge_artifact_ingest, test_knowledge_agent_source, test_knowledge_agent_sync_off_loop, test_knowledge_bundle_ownership, test_knowledge_bundle_ownership_withholding, test_knowledge_delete_off_loop, test_folder_watcher_off_loop_guard, test_agent_source_residue_sweep, test_store_parts_*: all pass. isort, flake8 and mypy src/kiro_crew/knowledge/ are clean.

The include-retirement and bundle-import release sites have no dedicated mutation test; they follow the same rule as the deletion path and are covered by their existing suites only.

Manual verification

N/A — the store and scanner paths are exercised end to end by the tests above against a real SQLite store. Linux only; nothing here is platform-specific.

Related Issues

Closes #21954. Follows #21778.

Pattern harvest

Not generalizable: the defect is one missing piece of bookkeeping in one collapse path; the claim hash is now read through one shared expression (_CLAIM_HASH_COL) by the claimant count, bundle import and include retirement. The crash-marker sweep's shield in maintenance_sweeps.py still reads _OWNERSHIP_HASH_COL; it covers only healthy rows and is left as on main.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Decision for the owner

Whether a fuzzy collapse should record the hash its claim moved to, so later deletes and edits can release it. This draft adds a nullable claimed_hash column, written only on a fuzzy collapse, that every release reads first; a fuzzy loser is never handed the winner's text (owner ruling), and leaves pre-change rows on today's behaviour rather than guessing a backfill.

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision d88028c2c8c87b4eba725ac3fcb69955a13ca23b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of d88028c2c8c87b4eba725ac3fcb69955a13ca23b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change adds one nullable column, and rows without it (NULL) release their claim by their own hash exactly as before; no user-visible surface changes.

Notes

  • Every reader (seven release and adoption paths) now has to remember COALESCE(claimed_hash, own). _CLAIM_HASH_COL centralises that rule, so any new release path should use it rather than read the row's own hash.

[DESIGN-REVIEWED] d88028c

Description read

[DESCRIPTION-READ] 216c403cee977e906b0cc99a38505d881267a95ed655375da3ae097bb34c7330

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of d88028c2c8c87b4eba725ac3fcb69955a13ca23b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

This fixes the cause: the row now records the hash its claim moved to. One rider changes exact collapses, which the Goal doesn't cover.

Notes

  • Item 5 — rides along, oversized: the Goal is about fuzzy merges, but _move_claims_to_winner also re-points merged_into_source_id "on an exact collapse too". That changes exact-collapse behaviour, and no linked defect or test in the listed set is red on base for it. Subtraction: drop the merged_into_source_id = CASE … END clause from _move_claims_to_winner, so it only moves claimed_hash on a fuzzy collapse. Clears when: the mechanism is removed, or a human amends the Goal.

What this change ships

Inventory (7 items) — 6 justified

Intent: when a file the knowledge base fuzzy-merged into another document is deleted or edited, its claim goes too, and no source keeps searchable text it never had. FIX (#21954).

  1. Deleting a fuzzy-merged loser file releases its claim on the winner's text — justified
  2. Editing or touching a fuzzy-merged loser file releases the moved claim — justified
  3. Deleting the winner rebuilds the loser from its own bytes instead of handing it the winner's text — justified
  4. Removing or re-ingesting a fuzzy-merged agent or artifact document releases its claim — justified
  5. On any collapse, other holders that deferred to the loser's source now defer to the winner's source, exact collapses included — rides along, oversized
  6. A new nullable claimed_hash column on three state tables, read by every release path, the claimant count and bundle import — justified
  7. An unchanged re-add with a fuzzy claim releases that claim early (declared under Not a goal) — justified

[FIRST-PRINCIPLES-REVIEWED] d88028c

Description read

[DESCRIPTION-READ] 0bc69ea0850e14fb721a3a1ce90d86bfc6ae34dc384e749ceb7744f72b49ce6a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — 🔴 changes requested (blocking)

Reviewed d88028c2c8c87b4eba725ac3fcb69955a13ca23b — this comment is updated in place on each push.

Blocking: deleting a fuzzy winner's source removes the claim of a holder that the pre-ingest gate attached without a marker, and never revives that holder. Its file stays on disk while the Library stops tracking it.

BLOCKING — src/kiro_crew/knowledge/store_parts/item_rows.py:548

self._drop_fuzzy_claimants(item_id, source_id)

Trigger: The pre-ingest gate attaches folder file T to loser L's items. _record_deduped_state writes T's row as deduped with merged_into_source_id NULL. A sweep then fuzzy-collapses L into winner W, and _move_claims_to_winner sets T's claimed_hash to W's hash; the CASE leaves the marker NULL. The user then deletes W's source. In delete_source_cascade, _drop_fuzzy_claimants deletes T's location, because T's own text_hash is not W's hash. The deferred loop only selects rows WHERE merged_into_source_id = ?, so it skips T, and the cascade never calls _revive_fuzzy_claimants.
Outcome: T's row stays deduped, with no items, no location, and a claimed_hash naming deleted items. The mtime gate at folder_watcher.py:552 skips it, so it never re-ingests. Only L's rebuilt copy holds T's text, and T is not a location on it. Deleting L later removes the content from the Library while T's file is still on disk.
Anchor: residual/crash-data-loss-corruption
Fix: Collect the hashes _drop_fuzzy_claimants returns in the cascade loop, then call self._revive_fuzzy_claimants(hashes, source_id) after the deferred loop, as delete_items_batch_in_txn does.

[BLOCK-MERGE] d88028c
[OPUS-REVIEWED] d88028c

Verdict parsed from the review's SHA-scoped output markers for commit d88028c2c8c87b4eba725ac3fcb69955a13ca23b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d88028c2c8c87b4eba725ac3fcb69955a13ca23b: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging d88028c2c8c87b4eba725ac3fcb69955a13ca23b. 2 of 2 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- Dropping a fuzzy artifact claim also destroys a surviving exact HTML claim

f"AND {_OWNERSHIP_HASH_COL[table]} = ? LIMIT 1",
Trigger: An HTML artifact exact-claims a folder document through the duplicate gate, then another artifact in the same aggregate fuzzy-collapses onto that document. Deleting the folder calls delete_source_cascade → _drop_fuzzy_claimants; this query compares the HTML row’s raw hash against the item’s extracted-text hash.
Outcome: The aggregate is incorrectly treated as having only fuzzy claims. Its shared location and the last indexed copy are deleted, making the surviving HTML artifact’s text disappear from search.
Anchor: residual/crash-data-loss-corruption
Fix: Preserve an item-domain hash for transformed artifacts’ exact claims and consult it before removing the shared location.

BLOCKING -- src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- A row claiming different content incorrectly qualifies its source as an heir

f"AND {_OWNERSHIP_HASH_COL[table]} = ? LIMIT 1",
Trigger: Agent document X (notes.md, hash P) fuzzy-loses to folder Q. Later, a folder stores P as guide.md, and agent document Y (guide.md, hash R) fuzzy-loses to it. Deleting P’s folder reaches this query, which accepts X’s original P hash despite its current Q claim; _adopt_reassigned_item assigns P to X. Removing X, then deleting Q’s folder, completes the failure.
Outcome: X’s removal deletes its adopted P items without releasing Q. Q is subsequently reassigned to the agent aggregate with no state row naming it, leaving searchable content that normal document removal cannot delete.
Anchor: residual/crash-data-loss-corruption
Fix: Exclude empty-group rows whose nonempty claimed_hash names different content from the genuine-holder check.

[BLOCK-MERGE] d88028c
[GPT-REVIEWED] d88028c

Adjudication (Opus 5.5) — is blocking on each finding proportionate?

F1 — unbounded (fenced). _drop_fuzzy_claimants deletes a claimant's location unless one of its rows' own hash equals the item's content_hash (item_ownership.py:158-170). delete_source_cascade calls it on every owned item before an heir is chosen (item_rows.py:280-289). An HTML artifact's raw-byte hash that differs from the extracted-text hash fails that check. The trigger is an ordinary folder delete with plausible inputs. Nothing in this run showed a re-index path for the artifact, so I could not complete the FLAG record.

F2 — unbounded (fenced). The genuine-holder check at item_ownership.py:161-163 matches on the row's own hash and does not care whether that row's non-empty claimed_hash points to different content. _adopt_reassigned_item then matches on the same column (item_ownership.py:236-242). The failure needs several steps, but none of them contradict each other or need input the writer can't produce. It ends in searchable content that no state row names, and I found no recovery path in this run.

[ADJUDICATION] d88028c2c8c87b4eba725ac3fcb69955a13ca23b total=0 uphold=0 downgrade=0
[GPT-ADJUDICATED] d88028c2c8c87b4eba725ac3fcb69955a13ca23b

[ADJUDICATION-FENCED] d88028c2c8c87b4eba725ac3fcb69955a13ca23b fenced=2 flagged=0
UPHOLD-FENCED F1 src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- A routine folder delete with an exact HTML claim and a fuzzy claim in one aggregate is a plausible combination, and it silently removes the surviving artifact's indexed text; no recovery path was confirmed.
UPHOLD-FENCED F2 src/kiro_crew/knowledge/store_parts/item_ownership.py:163 -- The multi-step sequence uses ordinary operations with no contradictory conditions, and it leaves content that normal removal cannot delete; the rarity argument could not be completed from code.
[GPT-ADJUDICATED-FENCED] d88028c2c8c87b4eba725ac3fcb69955a13ca23b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d88028c2c8c87b4eba725ac3fcb69955a13ca23b: <one-sentence reason>

@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from cb81138 to 71d0626 Compare October 21, 2026 18:56
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 findings on cb81138cbc are both valid and fixed in code in 71d0626a92. No override.

  • F1 (adoption cleared claimed_hash after the first chunk): adoption now keeps claimed_hash, so each chunk's own _adopt_reassigned_item call still matches the row. The row's next rewrite drops it, and the cascade's revive still clears it. Test: test_every_chunk_of_a_multi_chunk_winner_is_adopted (3 chunks). It goes red when the clear is restored.
  • F2 (a chained holder kept a marker naming the intermediate source): _move_claims_to_winner now moves merged_into_source_id from the loser's source to the winner's in the same UPDATE that moves claimed_hash. It also runs on an exact collapse. Test: test_a_chained_deferral_follows_the_claim_to_the_new_winner[fuzzy|exact] goes A→B→C, deletes B, deletes A's file, deletes C, and asserts nothing is left untracked. Both cases go red without the repoint.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from 71d0626 to 230bc55 Compare October 21, 2026 19:25
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on 71d0626a92 is valid. It is fixed in code in 230bc55443.

  • Revive cleared claimed_hash on a loser that did not inherit, and left its location on the item another holder now owns. The cascade's revive loop now checks a row with a claimed_hash first. If its own source owns no item under that hash, it calls detach_source_location_by_hash before any branch clears the column. If the copy was reassigned into the row's own source, the location stays (the ambiguous-adoption case). The detach keeps its existing shared-claim rule. With two rows in one source, the claim is released once the second row is processed. Test: test_a_loser_that_does_not_inherit_releases_its_claim covers two fuzzy losers. It deletes the winner, then the heir, and asserts nothing is left untracked. It goes red without the detach.

The GPT 6.1 red on 71d0626a92 was the installation token's primary rate limit, not a verdict. This push reruns that lane.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from 230bc55 to 89b4073 Compare October 21, 2026 20:25
@buluoray

buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on 230bc55443 holds. Fixed in code in 89b407355e.

  • span=231ae0a5fe55 Exact-collapse claim dropped when another holder inherits

Disposition: fixed

mechanism: the cascade's revive now detaches only when claimed_hash differs from the row's own hash (_OWNERSHIP_HASH_COL). An exact claim stays on the row's own content, so the location stays and the own-hash fallback keeps addressing it after the column is cleared.

Reproduced before the fix: an agent document and a folder file exact-lost to one winner; deleting the winner handed the item to the folder and dropped the agent's location. Test test_an_exact_claim_survives_the_heir_being_another_holder pins the folder as heir and then deletes the folder; the agent's copy survives. Removing the claimed != row["own_hash"] guard reddens it (3 of 3 runs). test_a_loser_that_does_not_inherit_releases_its_claim still reddens when the detach is removed, so the fuzzy release is still pinned.

Spec row in docs/system-specs/modules/knowledge.md updated in the same commit. Rebased onto current main for the push guard.

@buluoray

buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on 230bc55443 (security-class, fenced) holds. Fixed in code in 89b407355e; no override.

  • span=d2e0c64f8505 Preserve surviving exact-content claims

Disposition: fixed

mechanism: the cascade's revive detaches only when claimed_hash differs from the row's own item-domain hash (_OWNERSHIP_HASH_COL). An exact-content claim keeps its location and is addressable through the own-hash fallback once claimed_hash is cleared.

Test test_an_exact_claim_survives_the_heir_being_another_holder is red when the guard is reverted: deleting the winning folder hands the item to the other folder, and without the guard the agent's location is dropped, so deleting that folder would destroy the last copy. Same fix as the Opus 5.5 finding on the same line.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 21, 2026
@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from 89b4073 to 54af2b6 Compare October 21, 2026 20:49
@buluoray

buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on 89b407355e holds. Fixed in code in 54af2b6e3b.

  • span=453356bdfa03 A winner-source row repointed to defer to its own source

Disposition: fixed

mechanism: _move_claims_to_winner now skips the winner's own source (holders - {winner.source_id}). A row there keeps deferring to the loser's source, so deleting that source revives it (pending) exactly as on main.

Test test_a_winner_source_row_keeps_deferring_to_the_loser: folder W's row R deferred to L; W's newer file beats L; R still defers to L, and deleting L sets R to pending. Red when the skip is reverted (R defers to W and stays deduped). Spec row updated in the same commit.

@buluoray

buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on 89b407355e (security-class, fenced) holds for the exact claim it names. Fixed in code in 54af2b6e3b; no override.

  • span=2191c36291d4 Preserve unchanged claims when re-ingest is refused

Disposition: fixed

mechanism: release_stale_claim takes the claimed-hash release only for a FUZZY claim (prev_claimed_hash != prev_text_hash or prev_hash). An exact claim is the row's own hash and follows the main-branch rule: kept while the bytes are unchanged, so an add refused on ImportChunkBudgetError drops nothing.

Test test_an_unchanged_reingest_keeps_an_exact_claim[agent_item_state|artifact_item_state]: an exact-collapsed row re-added with unchanged bytes keeps its location. Both params red when the guard is reverted (released 1 claim).

Not changed, stated in the PR body under Not a goal: every ingest path releases a stale claim before the ingest, as main does for changed content. So a FUZZY claim on an unchanged re-add that the budget refuses is still released early. The next re-add retries and ingests the document's own text. Moving every release into each path's success hop is the full fix; it touches all three ingest paths.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from 54af2b6 to 4ec06b1 Compare October 21, 2026 21:19
@buluoray

buluoray commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on 54af2b6e3b (security-class, fenced) holds. Fixed in code in 4ec06b1a2b; no override.

  • span=d2e0c64f8505 Exact HTML-artifact claims are misclassified as fuzzy

Disposition: fixed

mechanism: the root cause was one inference -- exact vs fuzzy was guessed by comparing a row's own hash with the claim hash, and for an artifact those are in two domains (raw text vs extracted prose). The guess is gone. claimed_hash is now written only at the moment of a FUZZY collapse (dedup._fuzzy_claim_hash: loser and winner item hashes differ, both in the item domain), in _collapse_doc and _move_claims_to_winner. An exact collapse never records one. Both hash-comparison guards are removed: the revive in item_rows.py and release_stale_claim in item_ownership.py. A set claimed_hash now always means a fuzzy claim.

Tests:

  • test_an_exact_html_artifact_claim_survives_revive: artifact own hash (raw) != item hash (prose); exact-collapsed with a folder copy; the other folder inherits; deleting it hands the document to the artifact. Red when an exact collapse records a claim hash, and still red with the round-4 own-hash guard restored. That is the case you described.
  • test_deleting_the_loser_file_then_the_winner_leaves_nothing_untracked (fuzzy, the reporter's case): red when no claim hash is recorded.
  • test_an_unchanged_reingest_keeps_an_exact_claim now builds its state through _collapse_doc, not by hand.

Spec row updated in the same commit. Knowledge and store_parts suites: 1894 passed.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
…1954)

A fuzzy collapse merges two documents whose text differs. It moved the
loser's location claims onto the winner's items but left the loser's row
with only its own hash, which names no item afterwards. Every later
release and adoption resolves the claim by hash, so deleting or editing
the loser's file released nothing, and deleting the winner then handed
the loser's source searchable items no state row names.

A fuzzy collapse now writes the winner's items.content_hash to a new
nullable claimed_hash column on the loser's row in all three state
tables, and on the rows of other holders whose claim moved with it.
Folder deletion and re-read, include retirement, artifact and agent
removal and re-ingest, and bundle import release claimed_hash when set.
The claimant count reads COALESCE(claimed_hash, own hash). A fuzzy
loser never inherits the winner's text: when the winner's copy goes,
locations held only through claimed_hash are dropped before an heir is
chosen, and the loser returns to pending, as on main. Adoption matches
only a row's own hash.
A chained holder's deferral marker moves to the winner with its claim,
except in the winner's own source. An exact collapse writes no
claimed_hash, so no reader guesses exact vs fuzzy by comparing an
artifact's raw-text hash with an item hash, and never erases an
existing one. The cascade's revive clears it, first releasing the claim when the
claimed copy went to another holder. Rows written before the
column read NULL and behave exactly as before.

Closes #21954
@buluoray
buluoray force-pushed the fix/knowledge-fuzzy-dedup-claimed-hash-21954 branch from 4ec06b1 to d88028c Compare October 21, 2026 22:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 21, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Opus 5.5 finding on 4ec06b1a2b holds. Fixed in code in d88028c2c8.

  • span=23e201155984 Adoption through claimed_hash marks a fuzzy loser done on the winner's text

Disposition: fixed

mechanism: owner ruling. A fuzzy loser is never handed the winner's text; it is rebuilt from its own bytes, as on main. _adopt_reassigned_item again matches only a row's own hash. A new _drop_fuzzy_claimants drops every location a source holds only through a claimed_hash BEFORE an heir is chosen. It runs in delete_source_cascade and in delete_items_batch_in_txn (a winner file deleted or replaced). So a fuzzy loser cannot inherit the item, and the cascade's revive sends its row to pending. On the item path, _revive_fuzzy_claimants does the same after every chunk is handled.

Tests:

  • test_deleting_the_winner_rebuilds_the_loser_from_its_own_bytes: the winner's text is destroyed, the loser row is pending with no marker or claim, and the next scan stores the loser's own text. Red with the cascade drop removed, and with your round's adoption-through-claim restored.
  • test_deleting_the_winner_file_rebuilds_the_loser: the same via the item path. Red without the drop, and separately without the revive.
  • test_deleting_the_winner_hands_the_loser_row_the_item, which asserted the done outcome, is replaced by the first test above.

Push-driven losers keep main's behaviour (deduped, marker cleared). They are named in the PR body as a follow-up; no scanner is built here.

@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: GPT 6.1 finding on 4ec06b1a2b (security-class, fenced) holds. Fixed in code in d88028c2c8; no override.

  • span=250bdc77ea02 Preserve an adopted fuzzy claim across an exact re-collapse

Disposition: fixed

mechanism: both loser-row writes in _collapse_doc (folder and aggregate) now use claimed_hash = COALESCE(?, claimed_hash). An exact collapse supplies no claim hash and so keeps an existing one. The trigger also relied on a fuzzy loser adopting the winner's items through _handle_deleted -> _adopt_reassigned_item. Per the owner ruling that adoption is gone: the loser's claim is dropped before an heir is chosen, and the row returns to pending (see the Opus disposition).

Test test_an_exact_recollapse_keeps_an_earlier_fuzzy_claim: an exact collapse of a row that already carries a claimed_hash keeps it. Red when the folder COALESCE is reverted. The row is built by hand, because with adoption gone the code no longer produces it, so the test pins the write itself. The aggregate-row COALESCE is the same edit and has no separate test.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 21, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Parked. This PR stays a draft at d88028c2c8. No more patches will be pushed to it.

Claim-ownership findings, in order. There are 12 distinct ones. Rounds 1 and 7 each raised more than one, and round 3's GPT and Opus findings were the same bug.

  1. Round 1, GPT 6.1: adoption cleared claimed_hash after the first chunk, so the other chunks matched nothing. disposition
  2. Round 1, GPT 6.1: a chained holder kept a marker naming the intermediate source. disposition
  3. Round 2, Opus 5.5: the revive cleared a non-inheriting loser's claim but left its location on another holder's item. disposition
  4. Round 3, GPT 6.1 + Opus 5.5: an exact-collapse claim was dropped when another holder inherited the item. Opus, GPT
  5. Round 4, Opus 5.5: a row in the winner's source was repointed to defer to its own source, so it could never be revived. disposition
  6. Round 4, GPT 6.1: an unchanged re-add dropped an exact claim before an ingest the budget refused. disposition
  7. Round 5, GPT 6.1: an HTML artifact's exact claim read as fuzzy, because its row hashes raw text and its items hash prose. disposition
  8. Round 6, Opus 5.5: adoption through claimed_hash marked a fuzzy loser done on the winner's text, so its own file was never re-read. disposition
  9. Round 6, GPT 6.1: an exact re-collapse wrote NULL over an existing fuzzy claim. disposition
  10. Round 7, GPT 6.1 (open): dropping a fuzzy artifact claim also drops an exact HTML claim held in the same source. finding
  11. Round 7, GPT 6.1 (open): a row that claims different content still makes its source count as an heir. finding
  12. Round 7, Opus 5.5 (open): a pre-ingest-gate row with no marker loses its location when a source is deleted, and nothing revives it. finding

The lane comments for 10 to 12 are updated in place. Their content is current as of head d88028c2c8.

Root cause. Claim ownership is spread over three state tables (folder_file_state, artifact_item_state, agent_item_state) and several cascade paths: source deletion, item deletion, dedup collapse, re-ingest and bundle import. No single record says which row owns which content. Each path works ownership out from hashes and markers, so every fix to one path opened a gap in another.

Next step. Design one owner record that every path reads and writes, before any more code. The record should say which row holds which content, and by which claim.

Also red on this head, caused by this diff: the Comment History Gate, for the wording "no longer" at item_ownership.py:183 and test_knowledge_fuzzy_claim_hash.py:734. Left as is, since the PR is parked.

IngridMorstrad added a commit to IngridMorstrad/KiroCrew that referenced this pull request Oct 27, 2026
…ing filenames

## Why is the change being made

On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.

## What has changed

`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.

## Why was this solution chosen?

`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.

## What are the component relationships

```
folder_watcher.py  --stores backslash file_path-->  folder_file_state rows
                                                          |
                                                          v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
                                                          |
                                                          v
                                   filename_near_match (Tier-2 fuzzy gate)
```

Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.

## API changes

None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.

## How was this change tested

`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.

## Backward compatibility

Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.

## Additional I/O and performance impact

None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
IngridMorstrad added a commit to IngridMorstrad/KiroCrew that referenced this pull request Oct 27, 2026
…ing filenames

## Why is the change being made

On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.

## What has changed

`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.

## Why was this solution chosen?

`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.

## What are the component relationships

```
folder_watcher.py  --stores backslash file_path-->  folder_file_state rows
                                                          |
                                                          v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
                                                          |
                                                          v
                                   filename_near_match (Tier-2 fuzzy gate)
```

Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.

## API changes

None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.

## How was this change tested

`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.

## Backward compatibility

Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.

## Additional I/O and performance impact

None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
KiroCrewCaptain pushed a commit that referenced this pull request Oct 27, 2026
…ing filenames (#24834)

## Why is the change being made

On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.

## What has changed

`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.

## Why was this solution chosen?

`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR #22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.

## What are the component relationships

```
folder_watcher.py  --stores backslash file_path-->  folder_file_state rows
                                                          |
                                                          v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
                                                          |
                                                          v
                                   filename_near_match (Tier-2 fuzzy gate)
```

Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.

## API changes

None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.

## How was this change tested

`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.

## Backward compatibility

Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.

## Additional I/O and performance impact

None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 2, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Fuzzy dedup does not record the hash it moves a claim to, so a deleted fuzzy-merged file can leave untracked items

1 participant