Repository navigation
fix(knowledge): strip a Windows backslash folder prefix before comparing filenames - #24834
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (3 items) — 3 justifiedIntent: make dedup treat two copies of one note in different Windows folders as duplicates, the same way it already does on POSIX. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 095105d Description read[DESCRIPTION-READ] f0b30c68e4931af3ad89cf37fa2761641fe019a70b98a2420a559decab48a467 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a separator fix confined to three pure helpers on one compare path, and it changes nothing for paths without a backslash. [DESIGN-REVIEWED] 095105d Description read[DESCRIPTION-READ] cef9ea75cb7186069c0f0f32abbb4c60ebf7df412e313a6a49a8b371493e979b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
bbf2d2e to
b9cb258
Compare
…ing filenames
## Why is the change being made
On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.
## What has changed
`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.
## Why was this solution chosen?
`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.
## What are the component relationships
```
folder_watcher.py --stores backslash file_path--> folder_file_state rows
|
v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
|
v
filename_near_match (Tier-2 fuzzy gate)
```
Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.
## API changes
None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.
## How was this change tested
`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.
## Backward compatibility
Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.
## Additional I/O and performance impact
None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
b9cb258 to
095105d
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: knowledge dedup takes the basename after normalising a Windows backslash path, with tests for the stem, extension and date gates.
Problem / Motivation
Goal: On Windows, two copies of the same note in differently-named folders of a
local_folderknowledge source are recognised as duplicates, exactly as they already are on macOS/Linux.Bug fix:
folder_watcherstores each row'sfile_pathas a backslash OS path on Windows (e.g.C:\Users\me\Vault\Work\meeting-notes.md).knowledge/dedup.py's filename helpers take the basename withrsplit("/", 1)[-1], which does nothing to a backslash path, so the folder prefix survives into the normalized stem and genuine duplicates in different folders are never collapsed on Windows.Why it matters
A Windows user's knowledge base silently keeps duplicate copies of the same document — the same note indexed twice, diluting retrieval and wasting storage — while the identical setup de-duplicates correctly on POSIX. It is invisible (no error), so it is only ever noticed as "dedup doesn't work on Windows".
Not a goal
folder_watcher.pyor howfile_pathis stored._collapse_docor the dedup record-keeping (that is open PR fix(knowledge): record the hash a dedup collapse moves a claim to (#21954) #22859's area; this change is disjoint from it — no shared hunks, no new imports).What changed (motivation → approach → change)
Observed symptom: on Windows, two copies of one note in differently-named folders are not collapsed. Root cause: the three filename-compare helpers assume
/-separated paths and leave the backslash folder prefix in place, so two copies normalise towork\meeting notesvsarchive\meeting notesand the fuzzySequenceMatcherratio (~0.77) falls below the 0.9 floor.Change: normalise
\to/withstr.replacebefore the basename step in the three helpers that read a filename offfile_path—normalize_filename,_file_ext, and_extract_dates. The first two fix the stem/extension comparison._extract_datesis included because it is the date gate on the SAME compare path: once the stems match, if the date gate still scanned the whole backslash path it would read a date from a folder NAME. That is not just the mirror of the stem bug — it opens a data-loss path the stem fix alone would create: two different-period reports under a shared date-named folder (C:\Finance\2026-01-01\Report Apr 2026.mdvsD:\Archive\2026-01-01\Report Dec25.md) would clear the stem gate, and the shared folder date2026-01-01would make their incompatible in-filename dates pass the date gate, so_collapse_docwould silently delete one report's indexed text. Normalising all three keeps the date gate reading the filename's own date. Two one-line edits became three; no imports added, deliberately avoiding the import hunk open draft PR #22859 edits.Backwards compatibility
Compatible: output is unchanged for every input that contains no backslash. All existing sources (POSIX paths, uploads, and URL/vault-relative strings, which are always
/-separated) produce byte-identical stems, extensions and date lists; only a backslash OS path changes, and only to the basename it should already have reduced to.Removes nothing: no path is hidden, no record pruned, no validator tightened, no key or function removed or renamed — the diff only normalises a separator inside three pure helpers.
Tests
test/test_knowledge_dedup_winpath.py(new): the Windows two-folder case, the absolute-path case, and the shared-folder-date data-loss case (all red without the fix), plus POSIX and_file_extcontrols. The existingtest/test_knowledge_dedup.pysuite (69 tests) still passes.Proof: PROVEN test/test_knowledge_dedup_winpath.py::test_windows_same_note_two_folders_must_collapse, test/test_knowledge_dedup_winpath.py::test_windows_absolute_path_strips_to_basename, test/test_knowledge_dedup_winpath.py::test_windows_shared_folder_date_does_not_force_collapse (head b9cb258)
Gate proof: tree 40cd233615ab; ran 24; skipped none
Manual verification
N/A — unit coverage is sufficient; the defect is pure string logic and the backslash
file_pathfed to the tests is exactly what the Windowsfolder_watcherwalk persists.Screenshots / video
N/A — no user-visible frontend surface is touched (backend knowledge-dedup helpers only).
Related Issues
Closes #24833
Sibling sweep
Swept:
git grep -nE 'rsplit\("/", ?1\)\[-1\]' -- src/kiro_crew/knowledge/dedup.pydedup.py:normalize_filename(stem compare) — changed here.dedup.py:_file_ext(extension/format rank) — changed here.dedup.py:_extract_dates(date gate on the compare path) — changed here (closes the shared-folder-date data-loss path the stem fix alone would open).dedup.py:426anddedup.py:1050(enumerate_docs/ lookup setDocRef.filenamefor DISPLAY) — not affected for correctness — every compare the collapse decision uses (normalize_filename,_file_ext,_extract_dates) now normalises the separator itself, so a backslash-containingDocRef.filenamecannot mis-collapse; left as-is to keep the diff tight and disjoint from fix(knowledge): record the hash a dedup collapse moves a claim to (#21954) #22859.Pattern harvest
Rule candidate: review-prompt
Pattern: taking a path basename with
rsplit("/", 1)[-1](hard-coded/) on a value that can be a Windows backslash OS path — useos.path.basename/ntpath.basenameor normalise the separator first. A partial fix is its own hazard: normalising one of several helpers on the same compare path can turn a "fails to match" bug into a "wrongly matches / destroys data" bug.Checklist