Skip to content

fix(knowledge): strip a Windows backslash folder prefix before comparing filenames - #24834

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/knowledge-dedup-windows-backslash-filename
Oct 27, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
IngridMorstrad:fix/knowledge-dedup-windows-backslash-filename

Conversation

@IngridMorstrad

@IngridMorstrad IngridMorstrad commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: On Windows, two copies of the same note in differently-named folders of a local_folder knowledge source are recognised as duplicates, exactly as they already are on macOS/Linux.

Bug fix: folder_watcher stores each row's file_path as a backslash OS path on Windows (e.g. C:\Users\me\Vault\Work\meeting-notes.md). knowledge/dedup.py's filename helpers take the basename with rsplit("/", 1)[-1], which does nothing to a backslash path, so the folder prefix survives into the normalized stem and genuine duplicates in different folders are never collapsed on Windows.

Why it matters

A Windows user's knowledge base silently keeps duplicate copies of the same document — the same note indexed twice, diluting retrieval and wasting storage — while the identical setup de-duplicates correctly on POSIX. It is invisible (no error), so it is only ever noticed as "dedup doesn't work on Windows".

Not a goal

What changed (motivation → approach → change)

Observed symptom: on Windows, two copies of one note in differently-named folders are not collapsed. Root cause: the three filename-compare helpers assume /-separated paths and leave the backslash folder prefix in place, so two copies normalise to work\meeting notes vs archive\meeting notes and the fuzzy SequenceMatcher ratio (~0.77) falls below the 0.9 floor.

Change: normalise \ to / with str.replace before the basename step in the three helpers that read a filename off file_path — normalize_filename, _file_ext, and _extract_dates. The first two fix the stem/extension comparison. _extract_dates is included because it is the date gate on the SAME compare path: once the stems match, if the date gate still scanned the whole backslash path it would read a date from a folder NAME. That is not just the mirror of the stem bug — it opens a data-loss path the stem fix alone would create: two different-period reports under a shared date-named folder (C:\Finance\2026-01-01\Report Apr 2026.md vs D:\Archive\2026-01-01\Report Dec25.md) would clear the stem gate, and the shared folder date 2026-01-01 would make their incompatible in-filename dates pass the date gate, so _collapse_doc would silently delete one report's indexed text. Normalising all three keeps the date gate reading the filename's own date. Two one-line edits became three; no imports added, deliberately avoiding the import hunk open draft PR #22859 edits.

Backwards compatibility

Compatible: output is unchanged for every input that contains no backslash. All existing sources (POSIX paths, uploads, and URL/vault-relative strings, which are always /-separated) produce byte-identical stems, extensions and date lists; only a backslash OS path changes, and only to the basename it should already have reduced to.

Removes nothing: no path is hidden, no record pruned, no validator tightened, no key or function removed or renamed — the diff only normalises a separator inside three pure helpers.

Tests

test/test_knowledge_dedup_winpath.py (new): the Windows two-folder case, the absolute-path case, and the shared-folder-date data-loss case (all red without the fix), plus POSIX and _file_ext controls. The existing test/test_knowledge_dedup.py suite (69 tests) still passes.

Proof: PROVEN test/test_knowledge_dedup_winpath.py::test_windows_same_note_two_folders_must_collapse, test/test_knowledge_dedup_winpath.py::test_windows_absolute_path_strips_to_basename, test/test_knowledge_dedup_winpath.py::test_windows_shared_folder_date_does_not_force_collapse (head b9cb258)

Gate proof: tree 40cd233615ab; ran 24; skipped none

Manual verification

N/A — unit coverage is sufficient; the defect is pure string logic and the backslash file_path fed to the tests is exactly what the Windows folder_watcher walk persists.

Screenshots / video

N/A — no user-visible frontend surface is touched (backend knowledge-dedup helpers only).

Related Issues

Closes #24833

Sibling sweep

Swept: git grep -nE 'rsplit\("/", ?1\)\[-1\]' -- src/kiro_crew/knowledge/dedup.py

  • Sibling: dedup.py:normalize_filename (stem compare) — changed here.
  • Sibling: dedup.py:_file_ext (extension/format rank) — changed here.
  • Sibling: dedup.py:_extract_dates (date gate on the compare path) — changed here (closes the shared-folder-date data-loss path the stem fix alone would open).
  • Sibling: dedup.py:426 and dedup.py:1050 (enumerate_docs / lookup set DocRef.filename for DISPLAY) — not affected for correctness — every compare the collapse decision uses (normalize_filename, _file_ext, _extract_dates) now normalises the separator itself, so a backslash-containing DocRef.filename cannot mis-collapse; left as-is to keep the diff tight and disjoint from fix(knowledge): record the hash a dedup collapse moves a claim to (#21954) #22859.

Pattern harvest

Rule candidate: review-prompt
Pattern: taking a path basename with rsplit("/", 1)[-1] (hard-coded /) on a value that can be a Windows backslash OS path — use os.path.basename / ntpath.basename or normalise the separator first. A partial fix is its own hazard: normalising one of several helpers on the same compare path can turn a "fails to match" bug into a "wrongly matches / destroys data" bug.

Checklist

  • At most two commits (one here), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable) — N/A
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 27, 2026
@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 095105d241a478112ce6f6c46e5665db25af90cd touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 095105d241a478112ce6f6c46e5665db25af90cd via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: make dedup treat two copies of one note in different Windows folders as duplicates, the same way it already does on POSIX. This is a FIX.

  1. On Windows, copies of one note in differently named local_folder folders now collapse, because the folder prefix is stripped from the compared stem — justified
  2. On Windows, the date gate now reads the date from the filename, not from a folder name, so a date in a shared folder name can no longer clear two different reports for collapse — justified
  3. On Windows, the format rank now takes the extension from the filename, so a . in a folder name is no longer read as an extension — justified

[FIRST-PRINCIPLES-REVIEWED] 095105d

Description read

[DESCRIPTION-READ] f0b30c68e4931af3ad89cf37fa2761641fe019a70b98a2420a559decab48a467

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 095105d241a478112ce6f6c46e5665db25af90cd via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 095105d

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 095105d241a478112ce6f6c46e5665db25af90cd via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a separator fix confined to three pure helpers on one compare path, and it changes nothing for paths without a backslash.

[DESIGN-REVIEWED] 095105d

Description read

[DESCRIPTION-READ] cef9ea75cb7186069c0f0f32abbb4c60ebf7df412e313a6a49a8b371493e979b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 095105d241a478112ce6f6c46e5665db25af90cd via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 095105d

@IngridMorstrad
IngridMorstrad force-pushed the fix/knowledge-dedup-windows-backslash-filename branch from bbf2d2e to b9cb258 Compare October 27, 2026 09:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 27, 2026
…ing filenames

## Why is the change being made

On a Windows `local_folder` knowledge source, `folder_watcher` stores each
row's `file_path` as `os.path.join(dirpath, fname)` from `os.walk(uri)`, which
is a backslash-separated OS path (e.g. `C:\Users\me\Vault\Work\meeting-notes.md`).
`knowledge/dedup.py` derives the compare/display filename with
`rsplit("/", 1)[-1]`, which does nothing to a backslash path, so the whole
folder prefix survives into the normalized stem. Two genuine copies of one note
in differently-named folders then normalise to `work\meeting notes` vs
`archive\meeting notes`; the `SequenceMatcher` ratio (~0.77) drops below the 0.9
floor, `filename_near_match` returns False, and cross-source fuzzy dedup silently
fails to collapse them. POSIX collapses the same two notes correctly. The result
is a silent correctness gap: the knowledge base keeps duplicate copies of the
same document on Windows only.

## What has changed

`normalize_filename` and `_file_ext` now normalise `\` to `/` before taking the
basename, so the folder prefix is stripped on Windows exactly as on POSIX. The
change is two one-line edits plus comments; no imports added. A new test
`test/test_knowledge_dedup_winpath.py` adds the Windows two-folder case and the
absolute-path case (red before the fix) with a POSIX control and a `_file_ext`
control.

## Why was this solution chosen?

`str.replace("\\", "/")` is the minimal, dependency-free normalisation and keeps
the diff to the two functions the defect lives in. `ntpath.basename` was the
alternative but would add an import next to the hunk open draft PR kirodotdev#22859 edits,
and is POSIX-lossy for a genuine `/`-containing name; plain replace avoids both.

## What are the component relationships

```
folder_watcher.py  --stores backslash file_path-->  folder_file_state rows
                                                          |
                                                          v
dedup.py: enumerate_docs --> normalize_filename / _file_ext (basename extract)
                                                          |
                                                          v
                                   filename_near_match (Tier-2 fuzzy gate)
```

Sequence: a Windows `local_folder` walk persists `Work\notes.md` and
`Archive\notes.md` -> dedup normalises each to `meeting notes` -> fuzzy gate
collapses the duplicate.

## API changes

None. `normalize_filename` and `_file_ext` keep the same signature and, for any
input without a backslash (every POSIX path and URL/vault-relative string), the
same output.

## How was this change tested

`test/test_knowledge_dedup_winpath.py` (new) proves the Windows two-folder and
absolute-path cases fail without the fix and pass with it; the full existing
`test/test_knowledge_dedup.py` suite (69 tests) still passes. Run under the repo
venv with an isolated temp `HOME`/`KIROCREW_HOME`.

## Backward compatibility

Compatible: output is unchanged for every input that contains no backslash, so
all existing sources (POSIX paths, uploads, URL/vault-relative strings) behave
exactly as before; only a backslash OS path changes, and only to the basename it
should already have reduced to.

## Additional I/O and performance impact

None. One extra in-memory `str.replace` per filename normalisation; no new I/O.
@IngridMorstrad
IngridMorstrad force-pushed the fix/knowledge-dedup-windows-backslash-filename branch from b9cb258 to 095105d Compare October 27, 2026 12:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 27, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 27, 2026 21:06

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: knowledge dedup takes the basename after normalising a Windows backslash path, with tests for the stem, extension and date gates.

@KiroCrewCaptain
KiroCrewCaptain merged commit a6e0145 into kirodotdev:main Oct 27, 2026
153 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: knowledge dedup leaves the folder prefix in a backslash file_path, so duplicate folder copies are never collapsed

2 participants