Repository navigation
fix(knowledge): keep a folder file the scan could not see - #33963
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the fix stops the scan from reading "this walk could not see the file" as "the file was deleted", and real not-found errors still count as deletions. Notes
[DESIGN-REVIEWED] 9692699 Description read[DESCRIPTION-READ] e60ce5046d703ecdb57dbca2359bc4f9ff670cd901ac39c501309d831e00af0a That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Issue #33950 is the defect, and the tests the PR adds fail on main. Each item fixes the cause: a gap in the walk was being read as a deletion. What this change shipsInventory (4 items) — 4 justifiedIntent: indexed knowledge-folder documents survive a scan that could not read part of the folder. FIX.
[FIRST-PRINCIPLES-REVIEWED] 9692699 Description read[DESCRIPTION-READ] d3b4735433d7599f583a40dd3c2216c7636dc497d9cd8004add69cbfcae1e16a That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
9e43942 to
5fe0518
Compare
|
|
5fe0518 to
80367b5
Compare
|
80367b5 to
f07de0a
Compare
os.walk skips a directory it cannot list, and the folder walk skipped a file it could not stat. Either can be transient: a permission change, or a network mount that drops out mid-walk. The scan then read every tracked file in that gap as deleted and removed its indexed items, which only a fresh, paid re-ingest brings back. Record the directories the walk could not list and the files it could not stat, and keep their rows and items. A file that is really gone is still removed. A file whose stat answers not-found (or not-a-directory) is gone, so it is still removed: it was deleted after the listing, or it is a symlink whose target is gone. An AsciiDoc includer whose target the walk could not see keeps its indexed text too.
f07de0a to
9692699
Compare
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Approved. On this head the review checked that a directory that fails to list, or a file that fails to stat, now keeps its rows and items. A not-found result still counts as a deletion: a vanished directory, a deleted file, a dangling symlink, or a symlink whose target became a file. The sensitive-path and root-confinement screens still run first and are unchanged. The realpath work stays in the walk's worker thread, off the event loop. The spec in knowledge.md is updated in the same commit. Against main's folder_watcher.py, three of the new tests fail, and all six pass at this head.
Done well: the tests drive the real scan against a real store, and they inject the failure the way os.walk reports it. They also pin the delete side, so the fix cannot turn into "never delete". The include test checks that no re-ingest was paid for.
Optional:
_target_present(folder_watcher.py:543): no test reaches the prefix branch for an include target inside a directory that failed to list. A sibling of the include test that failssubthrough the os.walk onerror fake would pin it. A similar case: parametrize the onerror fake over FileNotFoundError and NotADirectoryError, and assert thatsub/deep.mdis deleted. That pins the not-found exclusion at line 867.- knowledge.md:298 and folder_watcher.py:519-521 say a network mount that drops out costs no indexed content. An unmount reads as not-found or as an empty directory, so it is still treated as a deletion. Wording like "a mount that errors mid-walk (EIO, ESTALE, a timeout)" would match the code.
- The two symlink tests skip on
os.name == "nt". The testing conventions treat symlinks as a capability: list these tests in test/requires-real-symlinks.txt and drop the platform skip.
os.walk skips a directory it cannot list, and the folder walk skipped a file it could not stat. Either can be transient: a permission change, or a network mount that drops out mid-walk. The scan then read every tracked file in that gap as deleted and removed its indexed items, which only a fresh, paid re-ingest brings back. Record the directories the walk could not list and the files it could not stat, and keep their rows and items. A file that is really gone is still removed. A file whose stat answers not-found (or not-a-directory) is gone, so it is still removed: it was deleted after the listing, or it is a symlink whose target is gone. An AsciiDoc includer whose target the walk could not see keeps its indexed text too. Backport note: conflicts only where main's hunks sit next to the ignore-pattern keep of #31410 (unreleased). folder_watcher.py adds the unlisted and unlisted_real sets to release's step 1 walk call (which has no tracked or pattern_ignored arguments) and the not-unseen check to release's removed list; _walk gains only the two new keywords. knowledge.md adds the two new sentences to release's bullets, without main's ignore_patterns paragraph. The test applies as on main. (cherry picked from commit 73e3275)
Problem / Motivation
Goal: A knowledge folder scan keeps the indexed text of files it could not see, instead of deleting it as if the files were gone.
FolderWatcher._walkcalledos.walkwithoutonerror, so a subdirectory that failed to list was skipped silently, and it skipped a file whoseos.statfailed. The scan then read every tracked file in that gap as deleted, and_handle_deletedremoved its items. An AsciiDoc includer whose include target was in the gap looked stale and had its text retired too.Why it matters
Both failures can be brief: a permission change, or a network mount that drops out mid-walk. One bad sweep removes documents from search, and getting them back costs a full, paid re-ingest once the folder is readable again.
Not a goal
.kiroignore, size and extension rules).What changed (motivation → approach → change)
Symptom: indexed files disappear after a scan that could not read part of the folder. Root cause: "not found by this walk" was treated as "deleted". Change:
_walkrecords each directory it could not list (os.walk(onerror=...)) and each file it could notstat; the scan leaves those rows and items alone, and counts an unseen include target as present. A not-found answer still means gone: a directory that vanished after its parent was listed, a file deleted after the listing, and a symlink whose target is gone are all still removed.Backwards compatibility
Compatible: nothing main accepts is refused; a scan only removes less.
Removes nothing: the change keeps rows a scan used to delete; it removes no row, key or behaviour a reader depends on.
Tests
test/test_folder_watcher_walk_errors.py:The first two fail on main. 133 folder-watcher tests pass, and 2452 knowledge tests passed on the first version of the fix.
Proof: PROVEN test/test_folder_watcher_walk_errors.py::test_a_directory_that_fails_to_list_keeps_its_files, test/test_folder_watcher_walk_errors.py::test_a_file_that_fails_to_stat_keeps_its_items, test/test_folder_watcher_walk_errors.py::test_an_unseen_include_target_keeps_the_includer_text (head 9692699)
Manual verification
N/A — unit coverage sufficient: the tests drive the real scan against a real store, with the failing listing and stat injected.
Related Issues
Fixes #33950
Open PRs on the same file: #19789 and #22859 change other parts of
folder_watcher.py; no lines overlap.Sibling sweep
Swept: git grep -n "os.walk(" -- src/kiro_crew/knowledge
Siblings: none -- folder_watcher.py holds the only walk in knowledge/ whose result decides a deletion
Pattern harvest
Rule candidate: review-prompt
Pattern: a directory walk whose skipped entries (listing errors, stat errors) are later read as deletions.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the Apache-2.0 license of this project; see the Licensing section of CONTRIBUTING.md.