Skip to content

fix(knowledge): keep a folder file the scan could not see - #33963

Merged
RohanK6 merged 1 commit into
mainfrom
crew/conductor/knowledge-walk-unlisted
Nov 10, 2026
Merged

RohanK6 merged 1 commit into
mainfrom
crew/conductor/knowledge-walk-unlisted

Conversation

@RohanK6

@RohanK6 RohanK6 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A knowledge folder scan keeps the indexed text of files it could not see, instead of deleting it as if the files were gone.

FolderWatcher._walk called os.walk without onerror, so a subdirectory that failed to list was skipped silently, and it skipped a file whose os.stat failed. The scan then read every tracked file in that gap as deleted, and _handle_deleted removed its items. An AsciiDoc includer whose include target was in the gap looked stale and had its text retired too.

Why it matters

Both failures can be brief: a permission change, or a network mount that drops out mid-walk. One bad sweep removes documents from search, and getting them back costs a full, paid re-ingest once the folder is readable again.

Not a goal

  • Changing what a scan treats as excluded (ignore patterns, .kiroignore, size and extension rules).
  • Retrying a failed listing within the same sweep.

What changed (motivation → approach → change)

Symptom: indexed files disappear after a scan that could not read part of the folder. Root cause: "not found by this walk" was treated as "deleted". Change: _walk records each directory it could not list (os.walk(onerror=...)) and each file it could not stat; the scan leaves those rows and items alone, and counts an unseen include target as present. A not-found answer still means gone: a directory that vanished after its parent was listed, a file deleted after the listing, and a symlink whose target is gone are all still removed.

Backwards compatibility

Compatible: nothing main accepts is refused; a scan only removes less.

Removes nothing: the change keeps rows a scan used to delete; it removes no row, key or behaviour a reader depends on.

Tests

test/test_folder_watcher_walk_errors.py:

  • a directory that fails to list keeps its files
  • a file that fails to stat keeps its items
  • an unseen include target keeps the includer text, with no new ingest
  • a dangling symlink is still removed
  • a symlink whose target directory became a file is still removed
  • a really deleted file is still removed

The first two fail on main. 133 folder-watcher tests pass, and 2452 knowledge tests passed on the first version of the fix.

Proof: PROVEN test/test_folder_watcher_walk_errors.py::test_a_directory_that_fails_to_list_keeps_its_files, test/test_folder_watcher_walk_errors.py::test_a_file_that_fails_to_stat_keeps_its_items, test/test_folder_watcher_walk_errors.py::test_an_unseen_include_target_keeps_the_includer_text (head 9692699)

Manual verification

N/A — unit coverage sufficient: the tests drive the real scan against a real store, with the failing listing and stat injected.

Related Issues

Fixes #33950

Open PRs on the same file: #19789 and #22859 change other parts of folder_watcher.py; no lines overlap.

Sibling sweep

Swept: git grep -n "os.walk(" -- src/kiro_crew/knowledge
Siblings: none -- folder_watcher.py holds the only walk in knowledge/ whose result decides a deletion

Pattern harvest

Rule candidate: review-prompt
Pattern: a directory walk whose skipped entries (listing errors, stat errors) are later read as deletions.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the Apache-2.0 license of this project; see the Licensing section of CONTRIBUTING.md.

@RohanK6
RohanK6 requested review from a team and KiroCrewCaptain as code owners November 9, 2026 16:11
@RohanK6
RohanK6 requested a review from Zedmor November 9, 2026 16:11
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 9, 2026
@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 969269926eaa4110b8e658e4dea977980bc36f46 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the fix stops the scan from reading "this walk could not see the file" as "the file was deleted", and real not-found errors still count as deletions.

Notes

  • A directory that stays unreadable for good (for example, after a chmod 000) now keeps its indexed items with no time limit. That is safe, but users get no signal that their search results are frozen. Reporting an unseen count in the scan stats next to deleted would show it at no real cost.

[DESIGN-REVIEWED] 9692699

Description read

[DESCRIPTION-READ] e60ce5046d703ecdb57dbca2359bc4f9ff670cd901ac39c501309d831e00af0a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 969269926eaa4110b8e658e4dea977980bc36f46 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Issue #33950 is the defect, and the tests the PR adds fail on main. Each item fixes the cause: a gap in the walk was being read as a deletion.

What this change ships

Inventory (4 items) — 4 justified

Intent: indexed knowledge-folder documents survive a scan that could not read part of the folder. FIX.

  1. A subdirectory that fails to list (permission error, a mount that drops out) no longer has its indexed files deleted. — justified
  2. A file whose stat fails keeps its row and items instead of being archived. — justified
  3. An AsciiDoc includer whose include target the walk could not see keeps its text, with no paid re-ingest. — justified
  4. _walk gains private unlisted / unlisted_real out-parameters, used only by _do_scan, its sole caller (grep _walk\(|os\.walk\( in knowledge/: 1 walk, 1 caller). A real not-found (vanished directory, deleted file, dangling symlink) is still removed. — justified

[FIRST-PRINCIPLES-REVIEWED] 9692699

Description read

[DESCRIPTION-READ] d3b4735433d7599f583a40dd3c2216c7636dc497d9cd8004add69cbfcae1e16a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 969269926eaa4110b8e658e4dea977980bc36f46 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 969269926eaa4110b8e658e4dea977980bc36f46 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9692699

Verdict parsed from the review's SHA-scoped output markers for commit 969269926eaa4110b8e658e4dea977980bc36f46.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 969269926eaa4110b8e658e4dea977980bc36f46 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 969269926eaa4110b8e658e4dea977980bc36f46 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9692699

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 969269926eaa4110b8e658e4dea977980bc36f46 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@RohanK6
RohanK6 force-pushed the crew/conductor/knowledge-walk-unlisted branch from 9e43942 to 5fe0518 Compare November 9, 2026 16:33
@RohanK6

RohanK6 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author
  • Resolve unseen paths off the gateway event loop span=27a975dd11d9

fixed in 5fe0518: the resolved, normcased unseen paths are now collected inside the walk, which already runs in a worker thread; the scan on the event loop only compares strings.

@RohanK6

RohanK6 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author
  • Unseen paths resolved on the event loop span=7a6e422a2256

fixed in 5fe0518: realpath now runs in the walk worker thread, so a stalled mount cannot block the loop. The sync-io-in-async gate passes locally.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026
@RohanK6
RohanK6 force-pushed the crew/conductor/knowledge-walk-unlisted branch from 5fe0518 to 80367b5 Compare November 9, 2026 17:12
@RohanK6

RohanK6 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author
  • Owning spec not updated span=0b8344a57c28

fixed in 80367b5: docs/system-specs/modules/knowledge.md now says an unseen file keeps its row and items, an unseen include target counts as present, and a not-found answer still means gone. It is part of the same single commit.

@RohanK6
RohanK6 force-pushed the crew/conductor/knowledge-walk-unlisted branch from 80367b5 to f07de0a Compare November 9, 2026 19:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 9, 2026
os.walk skips a directory it cannot list, and the folder walk skipped
a file it could not stat. Either can be transient: a permission
change, or a network mount that drops out mid-walk. The scan then
read every tracked file in that gap as deleted and removed its
indexed items, which only a fresh, paid re-ingest brings back.

Record the directories the walk could not list and the files it
could not stat, and keep their rows and items. A file that is
really gone is still removed.

A file whose stat answers not-found (or not-a-directory) is gone, so it is still removed:
it was deleted after the listing, or it is a symlink whose target is
gone. An AsciiDoc includer whose target the walk could not see keeps
its indexed text too.
@RohanK6
RohanK6 force-pushed the crew/conductor/knowledge-walk-unlisted branch from f07de0a to 9692699 Compare November 9, 2026 22:30
@RohanK6

RohanK6 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author
  • Treat ENOTDIR as a missing target, not a visibility failure span=27a975dd11d9

fixed in 9692699: NotADirectoryError now counts as gone in both places, the file stat and the directory listing, so such a link is removed. A new test covers a symlink whose target directory became a regular file.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 9, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 9, 2026
@RohanK6
RohanK6 enabled auto-merge (squash) November 9, 2026 23:53

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved. On this head the review checked that a directory that fails to list, or a file that fails to stat, now keeps its rows and items. A not-found result still counts as a deletion: a vanished directory, a deleted file, a dangling symlink, or a symlink whose target became a file. The sensitive-path and root-confinement screens still run first and are unchanged. The realpath work stays in the walk's worker thread, off the event loop. The spec in knowledge.md is updated in the same commit. Against main's folder_watcher.py, three of the new tests fail, and all six pass at this head.

Done well: the tests drive the real scan against a real store, and they inject the failure the way os.walk reports it. They also pin the delete side, so the fix cannot turn into "never delete". The include test checks that no re-ingest was paid for.

Optional:

  • _target_present (folder_watcher.py:543): no test reaches the prefix branch for an include target inside a directory that failed to list. A sibling of the include test that fails sub through the os.walk onerror fake would pin it. A similar case: parametrize the onerror fake over FileNotFoundError and NotADirectoryError, and assert that sub/deep.md is deleted. That pins the not-found exclusion at line 867.
  • knowledge.md:298 and folder_watcher.py:519-521 say a network mount that drops out costs no indexed content. An unmount reads as not-found or as an empty directory, so it is still treated as a deletion. Wording like "a mount that errors mid-walk (EIO, ESTALE, a timeout)" would match the code.
  • The two symlink tests skip on os.name == "nt". The testing conventions treat symlinks as a capability: list these tests in test/requires-real-symlinks.txt and drop the platform skip.

@RohanK6
RohanK6 merged commit 73e3275 into main Nov 10, 2026
99 of 108 checks passed
@RohanK6
RohanK6 deleted the crew/conductor/knowledge-walk-unlisted branch November 10, 2026 00:06
bolichen97 pushed a commit that referenced this pull request Nov 10, 2026
os.walk skips a directory it cannot list, and the folder walk skipped
a file it could not stat. Either can be transient: a permission
change, or a network mount that drops out mid-walk. The scan then
read every tracked file in that gap as deleted and removed its
indexed items, which only a fresh, paid re-ingest brings back.

Record the directories the walk could not list and the files it
could not stat, and keep their rows and items. A file that is
really gone is still removed.

A file whose stat answers not-found (or not-a-directory) is gone, so it is still removed:
it was deleted after the listing, or it is a symlink whose target is
gone. An AsciiDoc includer whose target the walk could not see keeps
its indexed text too.

Backport note: conflicts only where main's hunks sit next to the
ignore-pattern keep of #31410 (unreleased). folder_watcher.py adds the
unlisted and unlisted_real sets to release's step 1 walk call (which
has no tracked or pattern_ignored arguments) and the not-unseen check to
release's removed list; _walk gains only the two new keywords.
knowledge.md adds the two new sentences to release's bullets, without
main's ignore_patterns paragraph. The test applies as on main.

(cherry picked from commit 73e3275)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

knowledge: a folder scan deletes indexed files it could not see

2 participants