Repository navigation
fix(files): bound the project tree walk and build its body off the loop - #16029
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS Folder rows are now capped, so on a project over 10,000 items the Files rail can no longer reach some folders, and its filter is still local only. Watch
[DESIGN-REVIEWED] 4232b17 Description read[DESCRIPTION-READ] 2ca15e29fcceb4a9e6a8c13445f56366838dd9f8f31ae1b3aaab7b087443b2d9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Moving the work off the event loop and bounding the walk already end the stall. Capping folder rows at 10,000 removes navigation, and pinning folder identity adds hardening. The Goal needs neither. Not justified as shipped
What this change shipsInventory (10 items) — 6 justifiedIntent: keep the gateway responsive while a large project's Files tree is open — FIX.
[FIRST-PRINCIPLES-REVIEWED] 4232b17 Description read[DESCRIPTION-READ] aee371c9b6155d0ca48d26fde641b4b160e6f2f75d258b2a5d6f2921b18a456f That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
UX Review (Opus 5.5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS The empty truncated workspace says "some not shown" over an empty pane and offers no way forward, and the new search-scope line confused the blind reader. Watch
Suggestions
[UX-REVIEWED] 4232b17 Description read[DESCRIPTION-READ] 184e2931a033afade915983332b6b024c258f4985accae78a63b8ee227126a82 That sha256 names everything this verdict read: the PR description, plus the 10 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
9e366a1 to
deb05dc
Compare
|
|
|
|
|
Pushed
Rebutted (no code change):
Local gate: static gates pass and the related frontend tests pass. One timing test, |
deb05dc to
fb0a3cc
Compare
|
self-added: yes
|
fb0a3cc to
056bc3f
Compare
The non-git fallback of GET /api/project/tree walked the whole project twice with no bound on directories, then redacted every row and serialized the body on the event loop. The dashboard refetches the tree every 10 s while it is open, so a large non-git project, a repository whose ls-files overflows its cap or timeout, or a host with no sandbox backend stalled the gateway on every poll. The walk is now one breadth-first scandir pass under one read budget, look-ahead included. Each folder of a depth, and the depths below as one more claimant, is guaranteed half the even split the depth starts with, and a folder may read what the claimants after it do not need for that floor, up to the row cap: one large folder cannot starve its siblings, and a small tree beside one large folder is read whole. Each queued folder is read only if it is still the folder its parent listed: POSIX opens it O_DIRECTORY|O_NOFOLLOW and compares the fstat identity before listing the descriptor, Windows compares an lstat just before listing. A link is a symlink or a name-surrogate reparse point, so a cloud-files placeholder folder is still walked. Both branches spend one combined row cap on folder rows and files together: folders first, shallowest first, files keeping at least half the cap and sharing it round-robin by parent. Every shown folder whose listing was cut is named in truncatedDirectories; a listed folder that cannot be read, its identity lstat included, or that is no longer the folder its parent listed, is named in unreadableDirectories. Redaction and JSON serialization run in the same worker thread as the listing. The dashboard follows the capped contract: the truncation copy names an item limit, a folder folded into a chain row keeps its badge, a truncated payload with no rows shows the limit instead of an empty workspace, unreadable-folder dismissals survive a cut payload, and the folder tab sends its search to the server when the tree is truncated, merges in the tree's own matching paths, says the search also covers items not listed, and keeps the tree mounted behind the results.
273b12d to
4232b17
Compare
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
|
self-added: yes
|
buluoray
left a comment
There was a problem hiding this comment.
Reviewed at head 4232b17197. I checked the bound, the work moved off the event loop, the sensitive-path fence and symlink loops. None of them has a defect that should block this PR.
The bound limits the work
- Every folder read takes at most
allowance = min(share + 1, budget)entries, andbudget -= pulledruns on both the success path and the error path (project_tree.py_project_tree_walk). I wrappedos.scandirto count the entries actually pulled from a 125k-entry tree. With limit 200,000 it pulled 125,040, and the tree was read in full (that is under the budget). With limit 20,000 it pulled 19,822; with limit 1,000 it pulled 999. Rows returned stayed at or under the row cap each time, andtruncated=Truenamed 41 folders. A non-dot root took 0.08 s for the full 125k. - Only rows inside the cap are queued, so the number of folders the walk opens is bounded by the cap.
_project_tree_allotcaps directory rows and files together on both branches.truncatedisbool(truncatedDirectories). - Fail-closed: when the share is at least the budget, the look-ahead can mark a folder that was actually complete as truncated. That over-reports truncation and never under-reports it.
Work moved off the loop
_runand_project_tree_bodynow run in oneasyncio.to_threadcall._sel(), the allow-list match andDashboardStateaccess stay on the loop. The thread touches only the filesystem,_run_git_boundedand the redactors.redactisredact_via_context→current_context(), andto_threadcopies the contextvars. I found no loop-only state used from the thread.
Sensitive-path filtering
_project_tree_fenceis byte-identical to main's_project_tree_entries.fenced. It is still applied to every file entry and every directory entry before it is kept, including the per-folderrealpathshortcut. The handler still runsis_sensitive_path(base)on the resolved root before listing anything.
Symlink loops
- Links are never followed.
_project_tree_identityuseslstat, and a link becomes alinkedDirectoriesrow; it is never queued. Each queued folder is re-opened withO_DIRECTORY|O_NOFOLLOWand checked against the(st_dev, st_ino)its parent recorded. - I built a tree containing
a/b/up -> ..,self -> rootand a self-referential dangling linkx. The walk finished in 2 ms:selfanda/b/upcame back as linked rows, andxcame back as a file. There was no recursion.
Mutation probes (test/test_project_tree.py; each mutation was reverted, and the tree is clean at the end)
| Guard removed | Test that fails |
|---|---|
| look-ahead taken outside the budget | test_a_folder_the_scan_budget_never_reaches_is_named_and_never_read |
if budget <= 0 stop |
same test |
| per-folder share uncapped | test_one_large_folder_does_not_starve_its_siblings_of_the_scan[data,zz_data], test_a_large_folder_sorted_first_still_leaves_each_sibling_its_floor |
| descriptor identity compare | test_a_folder_swapped_after_its_parent_was_listed_is_not_read[descriptor-rename] |
| by-name identity/link check | ...[by-name-rename], ...[by-name-symlink], ...[descriptor-symlink] |
| link → linked row | test_non_repo_walk_lists_a_junction_as_a_link_and_never_walks_it and 2 others |
| fence disabled | 4 fence/sensitive tests |
| body built on the loop | test_redaction_and_serialization_run_off_the_event_loop[git,walk] |
Without any mutation, the related backend files passed 176/176 (test_project_tree.py, test_dashboard_files_composition_contract.py, test_pathres_loop_starvation.py), and the 4 touched frontend test files passed 170/170.
Earlier review: chenmingwei23's review on 273b12d3bb was dismissed. It has an empty body and no inline comments, so there is no point from it to re-check. The bot findings bolichen97 marked fixed were F1 (identity pin), F2 (look-ahead inside the budget) and the runtime-skip finding. I checked them at this head: the code is present, and the first two are covered by the probes above.
CI at 4232b17197: 95 success, 8 skipped, 7 cancelled. The cancelled runs are earlier runs that newer runs replaced. gh pr checks shows no failing check.
Non-blocking follow-ups
- Cost under a dot-named root. The PR body and the
_PROJECT_TREE_SCAN_LIMITcomment say the walk costs "about half a microsecond an entry" and takes a fraction of a second. That holds only for a root that is not under a dot path. Under a dot root (for example a project under~/.kiro/...or.worktrees),under_dotholds for every folder, so every entry pays a Python-levelos.path.realpath. I measured about 37 µs per entry, which is about 7 s for the full 200k budget on every 10 s poll. That time is spent in a worker thread, and it holds the GIL much of that time. This is not a regression: main paid the same per-entry cost with no bound at all. Consider lowering the budget for dot roots, or correcting the comment. - The early stop on rows past the cap is untested.
if len(rows) >= cap: incomplete.add(directory); breakcan be removed and every test still passes, because_project_tree_allotcaps the output anyway. The guard only saves reads; a test that counts pulled entries would pin it. - The test named
test_the_look_ahead_is_drawn_from_the_scan_budgetdoes not fail when the look-ahead is moved outside the budget. A different test catches that mutation, so the test name over-claims.
Problem / Motivation
Goal: Opening the workspace tree on a large project no longer stalls the gateway every 10 seconds.
GET /api/project/treereads a non-git project withos.walktwice, over the whole tree. Only files were capped; every directory row was returned. It then redacted every row and serialized the body ON the event loop. The tree refetches every 10 s while it is open (PierreWorkspaceTreeImpl.tsx), so a big non-git project dir froze every other request the gateway was serving, on every poll. The same walk also answers for a git repository whosels-filesoverflows its 8 MiB cap or 15 s timeout, and for a host with no sandbox backend.Why it matters
Anyone with a large non-git folder (data, logs, a vendored tree) open in the Files tab gets a gateway that pauses every 10 s, for chat, streaming and every other tab, for as long as the tree is open.
Not a goal
FileBrowserRail); its filter stays local, as it already was for files past the cap.What changed (motivation → approach → change)
os.walktwice, unboundedscandirpass, at most_PROJECT_TREE_SCAN_LIMIT(200,000) entries, each folder read once_project_tree_body)truncated= files were droppedtruncated=truncatedDirectoriesis non-empty; every shown folder whose listing was cut is namedThe root cause is that the walk's cost, and the size of the body, grew with the project. Bounding only files left directories free, and the per-row work on the loop made every row block the gateway. So the cost is bounded at three points, all in
src/kiro_crew/dashboard/file_api/project_tree.py(the tree route's owner; its two new constants live with the other module state inhandlers/files.py). The scan budget is shared fairly without cutting a tree that fits: each folder of a depth, and the depths below as one more claimant, is guaranteed half the even split the depth starts with, and a folder may read whatever the claimants after it do not need for that floor, up to the row cap. So one huge folder cannot starve its siblings wherever it sorts, and a small tree with one large folder is still read whole. The rows are shared fairly (_project_tree_allot): folder rows first, shallowest first, but never past half the cap while files remain, so root files likeREADME.mdstay listed; files share their rows round-robin by parent. Nothing is lost silently: a cut folder is named as truncated, never called empty.The dashboard (
website/src/pierre,website/src/pages/chat/FolderPanel.tsx) follows the capped contract. The truncation copy names an item limit (Not shown: limit of 10,000 items reached, in everywebsite/src/i18ncatalog). A truncated folder folded into a chain row byflattenEmptyDirectorieskeeps its badge. A truncated payload with no rows shows the limit, not "No files in this workspace yet". An unreadable-folder dismissal is not forgotten when the cut merely hid the folder. The project-root folder tab sends its search to/api/file-searchwhile the tree is truncated, because the local filter cannot find what was never listed. The tree's own paths that contain the query are merged into those matches (server first, no duplicates), because the server search has its own scan budget and can miss a file the tree shows. A line above the matches says the search also covers items not listed, the tree stays mounted (hidden) behind them so its expanded folders survive clearing the query, and Refresh re-runs that search. Each queued folder is read only if it is still the folder its parent listed (_project_tree_scandir). A link is a symlink or a name-surrogate reparse point (_project_tree_is_link,platform_compat.lstat_is_name_surrogate), so a cloud-files placeholder folder is still walked. A listed folder the walk cannot read, including one whose identitylstatis refused or one swapped after its parent was listed (the root included, as.), is named inunreadableDirectories, never as truncated: truncated means the item limit, and a race on a small project never reached it. Each folder carries its native path so an extended-length Windows root joins correctly. Spec:docs/system-specs/modules/learn-cron-dashboard.md(Project tree).Overlap: #13580 edits the de-duplication of the same response; this diff moves that loop into
_project_tree_bodyunchanged, so it rebases onto it.Backwards compatibility
Compatible: the payload keeps every field and shape.
directoriescan now be shorter than the full skeleton on a project over the cap, and those folders are named throughtruncatedDirectories, which older clients already render.Tests
test/test_project_tree.py: the combined cap on both branches; folder rows first but files keep half; spare rows never evict files that fit; a big tree returns at most the cap; one large folder does not starve its siblings, sorted first or last; a tree inside both limits is read whole beside one large folder, sorted first or last; folders larger than the budget sorted first still leave each small sibling its floor; a folder the budget never reaches is shown and never opened; each folder is read once; a listing that fails part-way is an unreadable row; a listed child whoselstatfails is unreadable and does not make the tree truncated; a folder swapped after its parent was listed is not read and is named unreadable, not truncated (descriptor and by-name branches, rename and symlink swaps); a root swapped for a link is unreadable (.), not truncated; only a name-surrogate reparse tag is a link; the look-ahead is drawn from the scan budget; a root cut by the budget is truncated, not hidden-only; a failedls-filesfalls back to the bounded walk; git folder order matches the walk; redaction and serialization run off the event loop thread.PierreWorkspaceTreeImpl.test.tsx/.realWidget.test.tsx: the chain-row badge (against the real@pierre/treeswidget), the truncated empty state, dismissal kept and forgotten in truncated payloads.FolderPanel.tree.test.tsx: a truncated tree sends the query to the server search, keeps the tree's own matches when the server search misses them (listed once when both name a file), says so above the matches, stays mounted behind them so the expansion survives clearing the query, and Refresh re-runs that search.Manual verification
scripts/capture-files-tree-empty-folder.mjsagainst a fresh build: every frame passes its rendered-surface gates with the new copy.CAPTURE_SCENES=folder-truncated scripts/capture-path-chips.mjsagainst the dev server: the truncated folder tab hands its query to the search, lists the tree's own match first while the search runs, and hides (does not unmount) the tree behind the results. Local gate: the targeted backend tests pass apart from five that fail identically on a cleanorigin/mainworktree on this host (test_file_grepPDF tests, aplatform_compatpeak-RSS test and a facade-loading test); the related frontend tests pass.Screenshots / video
The Files rail with a truncated payload: the workspace notice and the state row under
big/name the item limit and fit the rail.The folder tab on a truncated tree, before a query, while the server search runs (the tree's own match is already listed), and with the server's matches merged in. The line above the matches says the search also covers items not listed:
A truncated payload with no rows shows the limit, not an empty workspace:
A truncated folder folded into a chain row (
pkg / core) keeps its badge:Collapsed: the badge on the closed folder
Related Issues
None filed; reported in review of the dashboard file handlers.
Pattern harvest
Rule candidate: review-prompt
Pattern: a handler on a polled endpoint whose work grows with user data (a walk, a per-row loop, a
json_responseof an uncapped list) on the event loop.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Closes #16647