Skip to content

issue #7783: speed up explorer filter and large project tree open - #7784

Open
mattcasters wants to merge 1 commit into
apache:mainfrom
mattcasters:issue-7783
Open

issue #7783: speed up explorer filter and large project tree open#7784
mattcasters wants to merge 1 commit into
apache:mainfrom
mattcasters:issue-7783

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

Summary

Fixes #7783: the File Explorer filter became unusable with a few hundred–thousand files (multi-second freezes per keystroke). The same tree rebuild path also made opening a large flat project root expensive.

Two related problems were fixed:

1. Filter keystrokes (original issue)

  • Debounce filter apply (~250 ms; Enter applies immediately)
  • In-memory ExplorerTreeModel: one full VFS walk, then subsequent filter keystrokes match/render from memory only
  • Filter rebuilds do not call IExplorerRefreshListener.beforeRefresh() (no git status per character)
  • Removed recursive hasMatchingDescendant VFS probes (the O(n×depth) double-walk)
  • Explorer filter uses non-fuzzy SearchMatcher (substring/word only)

2. Project open / tree paint cost (same root causes)

  • HopFileTypeBase.isHandledBy: extension match is pure string — no HopVfs.getFileObject per plugin per file
  • Extension/basename maps in ExplorerPerspective for O(1) type resolution; folders take FolderFileType via known isFolder flag
  • FolderFileType / GenericFileType: no VFS isFolder/isFile probes during type resolution
  • Same string-extension approach for Archive, Excel, Parquet, NoExtension types
  • Single-pass folder listing (classify once, then sort folders/files)
  • Git paint path normalization without VFS (when git plugin is enabled)

Measured on a ~10k-file project root (git disabled): open dropped from ~10s to ~6s (~40%). Filter typing is interactive after the first index build.

Reviewer notes

Please focus review on these areas:

  1. ExplorerPerspective filter path

    • scheduleFilterApply / applyPendingFilter / applyFilterFromModel / ensureFullTreeModel
    • Confirm F5 still refreshes git status while filtering, but normal keystrokes do not
    • Confirm clearing the filter restores expand state via saveTreeState / restoreTreeState
    • Index invalidation: treeModel.clear() on structural refresh(), show-hidden, create/delete/rename (via refresh())
  2. getFileType(path, isFolder) + indexFileTypes()

    • First matching extension wins (plugin order). Flag any type that used non-extension logic that no longer maps cleanly
    • FolderFileType.isHandledBy now always returns false — callers must pass isFolder=true or select the type explicitly. Search for other callers if concerned
    • GenericFileType is the catch-all for unmatched files without VFS
  3. HopFileTypeBase.extractExtension / extractBaseName

    • Hidden files like .gitignore intentionally have no “extension”
    • Compound filters (*.xls;*.xlsx) are split on ;
  4. Git GitGuiPlugin.getAbsoluteFilename

    • Keys must stay consistent between refreshChangedFiles and filePainted (both use the same string normalizer now)
    • Worth a quick smoke test: git project with staged/modified files still get color in the explorer
  5. Out of scope / known remaining cost

    • Creating many SWT TreeItems for a flat 10k root is still inherently heavy; this PR does not virtualize the tree
    • First filter after open may still walk the full tree once to fill the index (by design)

Test plan

  • Unit: HopFileTypeBaseTest, ExplorerTreeModelTest
  • Manual: project with ~1k+ files — type in explorer filter; characters should not freeze UI; matches under nested folders appear after debounce
  • Manual: clear filter — previous expand/collapse state restored
  • Manual: F5 while filtering — tree updates and git colors (if git enabled) refresh
  • Manual: open large flat project — tree populates without multi-10s freezes
  • Manual: open pipeline/workflow/text files from explorer still uses correct handlers/icons
  • Manual (git): modified/staged/ignored files still colored after explorer refresh

Fixes #7783

Debounce the file explorer filter, rebuild filtered trees from an
in-memory index without re-running git status per keystroke, and remove
the recursive hasMatchingDescendant VFS probes. Make file-type detection
string/extension-map based so opening a project with thousands of files
no longer performs a VFS open per plugin per entry.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: explorer tree is unusably slow filtering files

1 participant