Repository navigation
fix(windows): remove git trees with rmtree_force, not ignore_errors - #19285
Conversation
git writes everything under .git/objects read-only. Windows refuses to unlink such a file, and shutil.rmtree(..., ignore_errors=True) swallows the refusal, so the checkout stays on disk while the caller carries on as if it were gone. The registry clone paths already use rmtree_force; the sites below still passed ignore_errors. Route every site that removes a tree git produced through the one helper (rmtree_force, or _rmtree_force_settled inside registry_pipeline): the Notes vault clone cleanup, the retention sweep over moved-aside checkouts, the rejected-install and failed-build rollbacks, the registry uninstall, the PR watcher clones, the auto-improvement gate snapshot, the design critique clones and the theme install scratch clone. A rejected fresh install also restored the previous checkout with a bare rename, which collides with a replacement that survived the removal (WinError 183). It now restores through _restore_moved_aside, which sets a survivor aside as a .partial-* sibling first. Keep ignore_errors where the tree holds nothing git wrote: copies that drop .git, scratch and staging directories, linked worktrees. Fixes kirodotdev#17769
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that a Windows CI run passes the two tests taken off [DESIGN-REVIEWED] 5cd459a Description read[DESCRIPTION-READ] 906ac8b7b84de7f4b8cf527ef1b0db183c43764184656975f801381fbc91afbc That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS Item 8 removes two Windows expected-failure entries. The comment it deletes blames line endings, a cause this fix never touches. Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: on Windows, deleting a git checkout really removes it, and a rejected install restores the previous checkout. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 5cd459a Description read[DESCRIPTION-READ] 28ab9dc6135f53c931503556cb73f7cbc9261ceaf71cedc473c6b1c6b082c59c That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
…d the old call The registry uninstall and the PR watcher orphan sweep now remove a git tree through platform_compat.rmtree_force, so two cases that pinned the old call failed on every backend shard that runs them. test_registry_workspace_removal_runs_off_the_loop_thread patched apps.routes.shutil, which that module no longer imports, and test_the_ownership_check_and_delete_are_under_one_held_lock searched the sweep's source for shutil.rmtree. Put the uninstall's double on platform_compat.rmtree_force, which takes only the path, and look for rmtree_force in the sweep's source. No other case in the test tree patches or scans for the old call at a site this PR changed. Two cases that windows-expected-failures.txt lists as known gaps now pass on Windows: both assert that a clone directory is gone afterwards, through the cleanup in setup_isolated_clone and PRWatcherRegistry._cleanup_clone that this PR moves onto rmtree_force. Their strict xfail reports an unexpected pass, so drop the two entries and the comment that introduced only them. Refs kirodotdev#17769
|
Re: the failed backend jobs on 7522592. Three of those failures came from this PR's tests. Each is a test that still used the call this PR replaces, so the new commit changes tests and one list only, no production code.
I looked for other cases that patch or scan for the old call at a site this PR changed, in Evidence: the two Linux cases fail on 7522592 and pass on the new head, 3 runs each (2 failed, then 2 passed). The new test files give the same 20 failed, 35 passed on main and 55 passed here, 3 runs each. 188 neighbour test files give 14723 passed, 67 skipped, 1 failed; the one failure, The three |
iamwhatever
left a comment
There was a problem hiding this comment.
This routes every removal of a git-produced tree (vault, watcher, design-critique and theme clones, the retention sweep, the rejected-install and failed-build rollbacks, and the registry uninstall) through platform_compat.rmtree_force, or _rmtree_force_settled inside the registry pipeline. That is the root cause: on Windows, git's read-only packs survive shutil.rmtree(..., ignore_errors=True), and the caller reports success. The rejected-install path now uses the existing _restore_moved_aside, which sets a surviving tree aside before the rename. That closes the WinError 183 restore failure. I checked the swaps that could change behaviour. _delete_clone_if_unowned still returns True only when the tree is gone. The sweep counts a removal only on rmtree_force's answer. A symlinked child is not followed, because the onexc hook retries only unlink/rmdir and chmods only on Windows and never through a link. Every touched module still imports what it calls. The First Principles item about the two entries removed from windows-expected-failures.txt clears on its own condition: those tests were reported as unexpected strict-xfail passes on Windows, and every Windows shard is green on this head. Checks are green and the Opus and GPT lanes have no findings. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Removing a git checkout on Windows leaves nothing behind, and a rejected install puts the previous checkout back.
git writes everything under
.git/objectsread-only. On Windows that attribute blocks the unlink, soshutil.rmtree(path, ignore_errors=True)swallows the failure and the checkout stays on disk while the caller carries on as if it were gone.platform_compat.rmtree_force(platform_compat.py#L7567 on main) clears the bit and retries, and #4616 applied it to the registry clone paths. These sites on main still passignore_errors:api_vault_clone(md_notebook/server.py#L1628)._sweep_stale_checkouts_sync(recovery.py#L356). An aged checkout that holds read-only packs stays on disk at every sweep._unpoison_rejected_checkout(install.py#L450). The surviving replacement then occupies the slot the restore rename targets (install.py#L453), so the previous checkout is not put back (WinError 183on Windows).Why it matters
On Windows, a vault clone, a rejected install, a watcher clone or an aged app source keeps its read-only packs on disk, so app sources and clone directories keep growing, and a rejected install leaves the previous app checkout under its
.stale-*name instead of putting it back.Not a goal
ignore_errors=True: copies that drop.git(manager._COPY_IGNORE), scratch and staging directories, and linked worktrees, whose.gitis a file.skills.pyandskill_runtime/auto_skills.py: open feat(skills): atomically promote safe auto-skill updates #18595 rewrites those removals.rmtree_forceitself, or what the registry uninstall logs.install.py#L801site) onto_restore_moved_aside: its failure is already logged and the stale path stays listed as stranded.What changed (motivation → approach → change)
Root cause: callers that remove a tree git produced use the stdlib call with
ignore_errors, which hides the one failure Windows produces for it. The fix routes each such site through the existing helper, and makes the one restore that can collide use the existing restore path.rmtree_force, or_rmtree_force_settledinsideregistry_pipeline, where it is already imported and a cancelled caller must not leave the delete running._delete_clone_if_unownedreturns the helper's answer, soTruestill means deleted. Fourimport shutillines that lost their last use go._unpoison_rejected_checkout(src/kiro_crew/apps/registry_pipeline/install.py:455-456) removes the rejected tree, then calls_restore_moved_aside, the module's one restore path. When a tree still survives (a file another process holds open), that function renames it to a.partial-*sibling first, so the rename lands. The existing sweep retires the sibling.docs/system-specs/common/platform-compat.mdgets a row: remove a git-produced tree withrmtree_force, and whenignore_errorsis still right.Backwards compatibility
Compatible: a delete that succeeded before succeeds the same way; the difference is a read-only entry being made writable and retried, and a survivor in the restore slot being set aside instead of blocking the restore. One log line changes: a
restore_fromthat no longer exists now logs nothing, because_restore_moved_asidereturns early, where the rename raised and logged a warning.Removes nothing: no key, field, route or capability is removed or renamed.
Tests
In short: on main
640abebd98the new test files fail 3 of 3 runs with 20 failed, 35 passed; on this branch they pass 3 of 3 with 55 passed. The two existing cases that pinned the old call fail on the previous head and pass on this one, 3 runs each. The 188 neighbour test files pass except one that also fails on main: 14723 passed, 67 skipped, 1 failed. Mutations: 3 of 3 fail only their own rule's cases.New file
test/test_git_tree_removal_windows.py, plus areadonly_unlink_refusedsimulator intest/windows_sim.pywith five self-tests intest/test_windows_sim.py. The simulator makesos.unlinkrefuse a read-only file the way Windows does. Three existing cases follow the removal ontormtree_force:test_a_delete_failure_leaves_the_directoryintest/test_ai_pr_watchers_coverage.py(_delete_clone_if_unowned),test_registry_workspace_removal_runs_off_the_loop_threadintest/test_apps_uninstall_workspace_off_loop.py:61(it patchedroutes.shutil, whichapps/routes.pyno longer imports, so it now patchesplatform_compat.rmtree_forceand expects the path alone), andtest_the_ownership_check_and_delete_are_under_one_held_lockinapps/builtins/auto_improvement/tests/test_reconcile_and_publish.py:263(it searched the sweep's source forshutil.rmtree(, so it now searches forrmtree_force(). The two cases that fail on the previous head fail 3 of 3 runs there (2 failed) and pass 3 of 3 here (2 passed). Two entries leavetest/windows-expected-failures.txt,test_a_missing_head_branch_is_an_error_not_a_silent_base_checkoutandtest_the_pass_fix_is_exported_to_the_pr_queue: both assert that a clone directory is gone afterwards, through the cleanup insetup_isolated_cloneandPRWatcherRegistry._cleanup_clonethat this change moves ontormtree_force, and the Windows job reported both as unexpected passes.test_the_retention_sweep_retires_an_aged_checkout,test_a_rejected_fresh_clone_is_removed_and_the_previous_checkout_restored,test_a_refused_vault_clone_leaves_no_checkout,test_a_watchers_clone_is_removed_with_its_objects,test_an_unowned_clone_is_reclaimed_with_its_objectstest_the_previous_checkout_is_restored_beside_the_survivortest_a_git_tree_is_never_removed_by_shutil_rmtree(14 sites),test_the_scan_sees_each_spellingRed: the new files copied onto main, three runs, 20 failed and 35 passed each time. The five rule-1 cases fail because the read-only tree survives; the restore case fails at the previous checkout's file because the rename collides with the survivor; the 14 site cases fail because the site still calls
shutil.rmtree. The scan self-test and the simulator self-tests pass on main.Mutations, one per rule, over the same files (baseline 55 passed): removing the
chmodfrom the retry hook fails only the five rule-1 cases; disabling the survivor set-aside in_restore_moved_asidefails only the restore case; putting the theme install scratch-clone removal back toshutil.rmtree(..., ignore_errors=True)fails only its site case.Neighbours: 188 test files: every file under
test/or a builtin app's owntests/folder that names a changed module (by dotted path or under an alias such asroutes_mod) or theregistry_pipeline,auto_improvement,design_critiqueormd_notebookpackages, the files that readwindows-expected-failures.txt, and everytest_platform_compat*,test_md_notebook*,test_apps_registry*andtest_theme*file. The first selection (54 files) missed the two cases above. They ran at-n 6and pass excepttest_design_critique_ssrf_guard_fail_closed.py::TestGetPlaywrightVersionGate::test_stale_cache_is_not_returned. That case fails the same way when run alone on main640abebd98in a clean checkout (1 failed, 7 passed in its file): it runsnodeand reads the machine's cached Playwright build, and this change does not touch it.Manual verification
Not run on a real Windows machine. The five rule-1 cases also carry a
realparameter, collected only on Windows, so CI's Windows job runs them against read-only files with no simulation. The other cases, and the simulated parameter, run on every OS. The two entries removed fromtest/windows-expected-failures.txtare checked by the Windows job alone: with the entries gone, the job fails if either case still fails there. There is no UI change.Related Issues
Fixes #17769
rmtree_forceto the registry clone paths; this extends it to the sites above.skills.pyandauto_skills.py; neither file is touched here.Pattern harvest
Rule candidate: review-prompt
Pattern:
shutil.rmtree(path, ignore_errors=True)over a path agit cloneorgit worktreewrote.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.