Skip to content

fix(windows): remove git trees with rmtree_force, not ignore_errors - #19285

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/windows-rmtree-readonly-packs
Oct 12, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/windows-rmtree-readonly-packs

Conversation

@javenciu

@javenciu javenciu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Removing a git checkout on Windows leaves nothing behind, and a rejected install puts the previous checkout back.

git writes everything under .git/objects read-only. On Windows that attribute blocks the unlink, so shutil.rmtree(path, ignore_errors=True) swallows the failure and the checkout stays on disk while the caller carries on as if it were gone. platform_compat.rmtree_force (platform_compat.py#L7567 on main) clears the bit and retries, and #4616 applied it to the registry clone paths. These sites on main still pass ignore_errors:

Why it matters

On Windows, a vault clone, a rejected install, a watcher clone or an aged app source keeps its read-only packs on disk, so app sources and clone directories keep growing, and a rejected install leaves the previous app checkout under its .stale-* name instead of putting it back.

Not a goal

  • Sites whose tree holds nothing git wrote keep ignore_errors=True: copies that drop .git (manager._COPY_IGNORE), scratch and staging directories, and linked worktrees, whose .git is a file.
  • skills.py and skill_runtime/auto_skills.py: open feat(skills): atomically promote safe auto-skill updates #18595 rewrites those removals.
  • Changing rmtree_force itself, or what the registry uninstall logs.
  • Moving the failed-build restore (the install.py#L801 site) onto _restore_moved_aside: its failure is already logged and the stale path stays listed as stranded.

What changed (motivation → approach → change)

Root cause: callers that remove a tree git produced use the stdlib call with ignore_errors, which hides the one failure Windows produces for it. The fix routes each such site through the existing helper, and makes the one restore that can collide use the existing restore path.

  • Removal. Sixteen calls in thirteen functions now go through rmtree_force, or _rmtree_force_settled inside registry_pipeline, where it is already imported and a cancelled caller must not leave the delete running. _delete_clone_if_unowned returns the helper's answer, so True still means deleted. Four import shutil lines that lost their last use go.
  • Restore. _unpoison_rejected_checkout (src/kiro_crew/apps/registry_pipeline/install.py:455-456) removes the rejected tree, then calls _restore_moved_aside, the module's one restore path. When a tree still survives (a file another process holds open), that function renames it to a .partial-* sibling first, so the rename lands. The existing sweep retires the sibling.
  • Rule. docs/system-specs/common/platform-compat.md gets a row: remove a git-produced tree with rmtree_force, and when ignore_errors is still right.

Backwards compatibility

Compatible: a delete that succeeded before succeeds the same way; the difference is a read-only entry being made writable and retried, and a survivor in the restore slot being set aside instead of blocking the restore. One log line changes: a restore_from that no longer exists now logs nothing, because _restore_moved_aside returns early, where the rename raised and logged a warning.

Removes nothing: no key, field, route or capability is removed or renamed.

Tests

In short: on main 640abebd98 the new test files fail 3 of 3 runs with 20 failed, 35 passed; on this branch they pass 3 of 3 with 55 passed. The two existing cases that pinned the old call fail on the previous head and pass on this one, 3 runs each. The 188 neighbour test files pass except one that also fails on main: 14723 passed, 67 skipped, 1 failed. Mutations: 3 of 3 fail only their own rule's cases.

New file test/test_git_tree_removal_windows.py, plus a readonly_unlink_refused simulator in test/windows_sim.py with five self-tests in test/test_windows_sim.py. The simulator makes os.unlink refuse a read-only file the way Windows does. Three existing cases follow the removal onto rmtree_force: test_a_delete_failure_leaves_the_directory in test/test_ai_pr_watchers_coverage.py (_delete_clone_if_unowned), test_registry_workspace_removal_runs_off_the_loop_thread in test/test_apps_uninstall_workspace_off_loop.py:61 (it patched routes.shutil, which apps/routes.py no longer imports, so it now patches platform_compat.rmtree_force and expects the path alone), and test_the_ownership_check_and_delete_are_under_one_held_lock in apps/builtins/auto_improvement/tests/test_reconcile_and_publish.py:263 (it searched the sweep's source for shutil.rmtree(, so it now searches for rmtree_force(). The two cases that fail on the previous head fail 3 of 3 runs there (2 failed) and pass 3 of 3 here (2 passed). Two entries leave test/windows-expected-failures.txt, test_a_missing_head_branch_is_an_error_not_a_silent_base_checkout and test_the_pass_fix_is_exported_to_the_pr_queue: both assert that a clone directory is gone afterwards, through the cleanup in setup_isolated_clone and PRWatcherRegistry._cleanup_clone that this change moves onto rmtree_force, and the Windows job reported both as unexpected passes.

Rule Cases
Removing a git tree clears the read-only bit test_the_retention_sweep_retires_an_aged_checkout, test_a_rejected_fresh_clone_is_removed_and_the_previous_checkout_restored, test_a_refused_vault_clone_leaves_no_checkout, test_a_watchers_clone_is_removed_with_its_objects, test_an_unowned_clone_is_reclaimed_with_its_objects
The restore lands beside a survivor test_the_previous_checkout_is_restored_beside_the_survivor
Every site that removes a git tree uses the helper test_a_git_tree_is_never_removed_by_shutil_rmtree (14 sites), test_the_scan_sees_each_spelling

Red: the new files copied onto main, three runs, 20 failed and 35 passed each time. The five rule-1 cases fail because the read-only tree survives; the restore case fails at the previous checkout's file because the rename collides with the survivor; the 14 site cases fail because the site still calls shutil.rmtree. The scan self-test and the simulator self-tests pass on main.

Mutations, one per rule, over the same files (baseline 55 passed): removing the chmod from the retry hook fails only the five rule-1 cases; disabling the survivor set-aside in _restore_moved_aside fails only the restore case; putting the theme install scratch-clone removal back to shutil.rmtree(..., ignore_errors=True) fails only its site case.

Neighbours: 188 test files: every file under test/ or a builtin app's own tests/ folder that names a changed module (by dotted path or under an alias such as routes_mod) or the registry_pipeline, auto_improvement, design_critique or md_notebook packages, the files that read windows-expected-failures.txt, and every test_platform_compat*, test_md_notebook*, test_apps_registry* and test_theme* file. The first selection (54 files) missed the two cases above. They ran at -n 6 and pass except test_design_critique_ssrf_guard_fail_closed.py::TestGetPlaywrightVersionGate::test_stale_cache_is_not_returned. That case fails the same way when run alone on main 640abebd98 in a clean checkout (1 failed, 7 passed in its file): it runs node and reads the machine's cached Playwright build, and this change does not touch it.

Manual verification

Not run on a real Windows machine. The five rule-1 cases also carry a real parameter, collected only on Windows, so CI's Windows job runs them against read-only files with no simulation. The other cases, and the simulated parameter, run on every OS. The two entries removed from test/windows-expected-failures.txt are checked by the Windows job alone: with the entries gone, the job fails if either case still fails there. There is no UI change.

Related Issues

Fixes #17769

Pattern harvest

Rule candidate: review-prompt
Pattern: shutil.rmtree(path, ignore_errors=True) over a path a git clone or git worktree wrote.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

git writes everything under .git/objects read-only. Windows refuses to
unlink such a file, and shutil.rmtree(..., ignore_errors=True) swallows
the refusal, so the checkout stays on disk while the caller carries on
as if it were gone. The registry clone paths already use rmtree_force;
the sites below still passed ignore_errors.

Route every site that removes a tree git produced through the one helper
(rmtree_force, or _rmtree_force_settled inside registry_pipeline): the
Notes vault clone cleanup, the retention sweep over moved-aside
checkouts, the rejected-install and failed-build rollbacks, the registry
uninstall, the PR watcher clones, the auto-improvement gate snapshot,
the design critique clones and the theme install scratch clone.

A rejected fresh install also restored the previous checkout with a
bare rename, which collides with a replacement that survived the
removal (WinError 183). It now restores through _restore_moved_aside,
which sets a survivor aside as a .partial-* sibling first.

Keep ignore_errors where the tree holds nothing git wrote: copies that
drop .git, scratch and staging directories, linked worktrees.

Fixes kirodotdev#17769
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 05:52
@javenciu
javenciu requested a review from dodgeblaster October 12, 2026 05:52
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 5cd459a4e8d87dbdf13e0010c05c4e603bd21395 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 5cd459a4e8d87dbdf13e0010c05c4e603bd21395 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that a Windows CI run passes the two tests taken off windows-expected-failures.txt and the real read-only parameter.

[DESIGN-REVIEWED] 5cd459a

Description read

[DESCRIPTION-READ] 906ac8b7b84de7f4b8cf527ef1b0db183c43764184656975f801381fbc91afbc

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 5cd459a4e8d87dbdf13e0010c05c4e603bd21395 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

Item 8 removes two Windows expected-failure entries. The comment it deletes blames line endings, a cause this fix never touches.

Not justified as shipped

  • Item 8 — rides along: the deleted pin gives its own cause: "git apply rejects an LF diff against a CRLF-checked-out tree … and \-vs-/ in git output … Porting these to Windows … is its own change". The rmtree_force swap fixes neither problem. The visible description never says why these two now pass (the description is cut off at 8000 bytes).
    Subtraction: put back the two test/windows-expected-failures.txt entries and their comment.
    Clears when: a Windows-shard run on this head shows both test_a_missing_head_branch_is_an_error_not_a_silent_base_checkout and test_the_pass_fix_is_exported_to_the_pr_queue passing, or the entries are restored.

What this change ships

Inventory (8 items) — 7 justified

Intent: on Windows, deleting a git checkout really removes it, and a rejected install restores the previous checkout. This is a FIX.

  1. Vault, watcher, design-critique and theme clones are fully removed on Windows — justified
  2. The retention sweep removes aged checkouts that hold read-only packs — justified
  3. A rejected fresh install restores the previous checkout through the existing _restore_moved_aside — justified
  4. The failed-build rollback and the registry uninstall fully remove pkg_dir/ws_dir — justified
  5. _delete_clone_if_unowned returns False when the tree survives — justified
  6. A missing restore_from no longer logs a warning — justified
  7. A platform-compat rule row and a source-scan pin over 14 sites — justified
  8. Two auto-improvement tests are no longer expected to fail on Windows — rides along: the pin's stated cause (line endings) is untouched

[FIRST-PRINCIPLES-REVIEWED] 5cd459a

Description read

[DESCRIPTION-READ] 28ab9dc6135f53c931503556cb73f7cbc9261ceaf71cedc473c6b1c6b082c59c

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 5cd459a4e8d87dbdf13e0010c05c4e603bd21395 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 5cd459a

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 5cd459a4e8d87dbdf13e0010c05c4e603bd21395 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 5cd459a

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
…d the old call

The registry uninstall and the PR watcher orphan sweep now remove a git
tree through platform_compat.rmtree_force, so two cases that pinned the
old call failed on every backend shard that runs them.
test_registry_workspace_removal_runs_off_the_loop_thread patched
apps.routes.shutil, which that module no longer imports, and
test_the_ownership_check_and_delete_are_under_one_held_lock searched the
sweep's source for shutil.rmtree.

Put the uninstall's double on platform_compat.rmtree_force, which takes
only the path, and look for rmtree_force in the sweep's source. No other
case in the test tree patches or scans for the old call at a site this PR
changed.

Two cases that windows-expected-failures.txt lists as known gaps now pass
on Windows: both assert that a clone directory is gone afterwards, through
the cleanup in setup_isolated_clone and PRWatcherRegistry._cleanup_clone
that this PR moves onto rmtree_force. Their strict xfail reports an
unexpected pass, so drop the two entries and the comment that introduced
only them.

Refs kirodotdev#17769
@javenciu

Copy link
Copy Markdown
Contributor Author

Re: the failed backend jobs on 7522592.

Three of those failures came from this PR's tests. Each is a test that still used the call this PR replaces, so the new commit changes tests and one list only, no production code.

  1. test_registry_workspace_removal_runs_off_the_loop_thread (backend shard 5, Linux and Windows) patched routes.shutil, and apps/routes.py no longer imports shutil, so the patch raised AttributeError. It now patches platform_compat.rmtree_force (test/test_apps_uninstall_workspace_off_loop.py:61) and expects the path alone (:78), which is what apps/routes.py:1860 passes.
  2. test_the_ownership_check_and_delete_are_under_one_held_lock (shard 7, Linux and Windows) searched the source of _delete_clone_if_unowned for shutil.rmtree(; that line is now return rmtree_force(child) (apps/builtins/auto_improvement/backend/pr_watchers.py:1518). It searches for rmtree_force( (apps/builtins/auto_improvement/tests/test_reconcile_and_publish.py:263); the check it makes, that the ownership test and the removal sit in one held lock, is unchanged.
  3. Two cases listed in test/windows-expected-failures.txt now pass on Windows, and the strict xfail reported both as unexpected passes (Windows shard 4): test_a_missing_head_branch_is_an_error_not_a_silent_base_checkout and test_the_pass_fix_is_exported_to_the_pr_queue. Both assert that a clone directory is gone afterwards, through apps/builtins/auto_improvement/backend/pr_watchers.py:273 and :1355, which this PR moves onto rmtree_force. I removed the two entries and the comment that introduced only them. Only the Windows job reads that list, so that edit is checked there, not on Linux.

I looked for other cases that patch or scan for the old call at a site this PR changed, in test/ and in the builtin apps' tests/ folders, and found none. The shutil.rmtree doubles in test/test_external_registry.py and test/test_catalog_inventory.py still intercept the removal, because rmtree_force calls shutil.rmtree itself.

Evidence: the two Linux cases fail on 7522592 and pass on the new head, 3 runs each (2 failed, then 2 passed). The new test files give the same 20 failed, 35 passed on main and 55 passed here, 3 runs each. 188 neighbour test files give 14723 passed, 67 skipped, 1 failed; the one failure, test_design_critique_ssrf_guard_fail_closed.py::TestGetPlaywrightVersionGate::test_stale_cache_is_not_returned, fails the same way on main (640abeb). On the branch merged with current main (0a47021), 205 neighbour test files give 16897 passed, 68 skipped, 1 failed, the same case.

The three test_app_cron_timeout_manifest.py failures on that run (backend shard 1, Linux and Windows) are not from this PR: main failed them until #19280.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This routes every removal of a git-produced tree (vault, watcher, design-critique and theme clones, the retention sweep, the rejected-install and failed-build rollbacks, and the registry uninstall) through platform_compat.rmtree_force, or _rmtree_force_settled inside the registry pipeline. That is the root cause: on Windows, git's read-only packs survive shutil.rmtree(..., ignore_errors=True), and the caller reports success. The rejected-install path now uses the existing _restore_moved_aside, which sets a surviving tree aside before the rename. That closes the WinError 183 restore failure. I checked the swaps that could change behaviour. _delete_clone_if_unowned still returns True only when the tree is gone. The sweep counts a removal only on rmtree_force's answer. A symlinked child is not followed, because the onexc hook retries only unlink/rmdir and chmods only on Windows and never through a link. Every touched module still imports what it calls. The First Principles item about the two entries removed from windows-expected-failures.txt clears on its own condition: those tests were reported as unexpected strict-xfail passes on Windows, and every Windows shard is green on this head. Checks are green and the Opus and GPT lanes have no findings. — crew-pr-reviewer (deep re-check)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows: rmtree(ignore_errors=True) leaves read-only .git packs; checkout residue (follow-up to #4615)

2 participants