Repository navigation
fix(dev-fleet): refuse a worktree removal during its pod start - #19001
iamwhatever merged 2 commits into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that agent pod starts refused for up to ~120 s, while a dead backend's lapsed lease sits in its grace barrier, return a retryable message, not a dead end. [DESIGN-REVIEWED] 8b6effc Description read[DESCRIPTION-READ] ac0a905fc291cd6fa8fbf98a8ed25b62d571a5eebffd8170b76405aa89686fb7 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: stop a Dev Fleet worktree removal from deleting the checkout of a pod that is starting. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 8b6effc Description read[DESCRIPTION-READ] e93a86852ae3ec33286f9118a81780bcc544fabf66ab2464b5bcdf7485386ecb That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings block this merge. Two advisory findings remain. FINDING — src/kiro_crew/apps/builtins/dev_fleet/live.py:49 — FINDING — src/kiro_crew/apps/builtins/dev_fleet/worktree_ops.py:162 — [OPUS-REVIEWED] 8b6effc |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
Thanks, the finding holds, and the second commit fixes it.
The second commit makes the exclusion cross-process through the state both processes already share for a removal: the gateway-held removal lease (
I did not reuse Tests: two new two-process tests in With the second commit, the file gives 11 passed in each of 3 serial runs. The 16 test files that name the pod start, the removal or the worktree lock give 617 passed, and the 17 Still outside this change: a pod started outside Dev Fleet (a |
7cd764a to
a262698
Compare
|
Thanks, the finding holds. The amended second commit fixes it. Before the fix, The fix registers the cleanup as the child is spawned:
Witness. A temporary mutation, reverted before the commit, made The remaining child in each run (400485, then 454812) belongs to the refused-lease test, which exits by itself. Both commits are now rebased onto current main (843abe3). The first commit's patch is unchanged: |
A worktree removal issued while that worktree's pod was starting could delete the checkout under the running pod. The removal's last pod check runs before its lease renewal and the `git worktree remove` spawn, and the pod's unit goes active from the separate `kirocrew pod up` child, so a start that landed between the two was never seen: the removal reported success and the pod kept running from deleted files. Nothing ordered a pod start against a removal: `_wt_lock(name)` was taken by the removal and the rebase only. `_pod_up` now takes the same lock with the same fail-fast refusal, and holds it from its guard through its verification, so a removal or rebase in flight refuses the start and a removal arriving mid-start refuses itself. The removal's refusal names both holders. A pod start on another worktree and the removal of an idle worktree are unchanged.
…processes The first commit made a pod start and a removal take the same `_wt_lock(name)`, but that lock lives in each process's memory. An agent's pod start (agent_pod_api) runs in the gateway, while the dashboard removal and the reaper run in the managed backend, so the two never saw each other: a removal lease was granted while the gateway was starting the pod, and the gateway started a pod while a removal held its lease. The removal lease is already the one piece of state both processes reach, and the gateway holds it. `acquire_removal_lease` now refuses while a pod start on that worktree runs in the gateway, and `_pod_up` holds `live.pod_start_reservation` on the checkout from its guard through its verification, refusing while a removal lease names the path. In the backend the reservation grants and records nothing, because its starts and removals share `_wt_lock`. The removal's busy-lease refusal now names the pod start as well as a cutover.
a262698 to
8b6effc
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This stops a Dev Fleet worktree removal from deleting the checkout of a pod that is starting, and it fixes the root cause: no pod-start path took any lock the removal held, and an agent's start runs in the gateway while every removal runs in the backend. _pod_up now takes _wt_lock(name) (same-process exclusion) and live.pod_start_reservation (cross-process: acquire_removal_lease refuses while a gateway start is recorded, and a start is refused while a removal lease, live or in its grace barrier, names the checkout), with the check and the record adjacent and no await between them on the loop that serves the lease routes. I tried to break it: both sides key on target["path"] from repository._find_worktree through the same _worktree_key normalizer, so the spellings agree; the removal takes its lease before the last "pod became active again" recheck (worktree_ops.py:1147 vs :1423), so a start that finished just before the lease is still seen by that recheck; and in the backend the reservation grants nothing but the start and every removal already share _wt_lock. The two-process tests run the real lease routes against a second Python process and fail on the first commit and on main (author shows 4 failed on main), and the child-cleanup fix keeps the test from leaking processes. The only non-green check is the on-demand macOS decider, whose run was cancelled and whose macOS suite was skipped; Dev Fleet pods are systemd-based, so that suite never exercised this code. The First Principles item (a set[str] would do for _POD_STARTS, since _wt_lock caps it at one) is fair simplification, not a defect, and the author may answer or take it. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A Dev Fleet worktree removal and a pod start on the same worktree never overlap, so a removal can no longer delete the checkout a pod is starting from.
_worktree_remove(apps/builtins/dev_fleet/worktree_ops.py:650) refuses to remove a worktree whose pod is active. Its last check is the recheck that refuses with "pod became active again before removal — refusing" (:1423). After that check the removal awaits the lease renewal in its pre-spawn gate and then runsgit worktree remove(:1550)._pod_up(:351) took no lock that the removal holds. It runskirocrew pod up, and the pod's unit goes active from that separate child process, so nothing in the gateway can observe the moment. A pod that became active after the removal's last check was not seen. The removal reported{"ok": true, "removed": true, "stopped_pod": false}, and the pod kept running from a deleted directory.Why it matters
The removal deletes the checkout, including the venv and the built dashboard the pod runs from. The pod is left serving from files that no longer exist, and its next import or asset read fails.
kirocrew pod upruns for seconds (_pod_upgives ittimeout=180), so a start issued just before a removal, a prune or the reaper can land inside the window. The recheck's own comment names this outcome as the one it prevents ("Removing the checkout under a live pod would leave its gateway running from deleted files").Not a goal
kirocrew pod uptyped in a terminal). An in-process lock cannot see it. The existing rechecks still refuse the removal when such a pod is active before the last check._pod_provision. Its build runs in a background run that outlives the request (fleet_state._PROVISION_INFLIGHT), so a request-scoped lock cannot cover it. Whether a removal should refuse while a provision run is in flight is a separate decision.What changed (motivation → approach → change)
Root cause.
_wt_lock(name)(dev_fleet/worktree_ops.py:2381) was taken only by_worktree_remove(:663) and_rebase(:2392). No pod-start path took it, so nothing ordered a start against a removal.Change.
_pod_upnow takes_wt_lock(name)with the same fail-fast refusal the removal and the rebase already use (:374-376), and holds it across the old body, now_pod_up_locked(:384). That covers everything from the guard through the CLI call and the post-start verification. The check and the acquisition are adjacent with no await between them, as in_worktree_remove._pod_upresolves the worktree (repository._find_worktree) before it takes_wt_lock(name), because_WT_LOCKSkeeps its rows for good: a name that names no worktree gets the "unknown worktree" error and no row. As on main, every_wt_lockcaller resolves the worktree first. The guard under the lock resolves it again, so a removal that lands in between still refuses the start.Across processes (second commit).
_wt_locklives in each process's memory. An agent's start (dev_fleet/agent_pod_api.py:312) runs in the gateway, while the dashboard removal (dev_fleet/server.py:228) and the reaper (dev_fleet/server.py:138) run in the managed backend, so the lock alone keeps them apart only inside one process. The removal lease is already the one piece of state both processes reach for a removal. The gateway holds it (dev_fleet/live.py:181), and the backend takes it before its pod checks and holds it throughgit worktree remove(dev_fleet/worktree_ops.py:1147). This commit holds the pod start in that same state for the whole start:acquire_removal_lease(dev_fleet/live.py:202) refuses while a pod start on that worktree is running in the gateway (:217), so a removal that arrives mid-start refuses itself. Its message is "refusing: a Make Live cutover is in progress or this worktree's pod is starting -- retry once it has completed"._pod_upholdslive.pod_start_reservation(dev_fleet/live.py:311) on the checkout across_pod_up_locked(dev_fleet/worktree_ops.py:378). In the gateway it refuses the start, with the refusal above, while a removal lease names the checkout, live or lapsed inside its grace barrier. Otherwise it records the start in_POD_STARTS(dev_fleet/live.py:295) until the start ends. In the backend it grants and records nothing, because there the start and every removal share_wt_lock._worktree_key,os.path.normcase(os.path.normpath(...))), and both processes take it from the samegit worktree list --porcelainparse (repository._find_worktree)._POD_STARTSholds an entry only while its start runs, and only for a worktree_find_worktreeresolved, so it holds at most one entry per existing worktree.pod_name_mutex(pod/runtime.py:1024), the cross-process flock the pod CLI already uses. Thekirocrew pod upchild takes it itself (pod/cli.py:481), so a parent holding it across that child would block the child until the lock times out. A new lock file was rejected for the reasondev_fleet/live.py:126gives for the lease: a same-uid process in the backend's namespace can replace the file.Why this option. Rechecking the pod state under the removal's locks right before the git spawn was rejected. The unit goes active from the
kirocrew pod upchild process at a moment no in-process check can observe, so a recheck only narrows the window: the unit can still go active between the recheck and the spawn, or duringgit worktree remove. Making the start and the removal take the same lock closes it for every start that goes through the gateway, and reuses the lock and the refusal pattern the module already has.Every path that starts a pod or removes a worktree.
api_dev_fleet_pod_up(dev_fleet/http_api.py:447) →_pod_up_wt_lock(name)in_pod_uphandle_pod_up(dev_fleet/agent_pod_api.py:290) →_run_op(_pod_up)_wt_lock(name), andlive.pod_start_reservationagainst the removal leaseapi_dev_fleet_pod_restart(dev_fleet/http_api.py:457) →_pod_restart(dev_fleet/worktree_ops.py:498) →_pod_up_wt_lock(name), for the start halfapi_dev_fleet_worktree_remove(dev_fleet/http_api.py:277) →_worktree_remove_wt_lock(name)in_worktree_remove(dev_fleet/worktree_ops.py:663), and the gateway's removal lease, taken in_worktree_remove_locked(:1147) and refused while an agent's start runs_worktree_removecall (dev_fleet/worktree_ops.py:2789)_worktree_remove(name, force=False, _caller="reaper")(dev_fleet/worktree_ops.py:3020)The
git worktree removecommand is built in one place,_worktree_remove_locked(dev_fleet/worktree_ops.py:1527), and spawned at:1550, under that lock and the lease.Hot path. A pod start adds one dict lookup and one lock acquire and release on the event loop. In the gateway it also scans the removal leases, at most 32 (
_REMOVAL_LEASE_MAX_OUTSTANDING), and records one dict entry for the start. A start on another worktree takes a different lock and is not delayed.Security: no check is removed or weakened.
_pod_checkout_guard, the removal's pod rechecks and the removal lease all run unchanged, and the lease routes keep their backend-only principal check. The change only refuses more combinations: a start during a removal or a rebase, and a removal during a start.Scope.
worktree_ops.pyandlive.pyinapps/builtins/dev_fleet/, a docstring ingateway_routes.py, plus one pin update intest/test_dev_fleet_node_toolchain.py: its source check for the build-env warm-up now names_pod_up_locked, which is where the build env is built. Four_pod_uptests intest/test_dev_fleet_server_coverage.pynow also stub the worktree lookup, which_pod_upmakes before its guard.Backwards compatibility
Compatible: a pod start or a removal that does not overlap the other behaves exactly as before (
test_control_an_idle_worktree_is_removed_as_before,test_control_a_pod_start_on_another_worktree_is_not_blocked).Reader: a pod start that overlaps a removal or a rebase of the same worktree now returns
{"ok": false, "error": "refusing: this worktree is being removed or rebased -- start its pod once that finishes"}instead of starting. The removal's refusal text, already returned during a rebase, now reads "a rebase or pod start is in progress". A removal whose lease the gateway refuses now reads "refusing: a Make Live cutover is in progress or this worktree's pod is starting -- retry once it has completed", where it named only the cutover. No other route, field or config changes.Tests
test/test_dev_fleet_removal_pod_start.pydrives the real_worktree_removeand the real_pod_upagainst a scratch git repository undertmp_path, with real worktrees and a realgit worktree remove. The seams are the pod backend (runtime.rt, an in-memory set of active pod names) and thekirocrew pod upCLI, which marks the pod active the way its unit does. Every real child the test starts runs withtmp_pathas its working directory. Every wait isasyncio.wait_for, bounded at 10 s, or 60 s for a child process's answer.The two-process tests split the work as production does. The test process plays the gateway: it serves the real removal-lease routes (
gateway_routes.register_routes) on 127.0.0.1 and runs the agent's_pod_up. A second Python process plays the backend: it takes the removal lease through the realGatewayPointerBrokerandlive.removal_lease, as_worktree_removedoes, with its ownKIROCREW_HOMEundertmp_path. That process does not outlive its test: the_backendcontext closes its stdin, kills it if it is still running and reaps it, however the test ends, a failed, timed-out or cancelled wait included. Two pieces of gateway plumbing are stand-ins, as intest/test_dev_fleet_gateway_routes.py: the app-secret token exchange and the token middleware.test_a_removal_issued_during_a_pod_start_does_not_delete_the_running_pods_checkoutAssertionError: {'ok': True, 'pr': None, 'reclaimed_pod_home': False, 'removed': True, ...}(assert True is False)test_a_pod_start_during_a_removal_is_refusedAssertionError: {'base_url': 'http://127.0.0.1:1', 'ok': True}(assert True is False)test_a_backend_removal_lease_is_refused_while_the_gateway_starts_the_podanother process's removal lease answered 'granted' while the gateway was starting the worktree's pod, so the removal would go on to delete its checkouttest_a_gateway_pod_start_is_refused_while_the_backend_holds_the_removal_leasethe gateway started the pod while another process held its worktree's removal lease: {'base_url': 'http://127.0.0.1:1', 'ok': True}test_control_a_pod_started_before_the_last_check_still_refuses_the_removaltest_control_a_pod_start_on_another_worktree_is_not_blockedtest_control_an_idle_worktree_is_removed_as_beforetest_a_pod_start_for_an_unknown_worktree_keeps_no_lock_row_pod_uptakes no lock)test_control_a_lease_on_another_worktree_is_granted_during_a_gateway_pod_starttest_control_a_gateway_pod_start_on_another_worktree_runs_during_a_backend_leasetest_control_a_lease_is_granted_once_the_gateway_pod_start_has_finishedOn main, with only this test file added: 4 failed, 7 passed, in each of 3 serial runs. On the first commit, with this test file: 2 failed, 9 passed, in each of 3 serial runs; the two failures are the two-process tests. On the first revision of this branch, which took the lock before resolving the name, the lock-row test fails:
AssertionError: a pod start for a name with no worktree left a permanent lock row.The cleanup was checked with a temporary mutation, reverted before the commit:
_releasewaits 1 s for the child to exit before sending the release line. A child holding a granted lease waits for that line, so_releasetimes out in the 4 two-process tests whose lease is granted. Before the cleanup, all 4 of those children were still running after their tests had ended. With it, none was.Green results, on both commits:
test/test_dev_fleet*.pyfiles: 779 passed.Manual verification
N/A — the tests run the real removal and pod-start code against a real git repository; there is no UI change.
Related Issues
None filed. Every open pull request that edits a file this change touches, and every one whose file list reached the 100-file limit of the list query, was checked at its current head with
git merge-tree, with and without this change. That is 13 pull requests: 1 editsworktree_ops.py, and 12 have long file lists._pod_up,_pod_restart,_worktree_removeor_wt_lock._release_channel_create, which takes_wt_lock(name)with the same fail-fast refusal. It merges cleanly with this change. Because a pod start now holds the same lock, that function also refuses while a pod start on the worktree is in flight, and the reverse.Pattern harvest
Rule candidate: review-prompt
Pattern: two operations on one resource where one destroys what the other starts from must take the same lock for their whole duration; a recheck of state that changes outside the process only narrows the window.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)