Skip to content

fix(dev-fleet): refuse a worktree removal during its pod start - #19001

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/dev-fleet-removal-excludes-a-pod-start
Oct 11, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/dev-fleet-removal-excludes-a-pod-start

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A Dev Fleet worktree removal and a pod start on the same worktree never overlap, so a removal can no longer delete the checkout a pod is starting from.

_worktree_remove (apps/builtins/dev_fleet/worktree_ops.py:650) refuses to remove a worktree whose pod is active. Its last check is the recheck that refuses with "pod became active again before removal — refusing" (:1423). After that check the removal awaits the lease renewal in its pre-spawn gate and then runs git worktree remove (:1550).

_pod_up (:351) took no lock that the removal holds. It runs kirocrew pod up, and the pod's unit goes active from that separate child process, so nothing in the gateway can observe the moment. A pod that became active after the removal's last check was not seen. The removal reported {"ok": true, "removed": true, "stopped_pod": false}, and the pod kept running from a deleted directory.

Why it matters

The removal deletes the checkout, including the venv and the built dashboard the pod runs from. The pod is left serving from files that no longer exist, and its next import or asset read fails. kirocrew pod up runs for seconds (_pod_up gives it timeout=180), so a start issued just before a removal, a prune or the reaper can land inside the window. The recheck's own comment names this outcome as the one it prevents ("Removing the checkout under a live pod would leave its gateway running from deleted files").

Not a goal

  • A pod started outside the gateway (for example kirocrew pod up typed in a terminal). An in-process lock cannot see it. The existing rechecks still refuse the removal when such a pod is active before the last check.
  • _pod_provision. Its build runs in a background run that outlives the request (fleet_state._PROVISION_INFLIGHT), so a request-scoped lock cannot cover it. Whether a removal should refuse while a provision run is in flight is a separate decision.

What changed (motivation → approach → change)

Root cause. _wt_lock(name) (dev_fleet/worktree_ops.py:2381) was taken only by _worktree_remove (:663) and _rebase (:2392). No pod-start path took it, so nothing ordered a start against a removal.

Change. _pod_up now takes _wt_lock(name) with the same fail-fast refusal the removal and the rebase already use (:374-376), and holds it across the old body, now _pod_up_locked (:384). That covers everything from the guard through the CLI call and the post-start verification. The check and the acquisition are adjacent with no await between them, as in _worktree_remove.

  • A removal or rebase that holds the lock makes the start refuse with "refusing: this worktree is being removed or rebased -- start its pod once that finishes".
  • A removal that arrives mid-start refuses itself. Its message now names both holders: "refusing: a rebase or pod start is in progress for this worktree -- wait for it to finish or abort it first".
  • _pod_up resolves the worktree (repository._find_worktree) before it takes _wt_lock(name), because _WT_LOCKS keeps its rows for good: a name that names no worktree gets the "unknown worktree" error and no row. As on main, every _wt_lock caller resolves the worktree first. The guard under the lock resolves it again, so a removal that lands in between still refuses the start.

Across processes (second commit). _wt_lock lives in each process's memory. An agent's start (dev_fleet/agent_pod_api.py:312) runs in the gateway, while the dashboard removal (dev_fleet/server.py:228) and the reaper (dev_fleet/server.py:138) run in the managed backend, so the lock alone keeps them apart only inside one process. The removal lease is already the one piece of state both processes reach for a removal. The gateway holds it (dev_fleet/live.py:181), and the backend takes it before its pod checks and holds it through git worktree remove (dev_fleet/worktree_ops.py:1147). This commit holds the pod start in that same state for the whole start:

  • acquire_removal_lease (dev_fleet/live.py:202) refuses while a pod start on that worktree is running in the gateway (:217), so a removal that arrives mid-start refuses itself. Its message is "refusing: a Make Live cutover is in progress or this worktree's pod is starting -- retry once it has completed".
  • _pod_up holds live.pod_start_reservation (dev_fleet/live.py:311) on the checkout across _pod_up_locked (dev_fleet/worktree_ops.py:378). In the gateway it refuses the start, with the refusal above, while a removal lease names the checkout, live or lapsed inside its grace barrier. Otherwise it records the start in _POD_STARTS (dev_fleet/live.py:295) until the start ends. In the backend it grants and records nothing, because there the start and every removal share _wt_lock.
  • The check and the record are adjacent with no await between them, on the gateway loop that also serves the lease routes, so a start and a lease request cannot both pass. Both sides compare the checkout path in one spelling (_worktree_key, os.path.normcase(os.path.normpath(...))), and both processes take it from the same git worktree list --porcelain parse (repository._find_worktree).
  • _POD_STARTS holds an entry only while its start runs, and only for a worktree _find_worktree resolved, so it holds at most one entry per existing worktree.
  • Rejected: pod_name_mutex (pod/runtime.py:1024), the cross-process flock the pod CLI already uses. The kirocrew pod up child takes it itself (pod/cli.py:481), so a parent holding it across that child would block the child until the lock times out. A new lock file was rejected for the reason dev_fleet/live.py:126 gives for the lease: a same-uid process in the backend's namespace can replace the file.

Why this option. Rechecking the pod state under the removal's locks right before the git spawn was rejected. The unit goes active from the kirocrew pod up child process at a moment no in-process check can observe, so a recheck only narrows the window: the unit can still go active between the recheck and the spawn, or during git worktree remove. Making the start and the removal take the same lock closes it for every start that goes through the gateway, and reuses the lock and the refusal pattern the module already has.

Every path that starts a pod or removes a worktree.

Path Process Entry point Exclusion
Pod start, dashboard backend api_dev_fleet_pod_up (dev_fleet/http_api.py:447) → _pod_up _wt_lock(name) in _pod_up
Pod start, agent tool gateway handle_pod_up (dev_fleet/agent_pod_api.py:290) → _run_op(_pod_up) _wt_lock(name), and live.pod_start_reservation against the removal lease
Pod restart backend api_dev_fleet_pod_restart (dev_fleet/http_api.py:457) → _pod_restart (dev_fleet/worktree_ops.py:498) → _pod_up _wt_lock(name), for the start half
Removal, dashboard backend api_dev_fleet_worktree_remove (dev_fleet/http_api.py:277) → _worktree_remove _wt_lock(name) in _worktree_remove (dev_fleet/worktree_ops.py:663), and the gateway's removal lease, taken in _worktree_remove_locked (:1147) and refused while an agent's start runs
Removal, prune backend a _worktree_remove call (dev_fleet/worktree_ops.py:2789) same
Removal, reaper backend _worktree_remove(name, force=False, _caller="reaper") (dev_fleet/worktree_ops.py:3020) same

The git worktree remove command is built in one place, _worktree_remove_locked (dev_fleet/worktree_ops.py:1527), and spawned at :1550, under that lock and the lease.

Hot path. A pod start adds one dict lookup and one lock acquire and release on the event loop. In the gateway it also scans the removal leases, at most 32 (_REMOVAL_LEASE_MAX_OUTSTANDING), and records one dict entry for the start. A start on another worktree takes a different lock and is not delayed.

Security: no check is removed or weakened. _pod_checkout_guard, the removal's pod rechecks and the removal lease all run unchanged, and the lease routes keep their backend-only principal check. The change only refuses more combinations: a start during a removal or a rebase, and a removal during a start.

Scope. worktree_ops.py and live.py in apps/builtins/dev_fleet/, a docstring in gateway_routes.py, plus one pin update in test/test_dev_fleet_node_toolchain.py: its source check for the build-env warm-up now names _pod_up_locked, which is where the build env is built. Four _pod_up tests in test/test_dev_fleet_server_coverage.py now also stub the worktree lookup, which _pod_up makes before its guard.

Backwards compatibility

Compatible: a pod start or a removal that does not overlap the other behaves exactly as before (test_control_an_idle_worktree_is_removed_as_before, test_control_a_pod_start_on_another_worktree_is_not_blocked).

Reader: a pod start that overlaps a removal or a rebase of the same worktree now returns {"ok": false, "error": "refusing: this worktree is being removed or rebased -- start its pod once that finishes"} instead of starting. The removal's refusal text, already returned during a rebase, now reads "a rebase or pod start is in progress". A removal whose lease the gateway refuses now reads "refusing: a Make Live cutover is in progress or this worktree's pod is starting -- retry once it has completed", where it named only the cutover. No other route, field or config changes.

Tests

test/test_dev_fleet_removal_pod_start.py drives the real _worktree_remove and the real _pod_up against a scratch git repository under tmp_path, with real worktrees and a real git worktree remove. The seams are the pod backend (runtime.rt, an in-memory set of active pod names) and the kirocrew pod up CLI, which marks the pod active the way its unit does. Every real child the test starts runs with tmp_path as its working directory. Every wait is asyncio.wait_for, bounded at 10 s, or 60 s for a child process's answer.

The two-process tests split the work as production does. The test process plays the gateway: it serves the real removal-lease routes (gateway_routes.register_routes) on 127.0.0.1 and runs the agent's _pod_up. A second Python process plays the backend: it takes the removal lease through the real GatewayPointerBroker and live.removal_lease, as _worktree_remove does, with its own KIROCREW_HOME under tmp_path. That process does not outlive its test: the _backend context closes its stdin, kills it if it is still running and reaps it, however the test ends, a failed, timed-out or cancelled wait included. Two pieces of gateway plumbing are stand-ins, as in test/test_dev_fleet_gateway_routes.py: the app-secret token exchange and the token middleware.

Test On main First commit Both commits
test_a_removal_issued_during_a_pod_start_does_not_delete_the_running_pods_checkout FAIL: AssertionError: {'ok': True, 'pr': None, 'reclaimed_pod_home': False, 'removed': True, ...} (assert True is False) PASS PASS
test_a_pod_start_during_a_removal_is_refused FAIL: AssertionError: {'base_url': 'http://127.0.0.1:1', 'ok': True} (assert True is False) PASS PASS
test_a_backend_removal_lease_is_refused_while_the_gateway_starts_the_pod FAIL FAIL: another process's removal lease answered 'granted' while the gateway was starting the worktree's pod, so the removal would go on to delete its checkout PASS
test_a_gateway_pod_start_is_refused_while_the_backend_holds_the_removal_lease FAIL FAIL: the gateway started the pod while another process held its worktree's removal lease: {'base_url': 'http://127.0.0.1:1', 'ok': True} PASS
test_control_a_pod_started_before_the_last_check_still_refuses_the_removal PASS PASS PASS
test_control_a_pod_start_on_another_worktree_is_not_blocked PASS PASS PASS
test_control_an_idle_worktree_is_removed_as_before PASS PASS PASS
test_a_pod_start_for_an_unknown_worktree_keeps_no_lock_row PASS (main's _pod_up takes no lock) PASS PASS
test_control_a_lease_on_another_worktree_is_granted_during_a_gateway_pod_start PASS PASS PASS
test_control_a_gateway_pod_start_on_another_worktree_runs_during_a_backend_lease PASS PASS PASS
test_control_a_lease_is_granted_once_the_gateway_pod_start_has_finished PASS PASS PASS

On main, with only this test file added: 4 failed, 7 passed, in each of 3 serial runs. On the first commit, with this test file: 2 failed, 9 passed, in each of 3 serial runs; the two failures are the two-process tests. On the first revision of this branch, which took the lock before resolving the name, the lock-row test fails: AssertionError: a pod start for a name with no worktree left a permanent lock row.

The cleanup was checked with a temporary mutation, reverted before the commit: _release waits 1 s for the child to exit before sending the release line. A child holding a granted lease waits for that line, so _release times out in the 4 two-process tests whose lease is granted. Before the cleanup, all 4 of those children were still running after their tests had ended. With it, none was.

Green results, on both commits:

  • this file: 11 of 11 pass, in each of 3 serial runs;
  • the 16 test files that name the pod start, the removal or the worktree lock: 617 passed;
  • the 17 test/test_dev_fleet*.py files: 779 passed.

Manual verification

N/A — the tests run the real removal and pod-start code against a real git repository; there is no UI change.

Related Issues

None filed. Every open pull request that edits a file this change touches, and every one whose file list reached the 100-file limit of the list query, was checked at its current head with git merge-tree, with and without this change. That is 13 pull requests: 1 edits worktree_ops.py, and 12 have long file lists.

  • Conflicts: none. No pull request gains a conflict from this change.
  • Same functions: none edits _pod_up, _pod_restart, _worktree_remove or _wt_lock.
  • Same lock: fix: resolve #12355 - dev-fleet: differentiate the unresolved-channel notice on an #12356 adds _release_channel_create, which takes _wt_lock(name) with the same fail-fast refusal. It merges cleanly with this change. Because a pod start now holds the same lock, that function also refuses while a pod start on the worktree is in flight, and the reverse.

Pattern harvest

Rule candidate: review-prompt
Pattern: two operations on one resource where one destroys what the other starts from must take the same lock for their whole duration; a recheck of state that changes outside the process only narrows the window.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 01:10
@javenciu
javenciu requested a review from buluoray October 11, 2026 01:10
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 8b6effc6959b2c21f1b68bc46c311750322668b7 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 8b6effc6959b2c21f1b68bc46c311750322668b7 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that agent pod starts refused for up to ~120 s, while a dead backend's lapsed lease sits in its grace barrier, return a retryable message, not a dead end.

[DESIGN-REVIEWED] 8b6effc

Description read

[DESCRIPTION-READ] ac0a905fc291cd6fa8fbf98a8ed25b62d571a5eebffd8170b76405aa89686fb7

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 8b6effc6959b2c21f1b68bc46c311750322668b7 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_POD_STARTS keeps a per-path reference count that can never go above 1. A set of paths would do the same job.

Not justified as shipped

  • Item 7 — one consumer, generalized: _POD_STARTS: dict[str, int] with _POD_STARTS.get(key, 0) + 1 and the remaining > 0 branch. The only caller is worktree_ops._pod_up. It enters pod_start_reservation while holding _wt_lock(name), and that lock refuses a second caller straight away (if worktree_lock.locked(): return ...). So one process can never run two reservations on the same worktree, and the count is always 0 or 1.
    Subtraction: change _POD_STARTS to a set[str], then add(key) and discard(key) in the finally. Delete the remaining arithmetic.
    Clears when: the count becomes a set, or the description names a path that reaches pod_start_reservation without _wt_lock(name).

What this change ships

Inventory (8 items) — 7 justified

Intent: stop a Dev Fleet worktree removal from deleting the checkout of a pod that is starting. This is a FIX.

  1. A pod start is refused while its worktree is being removed or rebased — justified
  2. A removal that arrives during a pod start refuses itself — justified
  3. The gateway refuses a removal lease while it runs a pod start on that worktree — justified
  4. A pod start run in the gateway is refused while a removal lease names its checkout — justified
  5. The removal's refusal text now names a pod start as well as a rebase — justified
  6. The lease refusal text now names a pod that is starting — justified
  7. _POD_STARTS counts starts per path — one consumer, generalized
  8. New exported live.pod_start_reservation (one consumer, _pod_up) — justified

[FIRST-PRINCIPLES-REVIEWED] 8b6effc

Description read

[DESCRIPTION-READ] e93a86852ae3ec33286f9118a81780bcc544fabf66ab2464b5bcdf7485386ecb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 8b6effc6959b2c21f1b68bc46c311750322668b7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings block this merge. Two advisory findings remain.

FINDING — src/kiro_crew/apps/builtins/dev_fleet/live.py:49 — acquire_removal_lease now also refuses while _POD_STARTS names the path, and _pod_up now holds _wt_lock(name). The docs were not updated to match. docs/system-specs/modules/dev-fleet.md still says "the gateway refuses a lease while _MAKE_LIVE_LOCK is held or a cutover has committed" (line 1541). Its "Rebase gate" section (lines 433–438) still treats a held _wt_lock as meaning a rebase. The LeaseOutcome docstring still lists only "a cutover is staged or in flight" as the cause of busy. AGENTS.md requires the owning spec to change in the same commit. → Fix: in dev-fleet.md, add the pod-start reservation to the removal-lease section and add _pod_up as a _wt_lock holder in the lock-order/rebase-gate section. Also add pod starts to the LeaseOutcome busy causes.

FINDING — src/kiro_crew/apps/builtins/dev_fleet/worktree_ops.py:162 — _pod_up now holds _wt_lock(name) while pod up runs. A dashboard Rebase on that worktree during that time hits the unchanged _rebase check if lock.locked(): return {"ok": False, "error": "rebase already running for this worktree"}. The user is told a rebase is running when it is actually a pod start. The PR fixed the matching _worktree_remove message but not this one. → Fix: change the _rebase refusal text to cover a pod start, e.g. "a rebase or pod start is in progress for this worktree". That line is in code this PR did not touch.

[OPUS-REVIEWED] 8b6effc

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 8b6effc6959b2c21f1b68bc46c311750322668b7 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8b6effc

@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the finding holds, and the second commit fixes it.

_wt_lock (dev_fleet/worktree_ops.py:2381) is a table in each process's memory. The agent's pod start runs in the gateway: dev_fleet/__init__.py:17-22 installs agent_pod_api on the gateway app, and handle_pod_up (dev_fleet/agent_pod_api.py:290) calls _pod_up at :312. The dashboard removal (dev_fleet/server.py:228 → dev_fleet/http_api.py:277) and the reaper (dev_fleet/server.py:138 → dev_fleet/worktree_ops.py:3020) run in the managed backend. So the first commit kept a start and a removal apart only inside one process: the dashboard's start against a removal, both in the backend, but not an agent's start against a removal.

The second commit makes the exclusion cross-process through the state both processes already share for a removal: the gateway-held removal lease (dev_fleet/live.py:181), which the backend takes before its pod checks and holds through git worktree remove (dev_fleet/worktree_ops.py:1147).

  • acquire_removal_lease (dev_fleet/live.py:202) refuses while a pod start on that worktree is running in the gateway (:217), so a removal that arrives mid-start refuses itself.
  • _pod_up holds live.pod_start_reservation (dev_fleet/live.py:311) on the checkout from its guard through its verification (dev_fleet/worktree_ops.py:378). In the gateway it refuses the start while a removal lease names the checkout, including a lapsed one inside its grace barrier, and otherwise records the start until it ends. In the backend it grants and records nothing, because there the start and every removal share _wt_lock.
  • The check and the record have no await between them and run on the loop that serves the lease routes, so a start and a lease request cannot both pass.

I did not reuse pod_name_mutex (pod/runtime.py:1024), the cross-process flock the pod CLI uses: the kirocrew pod up child takes it itself (pod/cli.py:481), so a parent holding it across that child would block the child. A new lock file is ruled out for the reason dev_fleet/live.py:126 gives for the lease: a same-uid process in the backend's namespace can replace the file.

Tests: two new two-process tests in test/test_dev_fleet_removal_pod_start.py. The test process serves the real lease routes on 127.0.0.1 and runs the agent's _pod_up. A second Python process, with tmp_path as its working directory, takes the lease through the real GatewayPointerBroker, as the backend does. Every wait is under asyncio.wait_for. On the first commit both fail, in each of 3 serial runs (2 failed, 9 passed):

AssertionError: another process's removal lease answered 'granted' while the gateway was starting the worktree's pod, so the removal would go on to delete its checkout
AssertionError: the gateway started the pod while another process held its worktree's removal lease: {'base_url': 'http://127.0.0.1:1', 'ok': True}

With the second commit, the file gives 11 passed in each of 3 serial runs. The 16 test files that name the pod start, the removal or the worktree lock give 617 passed, and the 17 test/test_dev_fleet*.py files give 779 passed. Three controls pin what stays allowed: a lease on another worktree during a start, a start on another worktree during a lease, and a lease once the start has finished.

Still outside this change: a pod started outside Dev Fleet (a kirocrew pod up typed in a terminal), as the description's Not a goal says, and a rebase in the backend against an agent's start in the gateway, since a rebase takes no removal lease.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
@javenciu
javenciu force-pushed the fix/dev-fleet-removal-excludes-a-pod-start branch from 7cd764a to a262698 Compare October 11, 2026 05:45
@javenciu

Copy link
Copy Markdown
Contributor Author

Thanks, the finding holds. The amended second commit fixes it.

Before the fix, _backend returned the child, and each caller released it on one path only. In _lease_during_a_held_start the release came after _first_answer (test/test_dev_fleet_removal_pod_start.py:414 on 7cd764a), so a timeout or a cancellation there skipped it. Where the release ran in a finally, a timeout inside _release still left the child running. The child cannot end itself either: its own 60 s bound (test/test_dev_fleet_removal_pod_start.py:317) cancels the wait, not the executor thread blocked on the stdin read.

The fix registers the cleanup as the child is spawned:

  • _backend is now an async context manager (test/test_dev_fleet_removal_pod_start.py:359-394), with no await between the spawn and its try (:388).
  • Its finally closes the child's stdin (:391), kills the child if it is still running, and reaps it within CHILD_BOUND (:394).
  • All three spawn sites use it: _lease_during_a_held_start at :427, _start_while_another_process_holds_the_lease at :439, and test_control_a_lease_is_granted_once_the_gateway_pod_start_has_finished at :500. The child is reaped however the block ends: a failed assertion, a timeout in _first_answer or _release, or a cancellation.

Witness. A temporary mutation, reverted before the commit, made _release wait 1 s for the child to exit before sending the release line. A child holding a granted lease waits for that line, so _release raised TimeoutError in the 4 two-process tests whose lease is granted. A temporary last test then read /proc/<pid>/stat for every child the mutated tests had started:

before the fix: children 400485 400649 400970 401000 401044; still running after their tests: 400649 400970 401000 401044 (state S, parent: the pytest process)
with the fix:   children 454812 456519 458730 461933 462486; still running after their tests: none

The remaining child in each run (400485, then 454812) belongs to the refused-lease test, which exits by itself.

Both commits are now rebased onto current main (843abe3). The first commit's patch is unchanged: git patch-id --stable gives the same id before and after the rebase. On the new head, this file gives 11 passed in each of 3 serial runs. The 16 test files that name the pod start, the removal or the worktree lock give 617 passed, and the 17 test/test_dev_fleet*.py files give 779 passed. On the first commit the two two-process tests still fail as the description states (2 failed, 9 passed, in each of 3 serial runs), and on main this file gives 4 failed, 7 passed.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026
A worktree removal issued while that worktree's pod was starting could
delete the checkout under the running pod. The removal's last pod check
runs before its lease renewal and the `git worktree remove` spawn, and
the pod's unit goes active from the separate `kirocrew pod up` child,
so a start that landed between the two was never seen: the removal
reported success and the pod kept running from deleted files.

Nothing ordered a pod start against a removal: `_wt_lock(name)` was
taken by the removal and the rebase only. `_pod_up` now takes the same
lock with the same fail-fast refusal, and holds it from its guard
through its verification, so a removal or rebase in flight refuses the
start and a removal arriving mid-start refuses itself. The removal's
refusal names both holders. A pod start on another worktree and the
removal of an idle worktree are unchanged.
…processes

The first commit made a pod start and a removal take the same
`_wt_lock(name)`, but that lock lives in each process's memory. An
agent's pod start (agent_pod_api) runs in the gateway, while the
dashboard removal and the reaper run in the managed backend, so the two
never saw each other: a removal lease was granted while the gateway was
starting the pod, and the gateway started a pod while a removal held
its lease.

The removal lease is already the one piece of state both processes
reach, and the gateway holds it. `acquire_removal_lease` now refuses
while a pod start on that worktree runs in the gateway, and `_pod_up`
holds `live.pod_start_reservation` on the checkout from its guard
through its verification, refusing while a removal lease names the
path. In the backend the reservation grants and records nothing,
because its starts and removals share `_wt_lock`. The removal's
busy-lease refusal now names the pod start as well as a cutover.
@javenciu
javenciu force-pushed the fix/dev-fleet-removal-excludes-a-pod-start branch from a262698 to 8b6effc Compare October 11, 2026 13:45
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This stops a Dev Fleet worktree removal from deleting the checkout of a pod that is starting, and it fixes the root cause: no pod-start path took any lock the removal held, and an agent's start runs in the gateway while every removal runs in the backend. _pod_up now takes _wt_lock(name) (same-process exclusion) and live.pod_start_reservation (cross-process: acquire_removal_lease refuses while a gateway start is recorded, and a start is refused while a removal lease, live or in its grace barrier, names the checkout), with the check and the record adjacent and no await between them on the loop that serves the lease routes. I tried to break it: both sides key on target["path"] from repository._find_worktree through the same _worktree_key normalizer, so the spellings agree; the removal takes its lease before the last "pod became active again" recheck (worktree_ops.py:1147 vs :1423), so a start that finished just before the lease is still seen by that recheck; and in the backend the reservation grants nothing but the start and every removal already share _wt_lock. The two-process tests run the real lease routes against a second Python process and fail on the first commit and on main (author shows 4 failed on main), and the child-cleanup fix keeps the test from leaking processes. The only non-green check is the on-demand macOS decider, whose run was cancelled and whose macOS suite was skipped; Dev Fleet pods are systemd-based, so that suite never exercised this code. The First Principles item (a set[str] would do for _POD_STARTS, since _wt_lock caps it at one) is fair simplification, not a defect, and the author may answer or take it. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 17:34
@iamwhatever
iamwhatever merged commit ecd79ff into kirodotdev:main Oct 11, 2026
103 of 104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants