Repository navigation
fix(dev-fleet): concurrent prune preview + verify merged deleted-branch worktrees - #11450
Conversation
|
Intent: Make the Dev Fleet prune preview usable on a large fleet: scan worktree verdicts concurrently so it stays under the 30s proxy cap, and verify a merged PR whose head branch was deleted so those clean worktrees become purge candidates instead of being withheld forever. |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify four concurrent [DESIGN-REVIEWED] f6502c6 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Still linear under the fixed 30s cap: at the measured 0.52s floor a ~230-worktree fleet 504s again — verify headroom on the largest real fleet. Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: make the Dev Fleet purge preview work on large fleets — stop the 504 and stop withholding merged worktrees whose branch was deleted. FIX.
[FIRST-PRINCIPLES-REVIEWED] f6502c6 |
Opus 4.8 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- test/test_dev_fleet_server_more_coverage.py:1451 -- Putting [GPT-REVIEWED] f6502c6 False positive or not applicable? A repository writer can comment: |
e92c5e4 to
ca02112
Compare
|
self-added: yes
|
ca02112 to
2fc5a7d
Compare
2fc5a7d to
ab53086
Compare
|
|
|
|
|
|
|
Note on the red The shard fails ~180 tests, all in unrelated modules ( Not folding a fix for those pod/win32 tests into this PR (out of scope). This PR's own lanes are green: GPT 5.6, Opus 4.8, Design, First Principles all PASS, and every raised finding is dispositioned. |
…ed-branch worktrees
874e0d4 to
f6502c6
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Approved after a full read of the diff on the current head: the change matches the body and linked issue, the destructive/decision path re-verifies independently and fails closed, tests pin the behaviour, and all review lanes are clean on this head.
…ed-branch worktrees (kirodotdev#11450)
Problem / Motivation
The Dev Fleet "purge" (prune) preview is broken two ways on a large fleet. First, clicking it can return a 504
backend timeout: the preview scans every worktree one at a time, and each scan runs several git calls plus up to two networkedghcalls. Second, even when it does return, it can show zero candidates forever, so the button does nothing.Why it matters
A fleet grows to 100+ worktrees, and that is exactly when an operator needs the purge to reclaim disk. Measured on this host: 111 worktree entries, a serial floor of 0.52s each with just one
ghcall is 57s, past the gateway proxy's 30s cap, so the preview 504s. Of those same 111 entries, 18 were merged and clean yet ALL 18 sat atmerged_unverifiedand none reached themergedbucket, so the preview offered nothing to purge even when it did not time out. The one feature meant to clean up a big fleet stops working precisely on a big fleet.What changed (motivation -> approach -> change)
Two independent root causes, one per defect.
Timeout:
_prune_candidates()inworktree_ops.pyawaited each worktree's verdict in a plainforloop, so wall time scaled linearly with fleet size and crossed the 30s proxy cap. The scan now issues the per-worktree verdicts concurrently throughasyncio.gatherunder a semaphore bounded by the existing_PRUNE_CONCURRENCY(4), the same bound the parallel prune workers use. Output order follows discovery order, not completion order, so the checklist stays stable. The preview path is read-only git (rev-parse, status, rev-list/cherry, merge-base) and never takes_GIT_MUTATION_LOCK, which only the destructive removal path holds -- confirmed, so concurrent verdicts do not race the worktree admin dir. At the shipped 4-way bound the same 111-worktree fleet clears the ~0.52s-each serial floor in roughly 15s, comfortably under the 30s cap; a dedicated higher preview bound is a possible follow-up, not needed to fit the cap.scannednow counts the non-main worktrees directly (len(prunable)), the same value as before whenever exactly one main checkout exists.Zero candidates: a
mergedverdict requires a head OID from_fetch_pr_head_oidinfleet_state.py, which rangh pr view <branch> --json headRefOid,state. Once a PR merges the head branch is deleted (the default here), andgh pr viewthen exits non-zero withno pull requests found for branch, returning None -- so a merged, clean worktree fell tomerged_unverifiedand was withheld permanently. The product's own_pr_query_onealready avoids this by usinggh pr list --head <branch> --state all, which still finds the merged PR and itsheadRefOidafter the branch is gone._fetch_pr_head_oidnow uses that shape and keeps its safety contract intact: (a) it reads no cache, so the verdict is fresh; (b) state and head OID come out of the same JSON row; (c) a non-MERGED state returns None. For the reused-head case -- one branch name carrying both a MERGED PR and an OPEN one -- it does not trustgh pr list's createdAt-descending sort: it reads every PR on the head and returns the merged head OID only when no open PR appears. To keep that "no open PR" conclusion provable at the edge, the query asks for one row beyond a fixed ceiling (_PR_HEAD_LOOKUP_LIMIT); a response that overflows the ceiling means the head carries more PRs than can be examined, so an open one could sit past it, and the lookup fails closed. A recycled branch name can never authorize removing new work.flowchart TD subgraph After["After -- merged PR, head branch deleted"] A2["prune preview"]:::ctx --> B2["gh pr list --head<br/>(survives deletion)"]:::added B2 --> C2["headRefOid found"]:::added --> D2["verdict: merged (ok)"]:::added end subgraph Before["Before -- merged PR, head branch deleted"] A1["prune preview"]:::ctx --> B1["gh pr view branch<br/>(errors: no PR found)"]:::removed B1 --> C1["headRefOid = None"]:::removed --> D1["verdict: merged_unverified<br/>(withheld forever)"]:::removed end classDef added fill:#DCFCE7,stroke:#16A34A; classDef removed fill:#FEE2E2,stroke:#DC2626,stroke-dasharray:4 3; classDef ctx fill:#E0F2FE,stroke:#0284C7;A merged worktree whose branch was deleted now verifies and becomes a purge candidate instead of being withheld forever. In the diagram, green is the added path, red (dashed) is the removed path, and blue is unchanged context.
Tests
In
test_dev_fleet_server_coverage.py,test_prune_candidates_scans_concurrentlypins that more than one verdict is in flight at once and never more than_PRUNE_CONCURRENCY(it fails on the old serial loop), andtest_prune_candidates_output_order_is_deterministicpins that candidates and kept follow discovery order even when verdicts finish out of order. For the merged-blindspot fix, intest_dev_fleet_server_more_coverage.py,test_fetch_pr_head_oid_merged_after_branch_deletedandtest_prunable_merged_deleted_branch_becomes_candidatedrive the real lookup with a scriptedgh pr listand assert verdictmerged(both fail on the oldgh pr viewquery);test_fetch_pr_head_oid_uses_list_not_viewpins the query shape, the same-response state+OID contract, and the--limitsentinel value;test_fetch_pr_head_oid_reused_head_with_open_pr_refuses,test_fetch_pr_head_oid_saturated_batch_fails_closedandtest_fetch_pr_head_oid_exactly_ceiling_is_completepin the reused-head refusal, the overflow fail-closed, and that a full-but-complete window still resolves. The two pre-existing_fetch_pr_head_oidgating tests (intest_dev_fleet_server_more_coverage.pyandtest_dev_fleet_app.py) were updated to the list-response shape.docs/system-specs/modules/dev-fleet.mdrecords the preview's new concurrent, discovery-order, lock-free contract and the deleted-branch head-OID lookup.Manual verification
Verified live against GitHub that the real
_fetch_pr_head_oidresolves the correct head OID for the proven merged PRfix/wave-digest-nested-spawn-10011whose head branch is deleted (gh pr viewreturnsno pull requests found,gh pr listreturns the MERGED PR withheadRefOid=7fead2a...), and returns None for a nonexistent branch. A full gateway/pod run was not used: the change is backend-only with no rendered surface, and the preview logic is covered by the unit tests above plus this live-gh check.Screenshots / video
Why no screenshot: backend-only change; the prune preview API returns the same shape faster and with the previously-withheld merged candidates, with no pixel change to any panel.
Related Issues
no linked issue: dispatched from a measured investigation, not a tracked issue.
Pattern harvest
Rule candidate: review-prompt
Pattern: a destructive-boundary lookup keyed on a branch NAME via
gh pr view <branch>breaks silently once the head branch is deleted; prefergh pr list --head <branch> --state all, which survives deletion, and decide reused-head cases from the full result set rather than a sort position.Other notes
Two things this PR does not do. It does not reclaim disk on its own: 17 stuck worktrees were removed manually on this host while diagnosing (7.32 GB freed); this change fixes why the button offered nothing, not the accumulation itself. And the 30s
_PROXY_TIMEOUTstays a global ceiling on every gateway app-proxy request, not just this one -- the concurrent scan now fits under it comfortably, but another endpoint can outgrow the same cap later, and raising or per-route-scoping that ceiling is out of scope here.