Skip to content

fix(dev-fleet): concurrent prune preview + verify merged deleted-branch worktrees - #11450

Merged
iamwhatever merged 1 commit into
mainfrom
fix/prune-preview-timeout-and-merged-blindspot
Sep 17, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/prune-preview-timeout-and-merged-blindspot

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

The Dev Fleet "purge" (prune) preview is broken two ways on a large fleet. First, clicking it can return a 504 backend timeout: the preview scans every worktree one at a time, and each scan runs several git calls plus up to two networked gh calls. Second, even when it does return, it can show zero candidates forever, so the button does nothing.

Why it matters

A fleet grows to 100+ worktrees, and that is exactly when an operator needs the purge to reclaim disk. Measured on this host: 111 worktree entries, a serial floor of 0.52s each with just one gh call is 57s, past the gateway proxy's 30s cap, so the preview 504s. Of those same 111 entries, 18 were merged and clean yet ALL 18 sat at merged_unverified and none reached the merged bucket, so the preview offered nothing to purge even when it did not time out. The one feature meant to clean up a big fleet stops working precisely on a big fleet.

What changed (motivation -> approach -> change)

Two independent root causes, one per defect.

Timeout: _prune_candidates() in worktree_ops.py awaited each worktree's verdict in a plain for loop, so wall time scaled linearly with fleet size and crossed the 30s proxy cap. The scan now issues the per-worktree verdicts concurrently through asyncio.gather under a semaphore bounded by the existing _PRUNE_CONCURRENCY (4), the same bound the parallel prune workers use. Output order follows discovery order, not completion order, so the checklist stays stable. The preview path is read-only git (rev-parse, status, rev-list/cherry, merge-base) and never takes _GIT_MUTATION_LOCK, which only the destructive removal path holds -- confirmed, so concurrent verdicts do not race the worktree admin dir. At the shipped 4-way bound the same 111-worktree fleet clears the ~0.52s-each serial floor in roughly 15s, comfortably under the 30s cap; a dedicated higher preview bound is a possible follow-up, not needed to fit the cap. scanned now counts the non-main worktrees directly (len(prunable)), the same value as before whenever exactly one main checkout exists.

Zero candidates: a merged verdict requires a head OID from _fetch_pr_head_oid in fleet_state.py, which ran gh pr view <branch> --json headRefOid,state. Once a PR merges the head branch is deleted (the default here), and gh pr view then exits non-zero with no pull requests found for branch, returning None -- so a merged, clean worktree fell to merged_unverified and was withheld permanently. The product's own _pr_query_one already avoids this by using gh pr list --head <branch> --state all, which still finds the merged PR and its headRefOid after the branch is gone. _fetch_pr_head_oid now uses that shape and keeps its safety contract intact: (a) it reads no cache, so the verdict is fresh; (b) state and head OID come out of the same JSON row; (c) a non-MERGED state returns None. For the reused-head case -- one branch name carrying both a MERGED PR and an OPEN one -- it does not trust gh pr list's createdAt-descending sort: it reads every PR on the head and returns the merged head OID only when no open PR appears. To keep that "no open PR" conclusion provable at the edge, the query asks for one row beyond a fixed ceiling (_PR_HEAD_LOOKUP_LIMIT); a response that overflows the ceiling means the head carries more PRs than can be examined, so an open one could sit past it, and the lookup fails closed. A recycled branch name can never authorize removing new work.

flowchart TD
    subgraph After["After -- merged PR, head branch deleted"]
        A2["prune preview"]:::ctx --> B2["gh pr list --head<br/>(survives deletion)"]:::added
        B2 --> C2["headRefOid found"]:::added --> D2["verdict: merged (ok)"]:::added
    end
    subgraph Before["Before -- merged PR, head branch deleted"]
        A1["prune preview"]:::ctx --> B1["gh pr view branch<br/>(errors: no PR found)"]:::removed
        B1 --> C1["headRefOid = None"]:::removed --> D1["verdict: merged_unverified<br/>(withheld forever)"]:::removed
    end
    classDef added fill:#DCFCE7,stroke:#16A34A;
    classDef removed fill:#FEE2E2,stroke:#DC2626,stroke-dasharray:4 3;
    classDef ctx fill:#E0F2FE,stroke:#0284C7;
Loading

A merged worktree whose branch was deleted now verifies and becomes a purge candidate instead of being withheld forever. In the diagram, green is the added path, red (dashed) is the removed path, and blue is unchanged context.

Tests

In test_dev_fleet_server_coverage.py, test_prune_candidates_scans_concurrently pins that more than one verdict is in flight at once and never more than _PRUNE_CONCURRENCY (it fails on the old serial loop), and test_prune_candidates_output_order_is_deterministic pins that candidates and kept follow discovery order even when verdicts finish out of order. For the merged-blindspot fix, in test_dev_fleet_server_more_coverage.py, test_fetch_pr_head_oid_merged_after_branch_deleted and test_prunable_merged_deleted_branch_becomes_candidate drive the real lookup with a scripted gh pr list and assert verdict merged (both fail on the old gh pr view query); test_fetch_pr_head_oid_uses_list_not_view pins the query shape, the same-response state+OID contract, and the --limit sentinel value; test_fetch_pr_head_oid_reused_head_with_open_pr_refuses, test_fetch_pr_head_oid_saturated_batch_fails_closed and test_fetch_pr_head_oid_exactly_ceiling_is_complete pin the reused-head refusal, the overflow fail-closed, and that a full-but-complete window still resolves. The two pre-existing _fetch_pr_head_oid gating tests (in test_dev_fleet_server_more_coverage.py and test_dev_fleet_app.py) were updated to the list-response shape. docs/system-specs/modules/dev-fleet.md records the preview's new concurrent, discovery-order, lock-free contract and the deleted-branch head-OID lookup.

Manual verification

Verified live against GitHub that the real _fetch_pr_head_oid resolves the correct head OID for the proven merged PR fix/wave-digest-nested-spawn-10011 whose head branch is deleted (gh pr view returns no pull requests found, gh pr list returns the MERGED PR with headRefOid=7fead2a...), and returns None for a nonexistent branch. A full gateway/pod run was not used: the change is backend-only with no rendered surface, and the preview logic is covered by the unit tests above plus this live-gh check.

Screenshots / video

Why no screenshot: backend-only change; the prune preview API returns the same shape faster and with the previously-withheld merged candidates, with no pixel change to any panel.

Related Issues

no linked issue: dispatched from a measured investigation, not a tracked issue.

Pattern harvest

Rule candidate: review-prompt
Pattern: a destructive-boundary lookup keyed on a branch NAME via gh pr view <branch> breaks silently once the head branch is deleted; prefer gh pr list --head <branch> --state all, which survives deletion, and decide reused-head cases from the full result set rather than a sort position.

Other notes

Two things this PR does not do. It does not reclaim disk on its own: 17 stuck worktrees were removed manually on this host while diagnosing (7.32 GB freed); this change fixes why the button offered nothing, not the accumulation itself. And the 30s _PROXY_TIMEOUT stays a global ceiling on every gateway app-proxy request, not just this one -- the concurrent scan now fits under it comfortably, but another endpoint can outgrow the same cap later, and raising or per-route-scoping that ceiling is out of scope here.

@chenmingwei23
chenmingwei23 requested review from a team and Zedmor September 17, 2026 04:09
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Intent: Make the Dev Fleet prune preview usable on a large fleet: scan worktree verdicts concurrently so it stays under the 30s proxy cap, and verify a merged PR whose head branch was deleted so those clean worktrees become purge candidates instead of being withheld forever.
Not a goal: Reclaiming disk on its own, raising or per-route-scoping the global 30s proxy ceiling, or changing any prune removal-safety gate.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of f6502c6a8d5c2567f93e0eba17411c44ec82239e — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify four concurrent gh lookups on a 100+ fleet don't trip GitHub secondary rate limits, which fail verdicts closed back to merged_unverified.

[DESIGN-REVIEWED] f6502c6

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of f6502c6a8d5c2567f93e0eba17411c44ec82239e — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Still linear under the fixed 30s cap: at the measured 0.52s floor a ~230-worktree fleet 504s again — verify headroom on the largest real fleet.

Not justified as shipped

  • Item 6 — undeclared: the spec edit also corrects "merged-verdict candidates" to "merged- or closed-verdict" gh lookups; a harm-free doc-accuracy rider the description never mentions.

What this change ships

Inventory (6 items) — 5 justified

Intent: make the Dev Fleet purge preview work on large fleets — stop the 504 and stop withholding merged worktrees whose branch was deleted. FIX.

  1. Prune preview checks worktrees concurrently (bounded by the existing prune-worker limit of 4), so large fleets no longer 504 — justified
  2. A merged, clean worktree whose head branch was deleted now becomes a purge candidate instead of sitting at merged_unverified forever — justified
  3. A branch name carrying any open PR, or more than 10 PRs, is refused at the destructive boundary (fails closed) — justified
  4. scanned counts non-main worktrees directly; differs only when there is not exactly one main checkout — justified
  5. dev-fleet spec updated in the same commit with the concurrent, discovery-order, lock-free preview contract — justified
  6. Spec prose corrected: closed-verdict candidates also trigger the gh lookup — undeclared (harm-free doc correction rider)

[FIRST-PRINCIPLES-REVIEWED] f6502c6

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review — ✅ no blocking findings

Reviewed f6502c6a8d5c2567f93e0eba17411c44ec82239e — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f6502c6

Verdict parsed from the review's SHA-scoped output markers for commit f6502c6a8d5c2567f93e0eba17411c44ec82239e.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f6502c6a8d5c2567f93e0eba17411c44ec82239e: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of f6502c6a8d5c2567f93e0eba17411c44ec82239e and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- test/test_dev_fleet_server_more_coverage.py:1451 -- Putting "OPEN" only first lets a first-row-only weakening pass -> Fix: assert both row orders.

[GPT-REVIEWED] f6502c6

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f6502c6a8d5c2567f93e0eba17411c44ec82239e: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/prune-preview-timeout-and-merged-blindspot branch from e92c5e4 to ca02112 Compare September 17, 2026 05:03
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: saturation sentinel -- fetch _PR_HEAD_LOOKUP_LIMIT+1 rows and fail closed when the count overflows the ceiling

  • bounded lookup can miss an open PR and authorize pruning (span=ad39618bba7b)

Fixed. The lookup now requests _PR_HEAD_LOOKUP_LIMIT + 1 rows (one saturation sentinel) and returns None when the response overflows the ceiling, so a head with more PRs than can be examined is refused rather than ruled on from a truncated batch. An OPEN row beyond ten MERGED rows can no longer be missed: overflow fails closed before the OPEN scan.
Verified live on real gh: a merged, branch-deleted head resolves its OID; a nonexistent branch returns None. test_fetch_pr_head_oid_saturated_batch_fails_closed pins the overflow-refuse; test_fetch_pr_head_oid_exactly_ceiling_is_complete pins that a full-but-complete window still resolves.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/prune-preview-timeout-and-merged-blindspot branch from ca02112 to 2fc5a7d Compare September 17, 2026 06:13
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/prune-preview-timeout-and-merged-blindspot branch from 2fc5a7d to ab53086 Compare September 17, 2026 08:23
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • "up to two networked gh calls" contradicts fallback traversal (span=e652daa61560)

Fixed. The _prune_candidates comment no longer claims a fixed call count; it now says "one or more networked gh calls (a PR-status lookup, its per-repo fallback traversal, and a merged/closed head-OID check)", which matches _fetch_pr_status iterating _FALLBACK_REPOS plus _fetch_pr_head_oid.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • spec "Scan feedback" still describes the serial-era preview (span=5029caa2c914)

Fixed. docs/system-specs/modules/dev-fleet.md "Scan feedback" now records the preview's new contract: per-worktree verdicts run concurrently bounded by _PRUNE_CONCURRENCY, read-only git that never takes _GIT_MUTATION_LOCK, discovery-order output, and the deleted-branch head-OID lookup via gh pr list.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • preview-specific higher bound would retire the linear-scaling cliff (span=80a8d0bf9ab5)

Accepted and deferred, out of scope. The shared _PRUNE_CONCURRENCY = 4 already clears the 30s cap on the measured fleet, so a preview-specific bound is an optimization, not needed for the fix. Tracked with the batched-gh-list follow-up in issue #11529.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • item 5 rides along: scanned recomputed as len(prunable) (span=da86882acdea)

Declared. The PR body's What-changed section states scanned is now len(prunable) (non-main worktrees counted directly), value-identical to the prior len(worktrees) - 1 whenever exactly one main checkout exists.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • 504 cliff moves rather than disappears; a higher preview-specific bound is safe (span=59ad5411dde1)

Accepted and deferred, out of scope. Correct that a fixed 4-way bound moves the cliff rather than removing it, and that the read-only lock-free preview could safely run a higher bound than the mutation workers'. This PR's contract is "clear the current fleet under the 30s cap", which 4-way does with margin; a preview-specific bound is an optimization tracked with the batched-gh-list follow-up in issue #11529, not folded in here.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author
  • scanned switches from len(worktrees) - 1 to len(prunable) (span=4214d39c5948)

Declared and intentional. len(prunable) counts the non-main worktrees actually scanned; it diverges from len(worktrees) - 1 only when the fleet has zero or multiple main checkouts, in which case len(prunable) is the correct count of what was scanned and len(worktrees) - 1 was already wrong. The PR body's What-changed section states the change. Harm-free and value-identical in the normal one-main case.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

Note on the red Backend Tests (3.12, 3) / PR Readiness: this is a repo-wide shard failure, not this PR's.

The shard fails ~180 tests, all in unrelated modules (test_mcp_discovery.py, test_meetings_routes.py, test_member_memory_api.py, and pod/Task-Scheduler tests), with the dominant signature WindowsTaskError: the Task Scheduler pod backend is win32-only; this host is linux plus PodError/LaunchdError. Zero of them are dev_fleet tests and this PR's diff (5 files: fleet_state.py, worktree_ops.py, three dev_fleet test modules, one spec doc) touches none of them. It reproduces deterministically across two runs and on an unrelated open PR, and coincides with #11528 (ci: split POSIX backend tests into 8 shards) reshuffling shard membership. PR Readiness and Coverage Gate are downstream aggregates of this shard.

Not folding a fix for those pod/win32 tests into this PR (out of scope). This PR's own lanes are green: GPT 5.6, Opus 4.8, Design, First Principles all PASS, and every raised finding is dispositioned.

@chenmingwei23
chenmingwei23 force-pushed the fix/prune-preview-timeout-and-merged-blindspot branch from 874e0d4 to f6502c6 Compare September 17, 2026 15:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 17, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after a full read of the diff on the current head: the change matches the body and linked issue, the destructive/decision path re-verifies independently and fails closed, tests pin the behaviour, and all review lanes are clean on this head.

@iamwhatever
iamwhatever enabled auto-merge (squash) September 17, 2026 19:11
@iamwhatever
iamwhatever merged commit b17ff49 into main Sep 17, 2026
114 of 117 checks passed
@iamwhatever
iamwhatever deleted the fix/prune-preview-timeout-and-merged-blindspot branch September 17, 2026 19:11
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 17, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants