Repository navigation
fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue - #23913
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The coalescing goes at the cause (one forced prune per end), it sits at the one call site that issues forced end prunes, and it removes nothing. Notes
[DESIGN-REVIEWED] f205d14 Description read[DESCRIPTION-READ] 5163287fabaa2fd1517dc0c164ab0926a085a7b74584af9cd2bb2dbb79ccddf3 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix works on the cause: forced end prunes skipped the spacing and piled up on the lock. It changes no defaults, and it has no sibling callers left unfixed. What this change shipsInventory (5 items) — 5 justifiedIntent: let a user fan out subagents without every new gateway session being refused at
[FIRST-PRINCIPLES-REVIEWED] f205d14 Description read[DESCRIPTION-READ] 1b5cfd891568ca20c19252c96c3a601e7318aeebe30fb2dfa38f4ec14585bc75 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
e55e115 to
de8e40a
Compare
|
🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Disposition of the Design Review note on
|
de8e40a to
f205d14
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approving at head f205d14.
What I checked:
- The state machine in
_coalesced_end_prune. The RUNNING key is set and cleared under_END_PRUNE_GUARD. A request that arrives before the last check of a pass is consumed by that check, so it gets one more pass. A request that arrives after the key is cleared starts a new run. No request is lost. - Coverage of each end. An end closes its lease before it calls
_coalesced_end_prune, so the follow-up pass, which starts after the request is consumed, sees that lease gone. - The first end is not held for a wave. Follow-up passes run on one daemon thread. If that thread cannot start, the passes run inline, which is the old behaviour.
- The exception paths clear both sets, so a failed pass cannot leave the key stuck in RUNNING.
- Sibling callers.
_end_projectionand the reaper reach the prune only through_end_projection_off_loop. The per-spawn prune and the ceiling fallback keep the_per_spawn_prune_duespacing. - Tests.
test_an_end_covered_by_a_running_prune_still_loses_its_viewfails without the follow-up pass, becausepassesstays[True]and the second event times out. The first test fails if the follow-ups run inline on the first end's thread.
Gates: the linked issue is T1, no security or approval path is touched, readiness is passed on this head, and all required checks are green.
…#24142) test_two_concurrent_releases_each_prune_their_own_view joined only the prune threads it saw right after both releases. Since #23913 an end that arrives while the other end's pass runs records a request and returns, and the running end starts the follow-up pass on a new prune thread just before it exits. The test never joined that thread, so on a loaded runner it asserted while the follow-up pass was still walking and reported whichever view that pass owed as lingering. The test now joins the release threads and then waits, under the bounded wait_until, until no end prune is running or requested for its agents directory. A new test forces the order (one end walks while the other still holds its lease, and the follow-up pass is held until the wait is polling) for each release leading, and fails with the old join-only wait. Closes #23965
Cherry-picks these main commits onto release/0.11.0, in merge order: - fix(schedule): return focus to the cron folder toggle after rename (#23907) (cherry picked from commit a5b78da) - fix(probes): declare an undated duplicate check's conservative bucket on the reading (#23912) (cherry picked from commit 95640ac) - fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue (#23913) (cherry picked from commit 47e0b8f) - fix(optimizer): keep the user's ask and add no requirements or solutions (#23915) (cherry picked from commit 1f0e872) - fix(sqlite): retry the WAL switch when another opener holds a lock (#22865) (#23920) (cherry picked from commit 03f3b2a) - fix(slack): post a quiet nudge cycle's outcome to the linked thread (#23911) (cherry picked from commit 32faa11) - feat(crew): run each crewmate's worker in its own git worktree (#23563) (cherry picked from commit 6b21419) - fix(members): finish the phone crewmates single top bar (#19629) (cherry picked from commit a6150d5) - fix(channels): stop Discord, Teams and Webex re-announcing while held (#23916) (cherry picked from commit 1f3210d) - fix(subagent): macOS spawn floor reads host_available_mib (#23914) (cherry picked from commit ed83ad2) - feat(crew): run a crewmate's worker on claude as well as KAS (#23538) (cherry picked from commit 32362f8) - fix(dashboard): read every body integer through body_int (#22502) (#23921) (cherry picked from commit dad46e9)
Cherry-picks these main commits onto release/0.11.0, in merge order: - fix(schedule): return focus to the cron folder toggle after rename (#23907) (cherry picked from commit a5b78da) - fix(probes): declare an undated duplicate check's conservative bucket on the reading (#23912) (cherry picked from commit 95640ac) - fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue (#23913) (cherry picked from commit 47e0b8f) - fix(optimizer): keep the user's ask and add no requirements or solutions (#23915) (cherry picked from commit 1f0e872) - fix(sqlite): retry the WAL switch when another opener holds a lock (#22865) (#23920) (cherry picked from commit 03f3b2a) - fix(slack): post a quiet nudge cycle's outcome to the linked thread (#23911) (cherry picked from commit 32faa11) - feat(crew): run each crewmate's worker in its own git worktree (#23563) (cherry picked from commit 6b21419) - fix(members): finish the phone crewmates single top bar (#19629) (cherry picked from commit a6150d5) - fix(channels): stop Discord, Teams and Webex re-announcing while held (#23916) (cherry picked from commit 1f3210d) - fix(subagent): macOS spawn floor reads host_available_mib (#23914) (cherry picked from commit ed83ad2) - feat(crew): run a crewmate's worker on claude as well as KAS (#23538) (cherry picked from commit 32362f8) - fix(dashboard): read every body integer through body_int (#22502) (#23921) (cherry picked from commit dad46e9)
The v0.11.0-rc.2 release run failed two Test Release Candidate shards on tests main has already fixed: - test(skills): wait for the coalesced follow-up prune before asserting (#24142) (cherry picked from commit 5d98031) - test(skills): freeze the lease-scan clock in the legacy-lock test (#24257) (cherry picked from commit 3ff45cb) - test: pin the prune clock in the seam pruning test (#24191) (cherry picked from commit 33fd421) #24191's test seeds a 2026-09-25 history file and prunes at 30 days, so it fails on every run after 2026-10-25. #24142 follows the end-prune coalescing that #23913 brought to this branch.
Problem / Motivation
Goal: A wide subagent fan-out no longer makes every new session start in the gateway fail at
set_modewith "its skill view could not be prepared".During a fan-out of a few dozen subagents, the gateway logged
skill projection lock still busy inside this process after 30sover a hundred times, and every session start in that window (interactive dashboard sessions included) was refused atset_modefor about 40 minutes.Why it matters
Any user who fans out subagents loses the whole gateway for new sessions, not just the subagents. Retrying does not help while the wave runs, and restarting the gateway kills the wave.
Not a goal
set_moderefusal itself or its remedy text.What changed (motivation → approach → change)
Symptom:
set_modere-preparation times out on the in-process alias lock (_projection_alias_lock, 30 s wait) during a fan-out.Root cause, measured: every projection END runs a forced eager prune (
_end_projection_off_loop→_eager_prune(force=True)). A forced prune skips the 5 s_per_spawn_prune_duespacing and queues for the full 30 s in-process wait. A fan-out ends projections all the time: eachset_modedrops the previous projection, and each subagent session ends its own. So the lock queue fills with prune passes, and each one spends its full walk budget. The spacing that was meant to cap pruning in a burst does not apply to ends.Measured locally (one process, 30 concurrent starts, each doing a spawn prepare, a
set_modeprepare that drops the first, then a release; 350 stale entries in the agents directory; lock hold and wait timed by wrapping_projection_alias_lock):The end prunes held the lock 20 times longer than the starts did. At a few dozen starts, with real kiro-cli and a slower disk, that is enough to run out the 30 s wait.
Change:
_coalesced_end_prune(inskill_projection_parts/alias_reclaim.py) runs at most one end prune per (side directory, agents directory, home) at a time. An end that arrives while one runs records a request and returns. When the running pass finishes, one daemon thread makes the follow-up passes and keeps going while requests arrive, so the end that ran first (arelease()caller or the reaper thread) is not held for the length of a wave. An end releases its lease before it asks, so a pass that starts after the request covers it: every end is still followed by a prune that sees its lease gone (the gateway-stop case the forced end prune exists for). The spec text indocs/system-specs/modules/acp-client.mdis updated: it said the eager prune honours the spacing and never queues, which was not true for ends.Backwards compatibility
Compatible: the same prune runs under the same lock and budgets; only duplicate passes are folded into one.
Removes nothing: no file, record, key or public function is removed; the views a coalesced end leaves are reclaimed by the follow-up pass, which the new test pins.
Tests
test_ends_that_arrive_while_one_prunes_share_its_next_pass: ten ends arrive while one end's prune runs; they return at once and exactly two prune passes run. On main each end runs its own pass (11).test_an_end_covered_by_a_running_prune_still_loses_its_view: a projection released while another end's prune has already passed its lease scan still loses its view, through the follow-up pass. Reddens when the follow-up pass is removed. The first test also reddens when the follow-up passes run inline on the first end's thread.Proof: PROVEN test_ends_that_arrive_while_one_prunes_share_its_next_pass (head f205d14)
Ran by path with
-n0:test/test_native_skill_projection_rewrites.py,test_native_skill_projection_alias_reclaim.py,test_native_skill_projection.py,test_native_skill_projection_lease_scan.py,test_native_skill_projection_scan_bounds.py,test_acp_set_mode_projection_fallback.py,test_skill_view_mode_recovery.py,test_skill_projection_facade.py: all pass. The two new tests passed 5 of 5 repeat runs.Manual verification
The fan-out measurement above was run on Linux against this branch and main with a scratch harness (not committed). Not verified with a live kiro-cli fan-out. Windows and macOS are unverified; the change uses only
threadingprimitives and touches no file-lock code.Related Issues
Closes #23276
Related, both closed: #17039 (added the 30 s in-process queue and the per-spawn spacing) and #19954 (refusal wording). Open PR #22361 also edits
_end_projection(where an end runs, not how many run); the two may need a rebase against each other but do not overlap in behaviour.Sibling sweep
Swept: git grep -nE "_projection_alias_lock(" -- src/
Sibling: src/kiro_crew/acp/skill_projection_parts/alias_reclaim.py -- changed here (the end prune,
_eager_prunewithforce=True)Sibling: src/kiro_crew/acp/skill_projection_parts/preparation.py -- not affected -- the ceiling-fallback prune is one waiter per spawn and gated by
_per_spawn_prune_dueSibling: src/kiro_crew/acp/skill_projection_parts/side_files.py -- not affected -- defines the lock; the per-spawn eager prune takes it with a zero wait
Sibling: src/kiro_crew/acp/skill_projection_parts/alias_reclaim.py:drain_stale_aliases -- not affected -- one boot thread, pausing between batches so starts can take the lock
Pattern harvest
Rule candidate: review-prompt
Pattern: a "terminal" path that bypasses a rate limit and queues on a lock shared with latency-critical callers must coalesce, or a burst of terminal events starves those callers.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.
Decision for the owner
The issue asks for a fix that stops a fan-out from refusing every session start, without just raising the 30 s wait. This draft folds the forced end-of-projection prunes into one running pass per directory, which removes the queue flood at its source, and leaves interactive-start priority and spawn back-off unbuilt because the measurement shows they are not needed for this symptom.