Skip to content

fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue - #23913

Merged
bolichen97 merged 1 commit into
mainfrom
fix/skill-view-fanout-queue-23276
Oct 25, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/skill-view-fanout-queue-23276

Conversation

@buluoray

@buluoray buluoray commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A wide subagent fan-out no longer makes every new session start in the gateway fail at set_mode with "its skill view could not be prepared".

During a fan-out of a few dozen subagents, the gateway logged skill projection lock still busy inside this process after 30s over a hundred times, and every session start in that window (interactive dashboard sessions included) was refused at set_mode for about 40 minutes.

Why it matters

Any user who fans out subagents loses the whole gateway for new sessions, not just the subagents. Retrying does not help while the wave runs, and restarting the gateway kills the wave.

Not a goal

  • Raising the 30 s in-process wait.
  • Giving interactive starts priority over subagent starts, or adding spawn admission back-off. With the queue no longer flooded, neither is needed for this symptom; both are admission design choices.
  • Changing the set_mode refusal itself or its remedy text.
  • Changing the per-spawn prune, the boot drain, or the lease scan budget.

What changed (motivation → approach → change)

Symptom: set_mode re-preparation times out on the in-process alias lock (_projection_alias_lock, 30 s wait) during a fan-out.

Root cause, measured: every projection END runs a forced eager prune (_end_projection_off_loop → _eager_prune(force=True)). A forced prune skips the 5 s _per_spawn_prune_due spacing and queues for the full 30 s in-process wait. A fan-out ends projections all the time: each set_mode drops the previous projection, and each subagent session ends its own. So the lock queue fills with prune passes, and each one spends its full walk budget. The spacing that was meant to cap pruning in a burst does not apply to ends.

Measured locally (one process, 30 concurrent starts, each doing a spawn prepare, a set_mode prepare that drops the first, then a release; 350 stale entries in the agents directory; lock hold and wait timed by wrapping _projection_alias_lock):

end prunes lock held by end prunes lock held by prepares worst prepare wait
main 60 24.2 s 1.2 s 8.7 s
this PR 3 1.2 s 0.7 s 0.5 s

The end prunes held the lock 20 times longer than the starts did. At a few dozen starts, with real kiro-cli and a slower disk, that is enough to run out the 30 s wait.

Change: _coalesced_end_prune (in skill_projection_parts/alias_reclaim.py) runs at most one end prune per (side directory, agents directory, home) at a time. An end that arrives while one runs records a request and returns. When the running pass finishes, one daemon thread makes the follow-up passes and keeps going while requests arrive, so the end that ran first (a release() caller or the reaper thread) is not held for the length of a wave. An end releases its lease before it asks, so a pass that starts after the request covers it: every end is still followed by a prune that sees its lease gone (the gateway-stop case the forced end prune exists for). The spec text in docs/system-specs/modules/acp-client.md is updated: it said the eager prune honours the spacing and never queues, which was not true for ends.

Backwards compatibility

Compatible: the same prune runs under the same lock and budgets; only duplicate passes are folded into one.

Removes nothing: no file, record, key or public function is removed; the views a coalesced end leaves are reclaimed by the follow-up pass, which the new test pins.

Tests

  • test_ends_that_arrive_while_one_prunes_share_its_next_pass: ten ends arrive while one end's prune runs; they return at once and exactly two prune passes run. On main each end runs its own pass (11).
  • test_an_end_covered_by_a_running_prune_still_loses_its_view: a projection released while another end's prune has already passed its lease scan still loses its view, through the follow-up pass. Reddens when the follow-up pass is removed. The first test also reddens when the follow-up passes run inline on the first end's thread.

Proof: PROVEN test_ends_that_arrive_while_one_prunes_share_its_next_pass (head f205d14)

Ran by path with -n0: test/test_native_skill_projection_rewrites.py, test_native_skill_projection_alias_reclaim.py, test_native_skill_projection.py, test_native_skill_projection_lease_scan.py, test_native_skill_projection_scan_bounds.py, test_acp_set_mode_projection_fallback.py, test_skill_view_mode_recovery.py, test_skill_projection_facade.py: all pass. The two new tests passed 5 of 5 repeat runs.

Manual verification

The fan-out measurement above was run on Linux against this branch and main with a scratch harness (not committed). Not verified with a live kiro-cli fan-out. Windows and macOS are unverified; the change uses only threading primitives and touches no file-lock code.

Related Issues

Closes #23276

Related, both closed: #17039 (added the 30 s in-process queue and the per-spawn spacing) and #19954 (refusal wording). Open PR #22361 also edits _end_projection (where an end runs, not how many run); the two may need a rebase against each other but do not overlap in behaviour.

Sibling sweep

Swept: git grep -nE "_projection_alias_lock(" -- src/
Sibling: src/kiro_crew/acp/skill_projection_parts/alias_reclaim.py -- changed here (the end prune, _eager_prune with force=True)
Sibling: src/kiro_crew/acp/skill_projection_parts/preparation.py -- not affected -- the ceiling-fallback prune is one waiter per spawn and gated by _per_spawn_prune_due
Sibling: src/kiro_crew/acp/skill_projection_parts/side_files.py -- not affected -- defines the lock; the per-spawn eager prune takes it with a zero wait
Sibling: src/kiro_crew/acp/skill_projection_parts/alias_reclaim.py:drain_stale_aliases -- not affected -- one boot thread, pausing between batches so starts can take the lock

Pattern harvest

Rule candidate: review-prompt
Pattern: a "terminal" path that bypasses a rate limit and queues on a lock shared with latency-critical callers must coalesce, or a burst of terminal events starves those callers.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

Decision for the owner

The issue asks for a fix that stops a fan-out from refusing every session start, without just raising the 30 s wait. This draft folds the forced end-of-projection prunes into one running pass per directory, which removes the queue flood at its source, and leaves interactive-start priority and spawn back-off unbuilt because the measurement shows they are not needed for this symptom.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f205d142703e2487b467e2473b54db58e1c47865 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 25, 2026
@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f205d142703e2487b467e2473b54db58e1c47865 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The coalescing goes at the cause (one forced prune per end), it sits at the one call site that issues forced end prunes, and it removes nothing.

Notes

  • During a long wave the follow-up thread runs passes back-to-back for as long as requests arrive, and the in-process lock may not hand off in order. Each pass gets its full walk budget, so starts could still lose the lock to it under steady churn. A short yield between follow-up passes would bound that, if a live kiro-cli fan-out ever shows it.

[DESIGN-REVIEWED] f205d14

Description read

[DESCRIPTION-READ] 5163287fabaa2fd1517dc0c164ab0926a085a7b74584af9cd2bb2dbb79ccddf3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f205d142703e2487b467e2473b54db58e1c47865 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix works on the cause: forced end prunes skipped the spacing and piled up on the lock. It changes no defaults, and it has no sibling callers left unfixed.

What this change ships

Inventory (5 items) — 5 justified

Intent: let a user fan out subagents without every new gateway session being refused at set_mode. This is a FIX (#23276).

  1. While one end-of-session prune is running, a later end records a request and returns at once instead of queueing on the alias lock — justified
  2. A single background thread (kirocrew-skill-view-prune) runs the follow-up passes, so the end that started first is not held for the whole wave — justified
  3. Every ended view is still reclaimed by a pass that starts after its lease is released, and a new test pins this — justified
  4. The spec in acp-client.md is corrected: it said the end prune "never queues", which was wrong — justified
  5. Six private names are re-exported through the skill_projection.py facade's type-checking list. Grep _END_PRUNE_|_coalesced_end_prune|... finds 0 consumers outside the owner, but the facade convention is "every owner-defined name in _EXPORTS" — justified

[FIRST-PRINCIPLES-REVIEWED] f205d14

Description read

[DESCRIPTION-READ] 1b5cfd891568ca20c19252c96c3a601e7318aeebe30fb2dfa38f4ec14585bc75

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f205d142703e2487b467e2473b54db58e1c47865 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f205d14

Verdict parsed from the review's SHA-scoped output markers for commit f205d142703e2487b467e2473b54db58e1c47865.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f205d142703e2487b467e2473b54db58e1c47865 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 25, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f205d142703e2487b467e2473b54db58e1c47865 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f205d14

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f205d142703e2487b467e2473b54db58e1c47865 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 25, 2026
@buluoray
buluoray force-pushed the fix/skill-view-fanout-queue-23276 branch from e55e115 to de8e40a Compare October 25, 2026 04:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@buluoray

Copy link
Copy Markdown
Contributor Author

🤖 Kiro Crew Auto-Pipeline [operator: buluoray#ce473fca]: Disposition of the Design Review note on e55e1157f9, and the lint red on that head.

  • Design note (the first end's thread stays busy for a whole wave): fixed in de8e40aa29. The end that finds no prune running still runs the first pass on its own thread. If more ends asked meanwhile, the follow-up passes go to one kirocrew-skill-view-prune daemon thread, so a release() caller or the reaper returns after one pass. test_ends_that_arrive_while_one_prunes_share_its_next_pass now holds the follow-up pass and asserts the first end has already returned; it reddens when the follow-ups run inline.
  • Backend Lint red: a black reformat on the new test's signature, caused by this diff. Fixed in the same push. Coverage Gate was red only because lint failed first.
  • The branch is also rebased onto current main.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 25, 2026
@buluoray
buluoray force-pushed the fix/skill-view-fanout-queue-23276 branch from de8e40a to f205d14 Compare October 25, 2026 04:25
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 25, 2026
@buluoray
buluoray marked this pull request as ready for review October 25, 2026 05:19
@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners October 25, 2026 05:19
@buluoray
buluoray requested a review from bolichen97 October 25, 2026 05:20
@buluoray
buluoray enabled auto-merge (squash) October 25, 2026 05:20
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 25, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at head f205d14.

What I checked:

  • The state machine in _coalesced_end_prune. The RUNNING key is set and cleared under _END_PRUNE_GUARD. A request that arrives before the last check of a pass is consumed by that check, so it gets one more pass. A request that arrives after the key is cleared starts a new run. No request is lost.
  • Coverage of each end. An end closes its lease before it calls _coalesced_end_prune, so the follow-up pass, which starts after the request is consumed, sees that lease gone.
  • The first end is not held for a wave. Follow-up passes run on one daemon thread. If that thread cannot start, the passes run inline, which is the old behaviour.
  • The exception paths clear both sets, so a failed pass cannot leave the key stuck in RUNNING.
  • Sibling callers. _end_projection and the reaper reach the prune only through _end_projection_off_loop. The per-spawn prune and the ceiling fallback keep the _per_spawn_prune_due spacing.
  • Tests. test_an_end_covered_by_a_running_prune_still_loses_its_view fails without the follow-up pass, because passes stays [True] and the second event times out. The first test fails if the follow-ups run inline on the first end's thread.

Gates: the linked issue is T1, no security or approval path is touched, readiness is passed on this head, and all required checks are green.

@bolichen97
bolichen97 merged commit 47e0b8f into main Oct 25, 2026
109 checks passed
@bolichen97
bolichen97 deleted the fix/skill-view-fanout-queue-23276 branch October 25, 2026 06:03
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 25, 2026
bolichen97 pushed a commit that referenced this pull request Oct 26, 2026
…#24142)

test_two_concurrent_releases_each_prune_their_own_view joined only the
prune threads it saw right after both releases. Since #23913 an end that
arrives while the other end's pass runs records a request and returns, and
the running end starts the follow-up pass on a new prune thread just before
it exits. The test never joined that thread, so on a loaded runner it
asserted while the follow-up pass was still walking and reported whichever
view that pass owed as lingering.

The test now joins the release threads and then waits, under the bounded
wait_until, until no end prune is running or requested for its agents
directory. A new test forces the order (one end walks while the other still
holds its lease, and the follow-up pass is held until the wait is polling)
for each release leading, and fails with the old join-only wait.

Closes #23965
bolichen97 added a commit that referenced this pull request Oct 29, 2026
Cherry-picks these main commits onto release/0.11.0, in merge order:

- fix(schedule): return focus to the cron folder toggle after rename (#23907)
  (cherry picked from commit a5b78da)
- fix(probes): declare an undated duplicate check's conservative bucket on the reading (#23912)
  (cherry picked from commit 95640ac)
- fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue (#23913)
  (cherry picked from commit 47e0b8f)
- fix(optimizer): keep the user's ask and add no requirements or solutions (#23915)
  (cherry picked from commit 1f0e872)
- fix(sqlite): retry the WAL switch when another opener holds a lock (#22865) (#23920)
  (cherry picked from commit 03f3b2a)
- fix(slack): post a quiet nudge cycle's outcome to the linked thread (#23911)
  (cherry picked from commit 32faa11)
- feat(crew): run each crewmate's worker in its own git worktree (#23563)
  (cherry picked from commit 6b21419)
- fix(members): finish the phone crewmates single top bar (#19629)
  (cherry picked from commit a6150d5)
- fix(channels): stop Discord, Teams and Webex re-announcing while held (#23916)
  (cherry picked from commit 1f3210d)
- fix(subagent): macOS spawn floor reads host_available_mib (#23914)
  (cherry picked from commit ed83ad2)
- feat(crew): run a crewmate's worker on claude as well as KAS (#23538)
  (cherry picked from commit 32362f8)
- fix(dashboard): read every body integer through body_int (#22502) (#23921)
  (cherry picked from commit dad46e9)
iamwhatever pushed a commit that referenced this pull request Oct 29, 2026
Cherry-picks these main commits onto release/0.11.0, in merge order:

- fix(schedule): return focus to the cron folder toggle after rename (#23907)
  (cherry picked from commit a5b78da)
- fix(probes): declare an undated duplicate check's conservative bucket on the reading (#23912)
  (cherry picked from commit 95640ac)
- fix(skill-projection): coalesce end prunes so a fan-out cannot hold the start queue (#23913)
  (cherry picked from commit 47e0b8f)
- fix(optimizer): keep the user's ask and add no requirements or solutions (#23915)
  (cherry picked from commit 1f0e872)
- fix(sqlite): retry the WAL switch when another opener holds a lock (#22865) (#23920)
  (cherry picked from commit 03f3b2a)
- fix(slack): post a quiet nudge cycle's outcome to the linked thread (#23911)
  (cherry picked from commit 32faa11)
- feat(crew): run each crewmate's worker in its own git worktree (#23563)
  (cherry picked from commit 6b21419)
- fix(members): finish the phone crewmates single top bar (#19629)
  (cherry picked from commit a6150d5)
- fix(channels): stop Discord, Teams and Webex re-announcing while held (#23916)
  (cherry picked from commit 1f3210d)
- fix(subagent): macOS spawn floor reads host_available_mib (#23914)
  (cherry picked from commit ed83ad2)
- feat(crew): run a crewmate's worker on claude as well as KAS (#23538)
  (cherry picked from commit 32362f8)
- fix(dashboard): read every body integer through body_int (#22502) (#23921)
  (cherry picked from commit dad46e9)
bolichen97 added a commit that referenced this pull request Oct 29, 2026
The v0.11.0-rc.2 release run failed two Test Release Candidate shards on
tests main has already fixed:

- test(skills): wait for the coalesced follow-up prune before asserting (#24142)
  (cherry picked from commit 5d98031)
- test(skills): freeze the lease-scan clock in the legacy-lock test (#24257)
  (cherry picked from commit 3ff45cb)
- test: pin the prune clock in the seam pruning test (#24191)
  (cherry picked from commit 33fd421)

#24191's test seeds a 2026-09-25 history file and prunes at 30 days, so it
fails on every run after 2026-10-25. #24142 follows the end-prune
coalescing that #23913 brought to this branch.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants