Skip to content

fix(skill-projection): never run a dropped projection's end inline on the dropping thread - #22361

Merged
iamwhatever merged 1 commit into
mainfrom
kc-wt-20565
Nov 4, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
kc-wt-20565

Conversation

@dodgeblaster

@dodgeblaster dodgeblaster commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop the skill-view projection finalizer from deadlocking the thread that drops it, which crashed an xdist worker on Windows CI.

A dropped projection's lease finalizer (_end_projection) only handed its work to the reaper thread when the CYCLIC collector was running. A plain refcount-zero drop runs the same weakref.finalize callback inline on whichever thread dropped the last reference, and off the event loop that thread ran the lease release + eager prune directly. The prune calls the security path-resolution subprocess pool (a ThreadPoolExecutor). When the drop fired while that same thread was already inside ThreadPoolExecutor.submit — constructing the Future whose allocation freed the projection, under _global_shutdown_lock — the end re-entered the executor and waited on a lock its own thread held, forever. On Windows CI this hung and killed the xdist worker, crashing the unrelated test_a_recovered_source_rebinds_after_being_retained (its rebuild resolves sensitive paths through that pool).

Observed on PR #20523 (gw3) and PR #21795 (gw2): worker 'gw2' crashed while running test/test_mcp_rebuild_reconsumption.py::...::test_a_recovered_source_rebinds_after_being_retained. Also tracked as the timeout in #21661.

Why it matters

A refcount-driven projection drop happens constantly in normal operation, so any drop that lands inside the path-resolve executor can hang a gateway thread forever. On CI it flakes unrelated Windows shards (worker-restart is disabled, so one crash aborts the whole run); in production it is a latent deadlock of a session thread.

Not a goal

  • N/A — no new feature or surface. The fix is confined to the dropped-projection end path; the on-loop daemon-thread path and the explicit release() path keep their existing behavior.

What changed (motivation → approach → change)

Fixes #20565
Fixes #20436

  • Symptom: Windows xdist worker crash on test_a_recovered_source_rebinds_after_being_retained (Flaky: test_a_recovered_source_rebinds_after_being_retained crashes its xdist worker on Windows #20565), a hang inside a ThreadPoolExecutor.submit reached re-entrantly from a weakref.finalize callback.
  • Root cause: _end_projection deferred to the reaper only for the cyclic collector (_COLLECTING_THREAD); a refcount-zero finalize ran inline on an arbitrary thread that could already hold the executor's non-reentrant lock the prune re-enters.
  • Change: off the event loop, a finalizer-fired end (refcount drop OR cyclic collection) is handed to the daemon reaper thread, which holds none of those locks — one rule for every finalizer path, so no gc-callback bookkeeping of which thread is collecting is needed. The one exception is interpreter shutdown: the reaper may already be gone, or still alive while atexit handlers run yet stopped by finalization before it reaches a queued end. An atexit hook (_release_queued_leases_at_exit) sets _EXITING, releases the lease of every end still queued, and gives an end the reaper already took up to 2 s to finish; once _EXITING is set, or when _reaper_alive() is false, a finalizer releases the lease INLINE. Whichever order the hook and the weakref sweep run in, the lock file is not left behind for a projection the shared runtime never explicitly released; the prune is dropped for the next boot drain. An EXPLICIT release() still runs the end inline off the loop, told apart from a dropped finalizer by a direct call argument, not process state: it detach()es the weakref.finalize and calls _end_projection with from_release=True, while the finalizer path uses the default from_release=False. No thread-local and no cross-module flag import, so the parts layer order is untouched.

Backwards compatibility

Compatible: the only observable change is that a dropped projection's lease release + prune now completes on the reaper thread rather than synchronously on the dropping thread (it already did for the cyclic-collector case). Explicit release() is unchanged. The prune is idempotent and bounded, so deferral never loses cleanup.

Tests

  • test_native_skill_projection_rewrites.py — new test_a_projection_dropped_off_the_loop_ends_on_the_reaper_not_the_dropping_thread forces the end to fire on a thread holding a lock the end would re-enter and asserts it runs on the reaper (red on parent logic, green with the fix); existing cyclic-collector and release tests still pass.
  • test_native_skill_projection_rewrites.py — interpreter-exit coverage: test_a_projection_held_until_exit_releases_its_lease_with_the_reaper_alive runs a real interpreter that exits holding a projection while the reaper is alive but busy, and asserts the lease file is gone after exit (red without the exit hook); test_an_off_loop_drop_once_exit_began_releases_the_lease_inline_with_the_reaper_alive, test_the_exit_hook_releases_the_lease_of_an_end_still_queued (a queued non-end item is left uncalled) and test_the_exit_hook_waits_for_the_end_the_reaper_already_took each fail on a mutant that drops the matching piece.
  • test_native_skill_projection_alias_reclaim.py, test_native_skill_projection_lease_scan.py, test_skill_view_relaunch_harness.py, test_skill_projection_facade.py, test_mcp_rebuild_reconsumption.py — all pass at -n0/-n4.
  • isort/flake8/black --check/mypy clean on changed source. docs/system-specs/modules/acp-client.md updated to match (AGENTS.md owning-spec rule).

Pattern harvest

Rule candidate: review-prompt / agents-md
Pattern: a weakref.finalize / GC finalizer that runs heavy, lock-taking work (file locks, a subprocess-pool submit) INLINE on the thread that dropped the last reference. A finalizer can fire mid-allocation on any thread, inside a non-reentrant stdlib lock, so finalizer-driven cleanup must be handed to a dedicated worker (a reaper queue), never run inline — guarding only the cyclic collector (gc.callbacks) misses the refcount-zero drop path.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of bc68d531bfa003abdb19a22f3fb5fe65e0f8100b and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bc68d53

False positive or not applicable? A repository writer can comment:
/ai-review override gpt bc68d531bfa003abdb19a22f3fb5fe65e0f8100b false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision bc68d531bfa003abdb19a22f3fb5fe65e0f8100b touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of bc68d531bfa003abdb19a22f3fb5fe65e0f8100b — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Sending every finalizer-fired end off the loop to the reaper fixes the deadlock at its root, which is lock-taking work running inline on whatever thread drops the last reference. The interpreter-exit lease fallback covers the one case where a queued end could be lost.

Notes

  • A dropped projection's lease is now released asynchronously off the loop, so code that drops one and immediately expects its aliases to be free must now wait on the reaper. The in-tree tests had to add _settle_reaper for exactly this. I found no production reader that relies on synchronous release.

[DESIGN-REVIEWED] bc68d53

Description read

[DESCRIPTION-READ] e7b60b1a950eec726d482d14b1f3f1c2218e4c2e661127bfbe47693e0f514374

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of bc68d531bfa003abdb19a22f3fb5fe65e0f8100b — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Every item traces to the deadlock in the linked issues (#20565, #20436), and the fix removes the cause.

Inventory (7 items) — 7 justified

Intent: stop a dropped skill projection's cleanup from hanging the thread that dropped it. FIX.

  1. When a projection's last reference goes off the event loop, its end now runs on the reaper thread instead of the dropping thread — justified
  2. The gc.callbacks collecting-thread hook and _COLLECTING_THREAD are deleted, so one rule covers every finalizer path — justified
  3. An explicit release() detaches the finalizer and still runs the end inline (from_release=True) — justified
  4. A dropped projection's lease release and prune now finish later, in the background — justified
  5. At interpreter exit, the lease of every end still queued is released on the exiting thread, and the hook waits up to 2 s for an end the reaper already took — justified
  6. During exit, or when the reaper is gone, a finalizer releases its lease inline and skips the prune, leaving it to the next boot drain — justified
  7. Existing drop tests now wait for the reaper before asserting — justified

[FIRST-PRINCIPLES-REVIEWED] bc68d53

Description read

[DESCRIPTION-READ] 3bfbb4e9f48c8c5499bafb5331a1994fe8f0ab4e6e62d35870647593949f9028

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed bc68d531bfa003abdb19a22f3fb5fe65e0f8100b — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] bc68d53

Verdict parsed from the review's SHA-scoped output markers for commit bc68d531bfa003abdb19a22f3fb5fe65e0f8100b.

False positive or not applicable? A repository writer can comment:
/ai-review override fable bc68d531bfa003abdb19a22f3fb5fe65e0f8100b false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 20, 2026
@dodgeblaster
dodgeblaster force-pushed the kc-wt-20565 branch 2 times, most recently from 35bbac4 to 7a61c27 Compare October 20, 2026 17:50
@dodgeblaster
dodgeblaster marked this pull request as ready for review October 20, 2026 21:18
@dodgeblaster
dodgeblaster requested review from a team and KiroCrewCaptain as code owners October 20, 2026 21:18
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 20, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at 7a61c27.

The main fix is sound: off the loop, a finalizer-driven end now always goes to the reaper, so a refcount-zero drop inside ThreadPoolExecutor.submit can no longer re-enter the executor on the dropping thread, and explicit release() stays inline via detach() + from_release=True. CI is green on this head, including all eight Windows backend shards.

One current-head AI concern has no reply. Design Review's Watch item points out that daemon threads are still alive while atexit handlers run, so during the weakref.finalize atexit sweep _reaper_alive() (alias_reclaim.py#L197-L206) returns True and the end is queued (#L314-L321) for a reaper that may never run before exit. The inline fallback the description relies on is then close to unreachable, and its test only passes because it monkeypatches _reaper_alive. Before this change the off-loop atexit path released the lease inline.

To clear: either make the fallback cover interpreter finalization (for example also take the inline branch when sys.is_finalizing() or an atexit flag is set), or reply on the PR that a leftover lease at exit goes stale on its own (orphan lock sweep at next boot), update the description/spec wording, and drop the unreachable branch.

— crew-pr-reviewer (r116)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 21, 2026 21:14

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding: the atexit path sees a live reaper and only queues cleanup. Python can exit before the daemon reaper runs, leaving the lease held.

Required fix: release the lease before process exit, or prove queued cleanup completes first. Add a test that exits with the reaper alive and verifies lease release.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 23, 2026
@dodgeblaster

Copy link
Copy Markdown
Contributor Author

@bolichen97 thanks, agreed. Daemon threads are still alive while atexit handlers run, so _reaper_alive() answered True during the weakref sweep and the end was only queued. Finalization could then stop the reaper before it reached that end, and the lease stayed held.

Fixed in a073b75 (src/kiro_crew/acp/skill_projection_parts/alias_reclaim.py):

  • A module-level atexit hook, _release_queued_leases_at_exit, sets _EXITING, releases the lease of every end still on the reaper queue (on the exiting thread, lease only; the prune is left to the next boot drain as on the other shutdown paths), then gives an end the reaper had already taken up to _EXIT_REAPER_WAIT_SECS (2 s) to finish.
  • _end_projection now queues a finalizer-driven end only when _reaper_alive() and not _EXITING.is_set(); otherwise it releases the lease inline.
  • atexit runs handlers LIFO, so the order relative to weakref.finalize's own sweep depends on which registered first. Both orders are covered. If the hook runs first, _EXITING is already set when the sweep fires and the release happens inline. If the sweep runs first, its ends are queued and the hook then drains them.

Tests (test/test_native_skill_projection_rewrites.py):

  • test_a_projection_held_until_exit_releases_its_lease_with_the_reaper_alive: the test you asked for. A real interpreter holds a projection until exit while the reaper is alive but busy (blocked on an earlier queue item). After the process exits, the test asserts that the lease file is gone. Without the atexit.register line, the test fails and the lease file is left behind.
  • test_an_off_loop_drop_once_exit_began_releases_the_lease_inline_with_the_reaper_alive: fails if the _EXITING check is removed.
  • test_the_exit_hook_releases_the_lease_of_an_end_still_queued: also checks that a queued item that is not a projection end is left uncalled.
  • test_the_exit_hook_waits_for_the_end_the_reaper_already_took: fails if the bounded wait is removed.

docs/system-specs/modules/acp-client.md and the PR body describe the new exit path.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 2, 2026
… the dropping thread

A skill-view projection's lease finalizer (`_end_projection`) only diverted to
the reaper thread when the CYCLIC collector was running (a `gc.callbacks` guard).
A plain refcount-zero drop runs the same `weakref.finalize` callback INLINE on
whichever thread dropped the last reference -- with no collection, so the guard
never fired -- and off the event loop that thread ran the lease release and eager
prune directly.

That prune calls the security path-resolution subprocess pool, whose bounded
request goes through a `ThreadPoolExecutor`. When the drop fired while that same
thread was already inside `ThreadPoolExecutor.submit` (constructing the `Future`
whose allocation freed the projection, under `_global_shutdown_lock`), the end
re-entered the executor and waited on a lock its own thread held -- forever.
On Windows CI this hung the test and killed the xdist worker, crashing an
unrelated test (`test_a_recovered_source_rebinds_after_being_retained`, #20565;
also the same finalizer-inline deadlock reported in #20436), because the rebuild
it exercised resolves sensitive paths through that pool.

Fix: off the event loop, a finalizer-fired end (refcount drop OR cyclic
collection) is handed to the daemon reaper thread, which holds none of those
locks -- ONE rule for every finalizer path, so the gc-callback bookkeeping of
which thread is collecting is no longer needed and is removed. The one exception
is interpreter shutdown. The daemon reaper may already be gone, or still alive
while `atexit` handlers run yet stopped by finalization before it reaches a
queued end. An `atexit` hook sets `_EXITING`, releases the lease of every end
still queued, and gives an end the reaper already took a bounded wait to finish;
once `_EXITING` is set, or when `_reaper_alive()` is false, a finalizer releases
the lease INLINE. So the lock file is never left behind for a projection the
shared runtime never explicitly released, whichever order the hook and the
weakref sweep run in (the prune, which re-enters the executor, is dropped for
the next process's boot drain to cover). An EXPLICIT `release()` still runs the end
inline off the loop, told apart from a dropped finalizer by a direct call
argument, not process state: it `detach()`es the `weakref.finalize` and calls
`_end_projection` with `from_release=True`, while the finalizer path uses the
default `from_release=False`. No thread-local and no cross-module flag import.
The on-loop daemon-thread path is unchanged.

Adds a refcount-drop regression test (companion to the existing cyclic-collector
one) that forces the end to fire on a thread holding a lock the end would
re-enter and asserts it runs on the reaper instead; interpreter-shutdown tests
for the inline release with the reaper gone or the exit begun, the exit hook
releasing a queued end's lease and waiting for the reaper's in-flight end; and a
real-interpreter test that exits holding a projection with the reaper alive but
busy and asserts the lease is gone afterwards. Updates
acp-client.md to describe the single finalizer rule and the at-exit fallback.

Fixes #20565
Fixes #20436
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 2, 2026
@dodgeblaster

dodgeblaster commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=c7173e98a7aa Isolate replacement queues from the real reaper

fixed in bc68d53: every test that installs a private _COLLECTED_ENDS queue now also patches _ensure_projection_reaper to a no-op, so prepare_native_skill_projection() can no longer start a real reaper on that queue.
No reaper drains a private queue, and none is left blocked on it after the test restores the real one. A reaper already running stays blocked in get() on the real queue, which these tests never feed.
Reproduced before the fix: test_the_exit_hook_releases_the_lease_of_an_end_still_queued, run alone in a fresh process (no reaper running yet), fails. It passes with the patch, as does every site at :1089, :1123, :1154 and :1191.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 2, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD at bc68d53.

The code now handles the exit case from our r116 hold and from @bolichen97's change request. An atexit hook sets _EXITING, releases the lease of every end still queued, and waits up to 2 s for an end the reaper already took (alias_reclaim.py#L223-L254). After that, a finalizer releases its lease inline (L356-L375). The deadlock fix itself is sound: a dropped projection's end never runs inline on the dropping thread, and explicit release() still runs inline through detach() and from_release=True. I found no defect, the GPT test-isolation finding is fixed, and every check is green.

What clears this hold: @bolichen97's change request on 7a61c27 is still active. The fix and the exit test they asked for are in this head; a re-review or dismissal by the reviewer clears it.

The Design Review note (a dropped projection's lease is now released in the background) does not block: no production reader relies on the synchronous release, and explicit release() is unchanged.

— crew-pr-reviewer (r141)

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at bc68d53. This replaces my earlier change request on 7a61c27.

That request was about the exit path: with the daemon reaper still alive, a finalizer fired by the atexit weakref sweep only queued its end, and the process could exit before the reaper ran it, leaving the lease held. This head fixes it:

  • _release_queued_leases_at_exit is registered with atexit. It sets _EXITING, closes the lease of every projection end still on _COLLECTED_ENDS, and gives an end the reaper already took up to _EXIT_REAPER_WAIT_SECS (2 s) to finish.
  • In _end_projection, a finalizer-driven end off the loop is queued only while the reaper is alive and _EXITING is not set. Otherwise it closes the lease inline and leaves the prune to the next boot drain.
  • release() now detaches the finalizer and calls the end with from_release=True, so only the explicit release runs the full end inline. This replaces the gc.callbacks collecting-thread check.
  • test_a_projection_held_until_exit_releases_its_lease_with_the_reaper_alive runs a real interpreter whose reaper is alive but busy at exit, and checks the lease directory is empty afterwards. The unit tests cover the hook draining a queued end, waiting for an end the reaper took, and the inline release once exit began.

Checks at this head: all green, no pending checks, no other open change request.

@iamwhatever
iamwhatever merged commit 7e142c3 into main Nov 4, 2026
96 checks passed
@iamwhatever
iamwhatever deleted the kc-wt-20565 branch November 4, 2026 09:31
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants