Repository navigation
fix(skill-projection): never run a dropped projection's end inline on the dropping thread - #22361
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
bfc3e28 to
76cf8fe
Compare
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Sending every finalizer-fired end off the loop to the reaper fixes the deadlock at its root, which is lock-taking work running inline on whatever thread drops the last reference. The interpreter-exit lease fallback covers the one case where a queued end could be lost. Notes
[DESIGN-REVIEWED] bc68d53 Description read[DESCRIPTION-READ] e7b60b1a950eec726d482d14b1f3f1c2218e4c2e661127bfbe47693e0f514374 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Every item traces to the deadlock in the linked issues (#20565, #20436), and the fix removes the cause. Inventory (7 items) — 7 justifiedIntent: stop a dropped skill projection's cleanup from hanging the thread that dropped it. FIX.
[FIRST-PRINCIPLES-REVIEWED] bc68d53 Description read[DESCRIPTION-READ] 3bfbb4e9f48c8c5499bafb5331a1994fe8f0ab4e6e62d35870647593949f9028 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
76cf8fe to
0336291
Compare
0336291 to
b92e2db
Compare
35bbac4 to
7a61c27
Compare
|
HOLD at 7a61c27. The main fix is sound: off the loop, a finalizer-driven end now always goes to the reaper, so a refcount-zero drop inside One current-head AI concern has no reply. Design Review's Watch item points out that daemon threads are still alive while To clear: either make the fallback cover interpreter finalization (for example also take the inline branch when — crew-pr-reviewer (r116) |
bolichen97
left a comment
There was a problem hiding this comment.
Blocking finding: the atexit path sees a live reaper and only queues cleanup. Python can exit before the daemon reaper runs, leaving the lease held.
Required fix: release the lease before process exit, or prove queued cleanup completes first. Add a test that exits with the reaper alive and verifies lease release.
7a61c27 to
a073b75
Compare
|
@bolichen97 thanks, agreed. Daemon threads are still alive while Fixed in a073b75 (
Tests (
|
… the dropping thread A skill-view projection's lease finalizer (`_end_projection`) only diverted to the reaper thread when the CYCLIC collector was running (a `gc.callbacks` guard). A plain refcount-zero drop runs the same `weakref.finalize` callback INLINE on whichever thread dropped the last reference -- with no collection, so the guard never fired -- and off the event loop that thread ran the lease release and eager prune directly. That prune calls the security path-resolution subprocess pool, whose bounded request goes through a `ThreadPoolExecutor`. When the drop fired while that same thread was already inside `ThreadPoolExecutor.submit` (constructing the `Future` whose allocation freed the projection, under `_global_shutdown_lock`), the end re-entered the executor and waited on a lock its own thread held -- forever. On Windows CI this hung the test and killed the xdist worker, crashing an unrelated test (`test_a_recovered_source_rebinds_after_being_retained`, #20565; also the same finalizer-inline deadlock reported in #20436), because the rebuild it exercised resolves sensitive paths through that pool. Fix: off the event loop, a finalizer-fired end (refcount drop OR cyclic collection) is handed to the daemon reaper thread, which holds none of those locks -- ONE rule for every finalizer path, so the gc-callback bookkeeping of which thread is collecting is no longer needed and is removed. The one exception is interpreter shutdown. The daemon reaper may already be gone, or still alive while `atexit` handlers run yet stopped by finalization before it reaches a queued end. An `atexit` hook sets `_EXITING`, releases the lease of every end still queued, and gives an end the reaper already took a bounded wait to finish; once `_EXITING` is set, or when `_reaper_alive()` is false, a finalizer releases the lease INLINE. So the lock file is never left behind for a projection the shared runtime never explicitly released, whichever order the hook and the weakref sweep run in (the prune, which re-enters the executor, is dropped for the next process's boot drain to cover). An EXPLICIT `release()` still runs the end inline off the loop, told apart from a dropped finalizer by a direct call argument, not process state: it `detach()`es the `weakref.finalize` and calls `_end_projection` with `from_release=True`, while the finalizer path uses the default `from_release=False`. No thread-local and no cross-module flag import. The on-loop daemon-thread path is unchanged. Adds a refcount-drop regression test (companion to the existing cyclic-collector one) that forces the end to fire on a thread holding a lock the end would re-enter and asserts it runs on the reaper instead; interpreter-shutdown tests for the inline release with the reaper gone or the exit begun, the exit hook releasing a queued end's lease and waiting for the reaper's in-flight end; and a real-interpreter test that exits holding a projection with the reaper alive but busy and asserts the lease is gone afterwards. Updates acp-client.md to describe the single finalizer rule and the at-exit fallback. Fixes #20565 Fixes #20436
a073b75 to
bc68d53
Compare
|
|
HOLD at bc68d53. The code now handles the exit case from our r116 hold and from @bolichen97's change request. An What clears this hold: @bolichen97's change request on 7a61c27 is still active. The fix and the exit test they asked for are in this head; a re-review or dismissal by the reviewer clears it. The Design Review note (a dropped projection's lease is now released in the background) does not block: no production reader relies on the synchronous release, and explicit — crew-pr-reviewer (r141) |
bolichen97
left a comment
There was a problem hiding this comment.
Approving at bc68d53. This replaces my earlier change request on 7a61c27.
That request was about the exit path: with the daemon reaper still alive, a finalizer fired by the atexit weakref sweep only queued its end, and the process could exit before the reaper ran it, leaving the lease held. This head fixes it:
_release_queued_leases_at_exitis registered withatexit. It sets_EXITING, closes the lease of every projection end still on_COLLECTED_ENDS, and gives an end the reaper already took up to_EXIT_REAPER_WAIT_SECS(2 s) to finish.- In
_end_projection, a finalizer-driven end off the loop is queued only while the reaper is alive and_EXITINGis not set. Otherwise it closes the lease inline and leaves the prune to the next boot drain. release()now detaches the finalizer and calls the end withfrom_release=True, so only the explicit release runs the full end inline. This replaces thegc.callbackscollecting-thread check.test_a_projection_held_until_exit_releases_its_lease_with_the_reaper_aliveruns a real interpreter whose reaper is alive but busy at exit, and checks the lease directory is empty afterwards. The unit tests cover the hook draining a queued end, waiting for an end the reaper took, and the inline release once exit began.
Checks at this head: all green, no pending checks, no other open change request.
Problem / Motivation
Goal: Stop the skill-view projection finalizer from deadlocking the thread that drops it, which crashed an xdist worker on Windows CI.
A dropped projection's lease finalizer (
_end_projection) only handed its work to the reaper thread when the CYCLIC collector was running. A plain refcount-zero drop runs the sameweakref.finalizecallback inline on whichever thread dropped the last reference, and off the event loop that thread ran the lease release + eager prune directly. The prune calls the security path-resolution subprocess pool (aThreadPoolExecutor). When the drop fired while that same thread was already insideThreadPoolExecutor.submit— constructing theFuturewhose allocation freed the projection, under_global_shutdown_lock— the end re-entered the executor and waited on a lock its own thread held, forever. On Windows CI this hung and killed the xdist worker, crashing the unrelatedtest_a_recovered_source_rebinds_after_being_retained(its rebuild resolves sensitive paths through that pool).Observed on PR #20523 (gw3) and PR #21795 (gw2):
worker 'gw2' crashed while running test/test_mcp_rebuild_reconsumption.py::...::test_a_recovered_source_rebinds_after_being_retained. Also tracked as the timeout in #21661.Why it matters
A refcount-driven projection drop happens constantly in normal operation, so any drop that lands inside the path-resolve executor can hang a gateway thread forever. On CI it flakes unrelated Windows shards (worker-restart is disabled, so one crash aborts the whole run); in production it is a latent deadlock of a session thread.
Not a goal
release()path keep their existing behavior.What changed (motivation → approach → change)
Fixes #20565
Fixes #20436
test_a_recovered_source_rebinds_after_being_retained(Flaky: test_a_recovered_source_rebinds_after_being_retained crashes its xdist worker on Windows #20565), a hang inside aThreadPoolExecutor.submitreached re-entrantly from aweakref.finalizecallback._end_projectiondeferred to the reaper only for the cyclic collector (_COLLECTING_THREAD); a refcount-zero finalize ran inline on an arbitrary thread that could already hold the executor's non-reentrant lock the prune re-enters.atexithandlers run yet stopped by finalization before it reaches a queued end. Anatexithook (_release_queued_leases_at_exit) sets_EXITING, releases the lease of every end still queued, and gives an end the reaper already took up to 2 s to finish; once_EXITINGis set, or when_reaper_alive()is false, a finalizer releases the lease INLINE. Whichever order the hook and the weakref sweep run in, the lock file is not left behind for a projection the shared runtime never explicitly released; the prune is dropped for the next boot drain. An EXPLICITrelease()still runs the end inline off the loop, told apart from a dropped finalizer by a direct call argument, not process state: itdetach()es theweakref.finalizeand calls_end_projectionwithfrom_release=True, while the finalizer path uses the defaultfrom_release=False. No thread-local and no cross-module flag import, so the parts layer order is untouched.Backwards compatibility
Compatible: the only observable change is that a dropped projection's lease release + prune now completes on the reaper thread rather than synchronously on the dropping thread (it already did for the cyclic-collector case). Explicit
release()is unchanged. The prune is idempotent and bounded, so deferral never loses cleanup.Tests
test_native_skill_projection_rewrites.py— newtest_a_projection_dropped_off_the_loop_ends_on_the_reaper_not_the_dropping_threadforces the end to fire on a thread holding a lock the end would re-enter and asserts it runs on the reaper (red on parent logic, green with the fix); existing cyclic-collector and release tests still pass.test_native_skill_projection_rewrites.py— interpreter-exit coverage:test_a_projection_held_until_exit_releases_its_lease_with_the_reaper_aliveruns a real interpreter that exits holding a projection while the reaper is alive but busy, and asserts the lease file is gone after exit (red without the exit hook);test_an_off_loop_drop_once_exit_began_releases_the_lease_inline_with_the_reaper_alive,test_the_exit_hook_releases_the_lease_of_an_end_still_queued(a queued non-end item is left uncalled) andtest_the_exit_hook_waits_for_the_end_the_reaper_already_tookeach fail on a mutant that drops the matching piece.test_native_skill_projection_alias_reclaim.py,test_native_skill_projection_lease_scan.py,test_skill_view_relaunch_harness.py,test_skill_projection_facade.py,test_mcp_rebuild_reconsumption.py— all pass at-n0/-n4.isort/flake8/black --check/mypyclean on changed source.docs/system-specs/modules/acp-client.mdupdated to match (AGENTS.md owning-spec rule).Pattern harvest
Rule candidate: review-prompt / agents-md
Pattern: a
weakref.finalize/ GC finalizer that runs heavy, lock-taking work (file locks, a subprocess-poolsubmit) INLINE on the thread that dropped the last reference. A finalizer can fire mid-allocation on any thread, inside a non-reentrant stdlib lock, so finalizer-driven cleanup must be handed to a dedicated worker (a reaper queue), never run inline — guarding only the cyclic collector (gc.callbacks) misses the refcount-zero drop path.