Repository navigation
fix(chat): replace a slot-switch lock bound to a closed event loop - #11481
Conversation
slot_switch_session_lock caches one asyncio.Lock per session key in a process-wide WeakValueDictionary. A lock binds to the loop it is first contended on, so a cached lock kept alive past its loop raises RuntimeError when the next loop contends it. The helper now replaces a lock bound to a loop other than the running one. Fixes kirodotdev#11477
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. Suggestions
[DESIGN-REVIEWED] 8968b6d |
GPT 5.6 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
First Principles Review (Fable 5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The loop-check rests on a zero-consumer premise and a private attribute; Not justified as shipped
What this change shipsInventory (3 items) — 1 justifiedIntent: stop an intermittent Windows-shard CI failure caused by a cached slot-switch lock bound to a closed event loop — a FIX.
[FIRST-PRINCIPLES-REVIEWED] 8968b6d |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsI have enough to judge both candidates from the diff, the surrounding code, and the well-known Candidate 1 (replacing a live cross-loop lock drops mutual exclusion): For serialization to be lost you need two event loops concurrently driving the same wire session key. The realistic triggers named (per-test loop, embedder on a fresh loop) are sequential Candidate 2 (depends on private No self-originated finding survives the same bar — the guard is synchronous, reuses same-loop and unbound locks, and only replaces locks bound to a different (closed/other) loop, which matches the actual binding-on-contention semantics. No findings. [OPUS-REVIEWED] 8968b6d |
…irodotdev#11481) slot_switch_session_lock caches one asyncio.Lock per session key in a process-wide WeakValueDictionary. A lock binds to the loop it is first contended on, so a cached lock kept alive past its loop raises RuntimeError when the next loop contends it. The helper now replaces a lock bound to a loop other than the running one. Fixes kirodotdev#11477
Problem / Motivation
test/test_dashboard_chat.py::TestRunChatRefusalFallback::test_pick_during_swap_set_model_await_is_not_revertedfails intermittently on the Windows backend shards withRuntimeError: <asyncio.locks.Lock ...> is bound to a different event loop, most recently on a docs-only PR (#11409, run). The test itself is sound; the lock it receives is stale.Why it matters
Every PR based on main can go red on a shard it did not touch, and a fork PR cannot rerun a job, so the only way past the flake is a fresh push and another maintainer approval click. The same mechanism would bite any embedder that runs the gateway on more than one event loop in one process.
What changed (motivation → approach → change)
slot_switch_session_lockinllm_helpers.pycaches oneasyncio.Lockper session key in a process-wideWeakValueDictionary. Anasyncio.Lockrecords the loop it is first contended on and raises when contended from any other loop. pytest-asyncio gives every test its own loop, and many tests intest_dashboard_chat.pyresolve the same session key for a slot nameds1; when an earlier test on the same xdist worker contends the lock and leaves a reference alive past its loop, the next test on that worker receives a lock it cannot use. Test order across workers decides whether such a pair meets, which is why the failure is intermittent and shard-specific.The helper now checks the cached lock's bound loop against the running loop and replaces a lock bound elsewhere. An unbound lock (never contended) is usable from any loop and is returned as before; with no running loop the caller is synchronous setup code and the cached lock is returned unchanged. Holders on the old loop keep their lock, and the two loops cannot contend with each other in any case, so no serialisation is lost. The loop-binding check reads the mixin's
_loopattribute, which is howasyncio.Lockhas recorded its loop since Python 3.10.An alternative was an autouse fixture in
test_dashboard_chat.pyclearing the cache between tests. That hides the symptom in one module and leaves the helper unusable across loops for every other caller, so the fix went into the helper.Tests
test/test_slot_switch_session_lock_loops.py(new):test_lock_bound_on_a_closed_loop_is_replaced_for_the_next_loop: binds a lock on one loop, keeps it alive, then contends the same key on a second loop. On the old helper this raises the exactRuntimeErrorfrom the CI log; on the new helper the second loop receives a fresh lock and the cache serves the replacement.test_lock_is_reused_within_one_loop: two lookups on the same running loop return the same lock, so per-session serialisation is unchanged.test_unbound_lock_is_reused_across_loops: a lock that was never contended is shared across loops, so the check does not churn locks handed out by synchronous setup code.Also run locally:
test_chat_slot_switch_atomicity.py,test_slot_reset_interleaving_seam.py,test_switch_handlers_slot_identity.py,test_llm_helpers.pyandTestRunChatRefusalFallback(308 passed); black, isort, flake8, mypy, comment-history, brand, agent-sdk-boundary gates pass.Manual verification
N/A: the unit test reproduces the CI failure verbatim on the old code.
Related Issues
Fixes #11477
Pattern harvest
Rule candidate: review-prompt
Pattern: a process-wide cache of
asyncioprimitives (Lock,Event,Condition,Semaphore) keyed by anything other than the running loop.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)