Skip to content

fix(chat): replace a slot-switch lock bound to a closed event loop - #11481

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/slot-switch-lock-loop
Sep 17, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/slot-switch-lock-loop

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

test/test_dashboard_chat.py::TestRunChatRefusalFallback::test_pick_during_swap_set_model_await_is_not_reverted fails intermittently on the Windows backend shards with RuntimeError: <asyncio.locks.Lock ...> is bound to a different event loop, most recently on a docs-only PR (#11409, run). The test itself is sound; the lock it receives is stale.

Why it matters

Every PR based on main can go red on a shard it did not touch, and a fork PR cannot rerun a job, so the only way past the flake is a fresh push and another maintainer approval click. The same mechanism would bite any embedder that runs the gateway on more than one event loop in one process.

What changed (motivation → approach → change)

slot_switch_session_lock in llm_helpers.py caches one asyncio.Lock per session key in a process-wide WeakValueDictionary. An asyncio.Lock records the loop it is first contended on and raises when contended from any other loop. pytest-asyncio gives every test its own loop, and many tests in test_dashboard_chat.py resolve the same session key for a slot named s1; when an earlier test on the same xdist worker contends the lock and leaves a reference alive past its loop, the next test on that worker receives a lock it cannot use. Test order across workers decides whether such a pair meets, which is why the failure is intermittent and shard-specific.

The helper now checks the cached lock's bound loop against the running loop and replaces a lock bound elsewhere. An unbound lock (never contended) is usable from any loop and is returned as before; with no running loop the caller is synchronous setup code and the cached lock is returned unchanged. Holders on the old loop keep their lock, and the two loops cannot contend with each other in any case, so no serialisation is lost. The loop-binding check reads the mixin's _loop attribute, which is how asyncio.Lock has recorded its loop since Python 3.10.

An alternative was an autouse fixture in test_dashboard_chat.py clearing the cache between tests. That hides the symptom in one module and leaves the helper unusable across loops for every other caller, so the fix went into the helper.

Tests

test/test_slot_switch_session_lock_loops.py (new):

  • test_lock_bound_on_a_closed_loop_is_replaced_for_the_next_loop: binds a lock on one loop, keeps it alive, then contends the same key on a second loop. On the old helper this raises the exact RuntimeError from the CI log; on the new helper the second loop receives a fresh lock and the cache serves the replacement.
  • test_lock_is_reused_within_one_loop: two lookups on the same running loop return the same lock, so per-session serialisation is unchanged.
  • test_unbound_lock_is_reused_across_loops: a lock that was never contended is shared across loops, so the check does not churn locks handed out by synchronous setup code.

Also run locally: test_chat_slot_switch_atomicity.py, test_slot_reset_interleaving_seam.py, test_switch_handlers_slot_identity.py, test_llm_helpers.py and TestRunChatRefusalFallback (308 passed); black, isort, flake8, mypy, comment-history, brand, agent-sdk-boundary gates pass.

Manual verification

N/A: the unit test reproduces the CI failure verbatim on the old code.

Related Issues

Fixes #11477

Pattern harvest

Rule candidate: review-prompt
Pattern: a process-wide cache of asyncio primitives (Lock, Event, Condition, Semaphore) keyed by anything other than the running loop.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

slot_switch_session_lock caches one asyncio.Lock per session key in a
process-wide WeakValueDictionary. A lock binds to the loop it is first
contended on, so a cached lock kept alive past its loop raises
RuntimeError when the next loop contends it. The helper now replaces a
lock bound to a loop other than the running one.

Fixes kirodotdev#11477
@jeeshofone
jeeshofone requested review from a team and chenmingwei23 September 17, 2026 07:26
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 17, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 8968b6d1d94011d3e259f4a5f9643b2fdb347717 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

Suggestions

  • Keying the cache per running loop (loop → key → lock) would also hold when two loops are live concurrently — replace-on-mismatch can ping-pong the cache there and silently lose same-loop serialization where the old code at least failed loudly — and would drop the peek at the private _loop attribute. Fine as a follow-up if multi-loop embedding becomes real.

[DESIGN-REVIEWED] 8968b6d

@github-actions

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 8968b6d1d94011d3e259f4a5f9643b2fdb347717 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 8968b6d

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — 🟡 CONCERNS

Premise-level review of 8968b6d1d94011d3e259f4a5f9643b2fdb347717 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The loop-check rests on a zero-consumer premise and a private attribute; _campaign_transition_lock already solves this by keying the cache on the running loop.

Not justified as shipped

  • Item 2 — duplicate of src/kiro_crew/apps/builtins/auto_research/handlers.py:981: "with no running loop the caller is synchronous setup code" — counted zero such callers (all 8 call sites, chat_runner.py:1496/1627 and 6 enter_async_context sites in chat_handlers.py, run under a loop), and _campaign_transition_lock already makes a lock cache loop-safe with public APIs by keying on get_running_loop(), no _loop peek.
    Subtraction: delete _bound_to_other_loop and the no-loop branch; key _slot_switch_session_locks by running loop as _campaign_transition_lock does, and drop test_unbound_lock_is_reused_across_loops, which pins cross-loop sharing nobody exercises.
    Clears when: a named caller with no running loop (or relying on cross-loop reuse of an unbound lock), or the replacement made.
  • Item 1 — symptom-level (point patch): grep WeakValueDictionary[str, asyncio.Lock] finds 2 caches; _autocompact_txn_locks (chat_handlers.py:7487) keeps the identical defect the PR's own "Pattern harvest" names.
    Clears when: the same loop-safety applied to _autocompact_txn_locks, or the description states this cache is deliberately deferred.

What this change ships

Inventory (3 items) — 1 justified

Intent: stop an intermittent Windows-shard CI failure caused by a cached slot-switch lock bound to a closed event loop — a FIX.

  1. A cached slot-switch lock bound to another loop is replaced instead of raising — symptom-level (point patch: 1 unfixed sibling, chat_handlers.py:7487)
  2. Locks never contended, or callers with no running loop, keep the cached lock across loops — duplicate of auto_research/handlers.py:981 (loop-keyed cache does the job without private _loop; zero counted consumers of these branches)
  3. New test file pinning replacement, same-loop reuse, and cross-loop reuse — justified

[FIRST-PRINCIPLES-REVIEWED] 8968b6d

@github-actions

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed 8968b6d1d94011d3e259f4a5f9643b2fdb347717 via the fork AI-review pipeline; updated in place on each push.

Review details

I have enough to judge both candidates from the diff, the surrounding code, and the well-known asyncio.Lock semantics.

Candidate 1 (replacing a live cross-loop lock drops mutual exclusion): For serialization to be lost you need two event loops concurrently driving the same wire session key. The realistic triggers named (per-test loop, embedder on a fresh loop) are sequential asyncio.run calls — the old loop is closed, so nothing on it can contend. Where two loops truly ran concurrently and both contended the same asyncio.Lock, the pre-change code would already raise RuntimeError on the second loop — so there was never a working cross-loop serialization guarantee to regress. No concrete concurrent-two-loop path exists; (a) is speculative. Dropped.

Candidate 2 (depends on private _loop): On every CPython version this repo targets (3.10–3.13), _LoopBoundMixin defines _loop = None as a class attribute set on first contention, so getattr(lock, "_loop", None) is correct and never absent. A hypothetical future rename is forward-compat fragility, not a live defect with a concrete input. Dropped.

No self-originated finding survives the same bar — the guard is synchronous, reuses same-loop and unbound locks, and only replaces locks bound to a different (closed/other) loop, which matches the actual binding-on-contention semantics.

No findings.

[OPUS-REVIEWED] 8968b6d

@iamwhatever
iamwhatever merged commit 2ca46f4 into kirodotdev:main Sep 17, 2026
80 of 82 checks passed
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Sep 17, 2026
@jeeshofone
jeeshofone deleted the fix/slot-switch-lock-loop branch September 17, 2026 07:40
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…irodotdev#11481)

slot_switch_session_lock caches one asyncio.Lock per session key in a
process-wide WeakValueDictionary. A lock binds to the loop it is first
contended on, so a cached lock kept alive past its loop raises
RuntimeError when the next loop contends it. The helper now replaces a
lock bound to a loop other than the running one.

Fixes kirodotdev#11477
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test_dashboard_chat refusal-fallback test fails on Windows shard: slot switch lock bound to a different event loop

2 participants