Repository navigation
fix(dashboard): stop _run_chat crashing when GC closes an orphaned turn - #7709
kyleseaman wants to merge 2 commits into
Conversation
Design Review (Fable 5, fork) — ✅ PASSDesign-level review of The design checks out: the guard reuses the existing Design-Verdict: PASS Real, reachable off-loop crash fixed at the right layer, keyed on existing serving-loop identity machinery, with a genuine falsified reproduction. Suggestions
[DESIGN-REVIEWED] c2a5448 |
First Principles Review (Fable 5, fork) — ✅ PASSPremise-level review of All checks done. The base tree confirms First-Principles-Verdict: PASS Shipped guard is stricter than described: it also refuses dispatch on a foreign running loop — undeclared; confirm that refusal is the intended behavior. Not justified as shipped
What this change shipsIntent: stop an orphaned
Verified by reading: provenance is #7643 plus a reproduction test that fails on base by construction; the guard sits at mechanism level inside the two shared functions, covering all 11 call sites (grepped [FIRST-PRINCIPLES-REVIEWED] c2a5448 |
Opus 4.8 Review (fork) — ✅ no blocking findingsReviewed Review detailsBoth candidates concern the off-loop / foreign-loop paths that the guard exists to handle, and both were flagged low-confidence by the discovery pass. Working through them against the actual code: CANDIDATE 1 — The mis-latching is real behavior of the CANDIDATE 2 — The off-loop branch does emit No new grounded finding surfaced while falsifying these. No findings. [OPUS-REVIEWED] c2a5448 |
GPT 5.6 Review (fork) — 🔴 changes requested (blocking)Reviewed 1 of 1 blocking finding(s) are security-class and were withheld from adjudication, so the blocking verdict stands. BLOCKING -- src/kiro_crew/dashboard/chat_runner.py:5218 -- Lazy binding accepts a foreign loop when no serving loop was pre-bound Adjudication (Opus 4.8) — is blocking on each finding proportionate?The adjudicable input block is empty (0 findings). One fenced finding, F1. F1 analysis. The claim is real as code: But every condition that reaches harm is test-only and they compound:
Evidence record is complete from code opened this run → FLAG. 🏷️ Fenced finding(s) machine-flagged as likely edge caseThe security fence keeps these findings blocking regardless of adjudication; the only clearance path is a human override recorded by a repository writer, who must independently verify a rationale before recording it — it is machine-authored, and a wrong override on a security-class finding ships exactly the class the fence exists to stop. (This lane's comment deliberately carries no override command.)
|
15e433b to
0d85d69
Compare
Open PR relationship auditThis is a consolidated, point-in-time code-level audit note. It compares complete merge-base diffs and current/merged code; it does not treat a shared topic as duplication or partial coverage as completion. Relationship findings
No PR, Issue, label, branch, or review state was changed by the relationship-note portion of this audit. |
0d85d69 to
db35753
Compare
db35753 to
71f0885
Compare
PR kirodotdev#7643 fixed _bounded_turn's half of this GeneratorExit race but flagged a second, separate defect in _run_chat's own queue-drain tail as "tracked separately" -- this is that fix. _run_chat's single finally block drains the queue via _start_next_queued_turn (spawn_guarded_turn -> asyncio.create_task) and falls back to _finish_queue_cycle (four more bare asyncio.create_task calls: synthesis, auto-title, title-refresh, session summary). Both assume a running loop. When a turn's coroutine is orphaned as a background task and never joined -- a test/harness bug, since spawn_guarded_turn always registers its task and finish_turn_task always retrieves it in production -- the GC eventually finalizes it via close(), throwing GeneratorExit at whatever await it was suspended on. That unwinds cleanly into the finally block until it hits one of these create_task calls with no loop to schedule onto, which raises RuntimeError and replaces the in-flight GeneratorExit -- surfacing as an unraisable exception blamed on whatever test the GC happened to fire in. Add _loop_is_running() and use it to make _start_next_queued_turn bail out before touching the queue when off-loop, and fold the same check into _finish_queue_cycle's will_synthesize and its other three create_task sites. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Reuse DashboardState._running_loop (state.py) instead of a duplicate _loop_is_running helper -- both call sites already take state, and the existing method is the same try/get_running_loop/except pattern. Also fix a test-hygiene issue: the deliberately-abandoned Task in the end-to-end reproduction test now silences its own "Task was destroyed but it is pending!" destructor log (task._log_destroy_pending = False) instead of letting it surface at some later, unpredictable GC moment. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
71f0885 to
c2a5448
Compare
|
Rebased onto main Conflicts: none, clean rebase. Both commits replayed unchanged; the diff is still 2 files, +278/-9, and Gates run locally (changed files only): Please review the rebased branch. Note that a maintainer push makes the maintainer the last pusher, so under this repo's last-push rule a second approver is needed. Reply here if anything looks wrong. |
|
Kiro Crew [operator: chenmingwei23#de330d0c] Triaged as solvable and claimed for the drive-to-green pipeline. No conflict, and Opus 4.8 came back with no blocking findings. The single blocker Opus adjudication flagged it as a likely edge case, since production binds the Your actual fix (surviving GC finalization of an orphaned turn) is not being |
|
Thank you for this fix, @kyleseaman — the diagnosis was spot on, and the loop-identity reasoning (guarding on the serving loop rather than on "is any loop running") was exactly the right distinction. We're closing this one as superseded by main, not rejected. Since you opened it, A note for maintainers while this area is fresh: the end-to-end "close an orphaned Your authorship is preserved in git history, and you're credited in the README Contributors list. Thanks again for the careful work. 🙏 |
Problem / Motivation
PytestUnraisableExceptionWarningnoise when a_run_chatchat-turn coroutine is orphaned as a background task and later finalized by the garbage collector after its owning event loop has already closed.Why it matters
This is the second half of the investigation that produced #7643. That PR's own description flags this exact defect ("
_run_chat's own queue-drain tail") as "tracked separately" — a real, reachable bug in the queue-drain / auto-title / synthesis dispatch path that nobody had fixed yet. Left in place, any orphaned_run_chattask (today: test teardown that doesn't join a spawned turn; tomorrow: any code path with the same hygiene bug) crashes with an unraisableRuntimeErrormisattributed to whatever test the GC happens to run in — exactly the kind of noise that corrupted PR #5411's CI signal in the first place.What changed (motivation → approach → change)
Symptom:
_run_chat's tail, inside its singlefinally:block, drains the queue via_start_next_queued_turn(spawn_guarded_turn→asyncio.create_task) and falls back to_finish_queue_cycle(four more bareasyncio.create_taskcalls: synthesis, auto-title, title-refresh, session summary).Root cause: all of these assume a running event loop. When the coroutine is resumed by
close()off-loop — the GC finalizing an orphaned, never-joined task after its owning loop already closed —asyncio.create_task's internalget_running_loop()raisesRuntimeError, which replaces the in-flightGeneratorExitmid-unwind and surfaces as an unraisable exception.Change: added
_loop_is_running()tochat_runner.pyand used it to make_start_next_queued_turnbail out withFalsebefore touching the queue when there's no running loop, and folded the same check into_finish_queue_cycle'swill_synthesizecondition and its other threecreate_tasksites — so an off-loop unwind degrades to "skip the background work" instead of raising.Tests
Added
TestQueueDrainOffLoopGuardintest/test_dashboard_chat.py: 5 tests pin the guard directly (monkeypatchedget_running_loop, no real event loop needed) plus one end-to-end reproduction — a real_run_chatcoroutine parked on a genuinely-unresolvedawait(standing in for a stalled ACP call), its owning loop closed out from under it, then.close()'d directly.Verified the reproduction is real, not a tautology: reverted the source fix and confirmed all 6 tests fail, with the integration test raising the exact reported
RuntimeError: no running event loopatspawn_guarded_turn→asyncio.create_task. Restored the fix; all pass clean under-W error::pytest.PytestUnraisableExceptionWarning, 20/20 repeat runs stable (fully deterministic — no GC-timing dependence).Also ran the full
test_dashboard_chat.py(724 tests),test_turn_dispatch.py,test_chat_runner_coverage.py,test_pending_title.py,test_session_summary_generate.py, and queue-drain test files — 1600+ tests, no regressions. black / isort / flake8 / mypy (--platform linux) clean on touched files.Manual verification
N/A — unit coverage sufficient. This is an internal async-cleanup path with no UI and no externally observable behavior to click through; the reproduction test above exercises the real failure mechanism directly.
Related Issues
Companion to #7643 — fixes the
_run_chathalf of the same investigation. Independent of it either way: this PR doesn't touchturn_dispatch.py, and neither PR depends on the other to be correct.Pattern harvest
Rule candidate: review-prompt
Pattern: a bare
asyncio.create_task/ensure_futurereachable from a coroutine'sfinally:/cleanup path assumes a running event loop — aGeneratorExit-driven off-loop unwind (the GC finalizing an orphaned task after its loop already closed) turns the implicitget_running_loop()inside it into an unraisableRuntimeErrorinstead of letting the close finish cleanly. Two independent instances already exist in this codebase:_bounded_turn(#7643) and this PR's_finish_queue_cycle/_start_next_queued_turn. Worth a review-prompt heuristic flagging newcreate_task/ensure_futurecalls added inside afinally:block for a running-loop guard.Checklist
feat|fix|docs|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
🤖 Generated with Claude Code