Repository navigation
fix(dashboard): stop speculative session starts when the gateway shuts down - #21552
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, check that [DESIGN-REVIEWED] bd46425 Description read[DESCRIPTION-READ] 4bc099e6bf4844881781c001b80a6476a095c8b9eb74930d4d48063c4afa4a48 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Check that a start cancelled mid-spawn (5 s SIGTERM grace plus 2 s reap) still finishes inside the 10 s graceful-shutdown cap it is now gathered under. What this change shipsInventory (3 items) — 3 justifiedIntent: shutting the gateway down never leaves behind an agent process spawned by a speculative session start. This is a FIX (Closes #18266, and each new test fails on base).
[FIRST-PRINCIPLES-REVIEWED] bd46425 Description read[DESCRIPTION-READ] 101c047cfaad63e408775f6a7e0a1d97c697309c64961426ea584015832dbdfa That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Frontend Tests (2) is red on |
|
Real Adapter Contract Tests is red on |
|
Two more reds on this head, neither from this diff:
|
|
rebutted: every production route into
|
|
rebutted: the window spawns nothing, so leaving it unawaited leaks no process.
|
b4106ec to
bd46425
Compare
|
rebutted: the event is set before
|
|
rebutted: gathering the start does not extend the 10 s cap, and the cleanup is bounded the same way
|
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix; gateway shutdown now cancels pending speculative session starts and the eager spawn returns once shutdown has begun, so no agent process outlives the teardown, with tests.
Problem / Motivation
Goal: A gateway shutdown ends every speculative session start it finds, so none spawns an agent mid-teardown or outlives the gateway.
test/integration/test_governance.py::test_a_non_durable_memory_wait_is_named_as_onefails now and then in its teardown:The test creates a chat slot. The slot schedules an eager spawn, which waits 1.5 s and then starts an agent. The test ends sooner than that, so the start is still pending when the gateway stops.
Why it matters
Every slot create, focus or project switch schedules one of these starts, so any shutdown can race one. A start that wakes during the shutdown spawns an agent process the shutdown does not own. Nothing is registered yet, so
sessions.close_all()cannot reach it. Only the start's own failed-start cleanup kills it: a shielded kill with a 5 s SIGTERM grace, a 2 s reap and more. In the integration layer that cleanup only begins at the harness's last-resort reap, and on a slow runner it overran that reap's 10 s bound (#16387 and #17613 both went red this way). In production the same start races the 10 s graceful-shutdown cap.Not a goal
What changed (motivation → approach → change)
Symptom: a cancelled
_eager_spawntask lived past the harness's 10 s reap bound. Cause: nothing in the shutdown stopped it. With a reap-time diagnostic on the named test, the start was still in its debounce when teardown began. It woke during the shutdown and was insideAcpRuntime._spawn_admittedwhen the harness reaped it. Cancelled there, it runs its shielded failed-start cleanup to the end, and that cleanup's length depends on the host.Two changes stop the start, in
dashboard/chat_runner.pyandslack/gateway_runtime/exit_path.py:_eager_spawnchecksshutdown_eventright before admission. A stop that arrives while the start is still reading its bindings now ends it before any agent process exists. No await before that check starts an agent process._shutdown()calls the newstop_eager_spawns(state)before its first await. It cancels every slot's pending start, then awaits them in the same gather assessions.close_all(). A start caught mid-spawn kills its own process inside the graceful budget, not after it.With both, the diagnostic shows the start standing down after its debounce. The reap finds no eager task, and teardown dropped from about 2.3 s to 1.6 s.
Backwards compatibility
Compatible: only the shutdown path changes; a running gateway starts sessions exactly as before.
Removes nothing: the diff adds one check and one helper; no key, route, field or caller goes away.
Tests
test/test_eager_spawn_shutdown.py::test_a_shutdown_during_the_reads_stands_down_before_admission[stopping]: a stop that lands during the last read before admission. Neither admission nor the spawn runs.[running]is the same path with no stop, and both run.test/test_eager_spawn_shutdown.py::test_stop_eager_spawns_cancels_only_pending_starts: only an unfinished task is cancelled and returned.test/test_slack_gateway_more_coverage.py::TestShutdownExtras::test_a_pending_eager_spawn_ends_inside_the_shutdown: a slot's debouncing start is cancelled by the time_shutdown()returns.Each new test fails with its fix reverted: with the check disabled,
[stopping]fails on the awaited admission; withexit_path.pyatorigin/main, the shutdown test fails oncancelled(). Determinism: the new tests ran 20 times at-n0, and the touched files ran 10 times at-n 4in shuffled order (147 tests per run), all green.Manual verification
KIROCREW_INTEGRATION=1 pytest -n 2 test/integration/test_governance.py, three times: 6 passed each._snapshot_descendants. After: the start ended in its debounce and the reap found no eager task. The slow cleanup itself did not reproduce on this host (forced cancels at the handshake unwound in about 0.2 s). The fix removes the race rather than shortening the cleanup.Screenshots / video
N/A: no frontend change.
Related Issues
Closes #18266
Pattern harvest
Rule candidate: review-prompt
Pattern: a fire-and-forget task that can start a child process needs a shutdown owner that cancels and awaits it, or a stop check before the spawn.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)