Skip to content

fix(dashboard): stop speculative session starts when the gateway shuts down - #21552

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/eager-spawn-teardown-18266
Oct 17, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/eager-spawn-teardown-18266

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway shutdown ends every speculative session start it finds, so none spawns an agent mid-teardown or outlives the gateway.

test/integration/test_governance.py::test_a_non_durable_memory_wait_is_named_as_one fails now and then in its teardown:

RuntimeError: 1 task(s) the boot started did not end within 10.0s of cancellation: ['Task-1035 (<coroutine object _eager_spawn at 0x...>)']

The test creates a chat slot. The slot schedules an eager spawn, which waits 1.5 s and then starts an agent. The test ends sooner than that, so the start is still pending when the gateway stops.

Why it matters

Every slot create, focus or project switch schedules one of these starts, so any shutdown can race one. A start that wakes during the shutdown spawns an agent process the shutdown does not own. Nothing is registered yet, so sessions.close_all() cannot reach it. Only the start's own failed-start cleanup kills it: a shielded kill with a 5 s SIGTERM grace, a 2 s reap and more. In the integration layer that cleanup only begins at the harness's last-resort reap, and on a slow runner it overran that reap's 10 s bound (#16387 and #17613 both went red this way). In production the same start races the 10 s graceful-shutdown cap.

Not a goal

  • Changing the runtime's failed-start cleanup or its kill timeouts. They exist so a cancelled start never leaks its process.
  • Changing the integration harness or its 10 s reap bound.
  • Refusing new eager-spawn signals that arrive after shutdown starts. The check before admission already stops those from spawning.

What changed (motivation → approach → change)

Symptom: a cancelled _eager_spawn task lived past the harness's 10 s reap bound. Cause: nothing in the shutdown stopped it. With a reap-time diagnostic on the named test, the start was still in its debounce when teardown began. It woke during the shutdown and was inside AcpRuntime._spawn_admitted when the harness reaped it. Cancelled there, it runs its shielded failed-start cleanup to the end, and that cleanup's length depends on the host.

Two changes stop the start, in dashboard/chat_runner.py and slack/gateway_runtime/exit_path.py:

  • _eager_spawn checks shutdown_event right before admission. A stop that arrives while the start is still reading its bindings now ends it before any agent process exists. No await before that check starts an agent process.
  • _shutdown() calls the new stop_eager_spawns(state) before its first await. It cancels every slot's pending start, then awaits them in the same gather as sessions.close_all(). A start caught mid-spawn kills its own process inside the graceful budget, not after it.

With both, the diagnostic shows the start standing down after its debounce. The reap finds no eager task, and teardown dropped from about 2.3 s to 1.6 s.

Backwards compatibility

Compatible: only the shutdown path changes; a running gateway starts sessions exactly as before.

Removes nothing: the diff adds one check and one helper; no key, route, field or caller goes away.

Tests

  • test/test_eager_spawn_shutdown.py::test_a_shutdown_during_the_reads_stands_down_before_admission[stopping]: a stop that lands during the last read before admission. Neither admission nor the spawn runs. [running] is the same path with no stop, and both run.
  • test/test_eager_spawn_shutdown.py::test_stop_eager_spawns_cancels_only_pending_starts: only an unfinished task is cancelled and returned.
  • test/test_slack_gateway_more_coverage.py::TestShutdownExtras::test_a_pending_eager_spawn_ends_inside_the_shutdown: a slot's debouncing start is cancelled by the time _shutdown() returns.

Each new test fails with its fix reverted: with the check disabled, [stopping] fails on the awaited admission; with exit_path.py at origin/main, the shutdown test fails on cancelled(). Determinism: the new tests ran 20 times at -n0, and the touched files ran 10 times at -n 4 in shuffled order (147 tests per run), all green.

Manual verification

  • KIROCREW_INTEGRATION=1 pytest -n 2 test/integration/test_governance.py, three times: 6 passed each.
  • The named test ran with a throwaway plugin that prints each leftover task's await chain at reap. Before: the eager task was alive at reap, inside _snapshot_descendants. After: the start ended in its debounce and the reap found no eager task. The slow cleanup itself did not reproduce on this host (forced cancels at the handshake unwound in about 0.2 s). The fix removes the race rather than shortening the cleanup.

Screenshots / video

N/A: no frontend change.

Related Issues

Closes #18266

Pattern harvest

Rule candidate: review-prompt
Pattern: a fire-and-forget task that can start a child process needs a shutdown owner that cancels and awaits it, or a stop check before the spawn.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision bd4642508a63787b6b7684a972305a7897840170 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of bd4642508a63787b6b7684a972305a7897840170 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, check that shutdown_event is set before _shutdown() runs on every exit path, including the integration harness. Otherwise a start scheduled after stop_eager_spawns is not stopped by the new check.

[DESIGN-REVIEWED] bd46425

Description read

[DESCRIPTION-READ] 4bc099e6bf4844881781c001b80a6476a095c8b9eb74930d4d48063c4afa4a48

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of bd4642508a63787b6b7684a972305a7897840170 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check that a start cancelled mid-spawn (5 s SIGTERM grace plus 2 s reap) still finishes inside the 10 s graceful-shutdown cap it is now gathered under.

What this change ships

Inventory (3 items) — 3 justified

Intent: shutting the gateway down never leaves behind an agent process spawned by a speculative session start. This is a FIX (Closes #18266, and each new test fails on base).

  1. A speculative start that wakes after shutdown has begun gives up before admission, so no agent spawns — justified
  2. Gateway shutdown cancels pending speculative starts before its first await and waits for them alongside session teardown — justified
  3. New helper stop_eager_spawns(state) in dashboard/chat_runner.py, with one consumer, exit_path._shutdown — justified

[FIRST-PRINCIPLES-REVIEWED] bd46425

Description read

[DESCRIPTION-READ] 101c047cfaad63e408775f6a7e0a1d97c697309c64961426ea584015832dbdfa

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed bd4642508a63787b6b7684a972305a7897840170 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] bd46425

Verdict parsed from the review's SHA-scoped output markers for commit bd4642508a63787b6b7684a972305a7897840170.

False positive or not applicable? A repository writer can comment:
/ai-review override fable bd4642508a63787b6b7684a972305a7897840170: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of bd4642508a63787b6b7684a972305a7897840170 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bd46425

False positive or not applicable? A repository writer can comment:
/ai-review override gpt bd4642508a63787b6b7684a972305a7897840170: <one-sentence reason>

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Frontend Tests (2) is red on src/test/MarkdownRenderer.domGolden.test.tsx > MarkdownRenderer golden DOM > renders the prose document (katex snapshot mismatch). This PR changes no frontend file; the same snapshot breaks on main after the katex 0.18.2 class rename, and #21536 updates it. This PR waits for #21536 to merge, then rebases onto it. No override and no rerun for this red.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Real Adapter Contract Tests is red on test/test_codex_session_mcp.py::test_the_installed_adapter_still_builds_the_frames_the_refusal_reads. This PR touches no adapter code; the same node fails on main (run https://github.com/kirodotdev/KiroCrew/actions/runs/39015978059, 343e4fea80) after the codex-acp 2.2.2 bump, and #21530 fixes it. This PR waits for #21530 and #21536, then rebases onto them. No override and no rerun for these reds.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Two more reds on this head, neither from this diff:

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

rebutted: every production route into _shutdown() runs after shutdown_event is set.

_shutdown() has one production caller, _shutdown_and_exit (exit_path.py), and it is reached from two places in service_boot.py.
run() reaches it only after await shutdown_event.wait().
The early stop, if not await self._wait_for_memory_preparation(), returns False only once the event is set (memory_lifecycle.py: stopping in done or shutdown_event.is_set(), the failed pass's await stopping, and the final shutdown_event.is_set() check). The one other False, startup.stopped, follows a stop that only the shutdown itself or a cancelled boot issues.
So a start scheduled after stop_eager_spawns() reads the event as set at its admission check and stands down before any agent process exists.
Setting the event inside _shutdown() was not added: direct test callers would leave the process-wide event set across tests.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

rebutted: the window spawns nothing, so leaving it unawaited leaks no process.

A start scheduled after stop_eager_spawns() runs with shutdown_event already set: every production route into _shutdown() comes after the event (see the Design disposition on this head).
The new check sits before admission, after the last read, so that start returns before _admit_prefetch or get_or_create runs. It owns no process and no registry entry.
It is not awaited, but it ends on its own once its debounce and reads finish, and in production os._exit ends it. The harness reap then sees a task that honours cancel at once (debounce sleep or a read).
Pinned by test_eager_spawn_shutdown.py::test_a_shutdown_during_the_reads_stands_down_before_admission[stopping].

@iamwhatever

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in src/kiro_crew/dashboard/state_parts/slots_frame.py (import) since this head went green, and #21581 fixed the three main reds (PyJWT pin, katex golden, codex-acp re-measure) (6a2bba1 -> 5682590); CI re-running.

@iamwhatever
iamwhatever force-pushed the fix/eager-spawn-teardown-18266 branch from b4106ec to bd46425 Compare October 17, 2026 10:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

rebutted: the event is set before _shutdown() on every production path, and the harness sets it too.

Production: _shutdown() is reached only through _shutdown_and_exit. run() gets there after await shutdown_event.wait(), and the memory-preparation early exit returns False only once the event is set (memory_lifecycle.py).
Harness: _teardown_boot calls shutdown_event.set() before letting run() exit, and its orchestrator._shutdown() fallback runs after that set.
The one unset case is a run() that already ended on its own before teardown. That leaves a narrow window between stop_eager_spawns() and close_all() that a new request would have to hit. Such a start is still cancelled by the harness reap, and once close_all() runs, get_or_create refuses with SessionClosingError.
Setting the event inside _shutdown() was left out: direct test callers would leave the process-wide event set across tests.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

rebutted: gathering the start does not extend the 10 s cap, and the cleanup is bounded the same way close_all() is.

The awaited start runs in the same asyncio.gather as sessions.close_all(). Both kill through AcpRuntime.kill: a 5 s SIGTERM grace, then SIGKILL and a 2 s reap. The orphaned-group 5 s grace runs only when that escalation did not, so the two are exclusive.
_shutdown_and_exit wraps _shutdown() in wait_for(..., GRACEFUL_SHUTDOWN_SECS). A cleanup that outlives it is cut off exactly like a slow close_all(): one warning, then os._exit.
On main the same start was not awaited at all, so a slow cleanup was already cut off by os._exit. Here it gets the cap's budget, concurrently, instead of none.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 17, 2026
@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 17, 2026 12:03

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (4 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix; gateway shutdown now cancels pending speculative session starts and the eager spawn returns once shutdown has begun, so no agent process outlives the teardown, with tests.

@KiroCrewCaptain
KiroCrewCaptain merged commit 8b709b4 into main Oct 17, 2026
104 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/eager-spawn-teardown-18266 branch October 17, 2026 12:04
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Integration: test_a_non_durable_memory_wait_is_named_as_one leaves _eager_spawn running past the 10 s teardown bound

2 participants