Skip to content

fix(session): reap a crashed gateway's backends in the periodic sweep - #18877

Closed
javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:fix/periodic-sweep-reaps-a-dead-gateways-backends
Closed

javenciu wants to merge 1 commit into
kirodotdev:mainfrom
javenciu:fix/periodic-sweep-reaps-a-dead-gateways-backends

Conversation

@javenciu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A backend left running by a gateway that crashed inside the spawn grace is reaped by the next gateway's first periodic sweep after the grace, instead of living for the whole life of the new gateway.

A kiro-cli backend is started in its own session (start_new_session=True, acp/launch.py), so it outlives a gateway that is SIGKILLed, OOM-killed or loses power. The next boot's reap, cleanup_orphaned_sessions, then skips any backend younger than SWEEP_SPAWN_GRACE_SECONDS (120 s). The grace's recorded reason (session_pid.py, above the constant) is that "a missed kill self-heals next cycle". For a crashed gateway's backend there is no next cycle:

  • The periodic sweep takes only its own gateway's entries (should_skip_tagged=lambda gw, _p: gw != my_gw_pid).
  • Its kill phase indexes only its own gateway's entries (_session_pid_entry_index).
  • cleanup_orphaned_sessions runs again only at the next clean shutdown or the next boot.

Why it matters

A supervisor restarts a crashed gateway within seconds, so every backend spawned in the 120 s before the crash keeps running for the whole life of the new gateway. It keeps its memory, and possibly its turn: whether a real kiro-cli keeps working after its stdin closes was not driven here, because that would spend real model quota.

Not a goal

  • Dropping the grace for dead-gateway entries at boot. The grace's reason was not re-examined here, and the periodic sweep is the smaller change.
  • PR_SET_PDEATHSIG on the spawn. It is Linux-only, it fires on the spawning thread's exit rather than the process's, and it defeats the separate session setsid sets up on purpose.
  • Deciding whether a surviving kiro-cli keeps running its turn after its stdin closes. That would decide whether a survivor also spends quota; it is not driven here.
  • A second pid-level kill path for unowned runtimes, which the leaked-runtime RFC's Decision 1 rules out (docs/request-for-change/rfc-leaked-runtime-reclaim.md, the closed fix(session): reclaim untracked agent runtimes this install spawned #12612): these rows are tracked runtimes, each a line _track_session_pid wrote in this data home's session PID file with its start token, which the boot reap already ends on main, so this is tracked teardown, which the RFC's Non-goals leave as it is.

What changed (motivation → approach → change)

  • Symptom: a crashed gateway's backend younger than the grace is never reaped.
  • Root cause: both phases of the periodic sweep skip every entry whose gateway PID is not this gateway's, alive or dead, and nothing else runs again until the next boot.
  • Change, in src/kiro_crew/session_pid.py:
    • _owning_gateway_may_be_alive(gw_pid) is the one probe both sweeps use to decide whose entries they may take. It is platform_compat.pid_exists, exactly what the boot path's _skip_tagged already called, and _skip_tagged now calls it, so the boot reap is unchanged. A gateway PID that now names another process, or one this user cannot signal, reads as alive. That is the safe direction: an entry counts as a dead gateway's only when its gateway PID names no process.
    • The periodic scan phase also takes the entries of a gateway that is gone. A live gateway's entries are still skipped.
    • The kill phase's index (_session_pid_entry_index) covers the same set, so a candidate from a dead gateway's entry is re-judged against its own recorded line and token rather than skipped. This gateway's own entry wins when both name one PID.
  • Every existing guard applies unchanged to these entries: the spawn grace; the active-session check, in both phases; the start-token identity check, in both phases, which prunes an entry whose PID now names a different process and never signals it; the argv gate in _kill_pid_tree; the session cleaner's ownership gate and teardown barrier.
  • The periodic sweep does its file and process work off the event loop. The scan phase runs in asyncio.to_thread (session_cleanup.py, _sweep_periodic_pids), and so does the kill phase. The added probe is one kill(pid, 0) per entry of another gateway, inside those threads.
  • Tests: test/test_pid_sweep_dead_gateway.py (new, four tests, listed below). In test/test_pid_sweep_helpers.py, test_only_sweeps_own_gateway_entries is renamed test_sweeps_own_entries_and_skips_a_live_gateways_entries: its fake made every PID dead, the other gateway's too, which under this change correctly makes that gateway's dead entry prunable. The fake now keeps the other gateway alive, so the test still checks what it was for: a live gateway's entries are preserved.
  • Security: no credential, auth or permission path changes. Only PIDs whose recorded start token still matches are signalled, as before.

Backwards compatibility

Compatible: the boot reap, the grace, the identity checks and the argv gate behave as before; the only new behavior is that a periodic sweep also judges the entries of a gateway whose PID names no process.
Removes nothing: no function, constant, flag or file format is removed. One test is renamed and its fake adjusted, as described above.

Tests

test/test_pid_sweep_dead_gateway.py drives real processes.

  • Setup: a child Python process stands in for the gateway. It spawns a fake kiro-cli the way acp/launch.py does (own session, the KIROCREW_SPAWNED marker, pipes) and records it with the real _track_session_pid. The fake is the interpreter reached through a symlink named kiro-cli, writing a line every 0.1 s. The test process plays the next gateway and runs the real boot reap and both real phases of the periodic sweep.
  • Isolation: all paths are under tmp_path. The boot reap's host-wide MCP sweep and its prunes are replaced by recorders. Only PIDs the test started are signalled, by recorded PID, and teardown checks that every one is gone. The one text-mode subprocess call passes encoding="utf-8", and every child that writes a file starts in a directory under tmp_path (cwd=).
Test On main With this change
test_a_crashed_gateways_backend_is_reaped_by_the_first_periodic_sweep_after_its_grace FAIL (below) PASS
test_a_live_gateways_backend_is_left_alone_young_or_old (control) PASS PASS
test_an_entry_whose_pid_now_names_another_process_is_not_signalled (control) PASS PASS
test_the_boot_reap_still_kills_a_crashed_gateways_backend_past_its_grace (control: boot unchanged) PASS PASS

Red on origin/main d32c084 (the test file copied onto it, 3 of 3 serial runs): 1 failed, 3 passed each.

E       AssertionError: the first periodic sweep past the grace left backend 2108243 of dead gateway 2108242 running: candidates [], killed 0, entry '2108242:2108243:56228911'; it has written 51 ticks
FAILED test/test_pid_sweep_dead_gateway.py::test_a_crashed_gateways_backend_is_reaped_by_the_first_periodic_sweep_after_its_grace

Green on this branch (rebased onto origin/main d32c084): the new file 4 passed in 3 of 3 serial runs. The 79 test files that name session_pid, the new file first: 5,397 passed, 41 skipped, 5 xfailed, 1 failed. The failure, test_browser_launcher.py::TestOpenUrlArgv::test_missing_cli_is_an_honest_error, fails the same way on main alone on this host, which has playwright-cli installed outside the managed location. test_acp_client.py and test_ephemeral_sessions.py were left out because they can write under the real home directory. mypy (session_pid.py), flake8, isort, black (no changed file is baselined), scripts/check_black_formatting.py, scripts/check_subprocess_encoding.py, COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py and ./scripts/docs-lint.sh: all exit 0.

Manual verification

The tests run the real sweep code against real child processes that stand in for a gateway and its kiro-cli. Not run against a real kiro-cli, and not run with a real supervisor restart.

Related Issues

Open #17765 (apply the orphan-sweep spawn grace on Windows) changes the grace itself at different lines and merges cleanly with this change. Merged #767 and #8039 added the identity checks this change relies on.

Pattern harvest

Rule candidate: review-prompt
Pattern: a periodic cleaner that scopes itself to its own owner's records, with the only cross-owner cleanup at boot; a record whose owner died inside a grace window is then never revisited.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

A kiro-cli backend runs in its own session, so it outlives a gateway that
is SIGKILLed. The next boot's reap skips any backend inside the 120 s
spawn grace, and the periodic sweep took only its own gateway's entries,
so a backend spawned shortly before a crash ran for the whole life of the
next gateway.

The periodic sweep, both its scan phase and its kill-phase index, now
also takes the entries of a gateway whose PID names no process, using
the same pid_exists probe the boot reap uses. A live gateway's entries
are still skipped, and the spawn grace, the active-session and
start-token identity checks and the argv gate apply unchanged.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 10, 2026 15:37
@javenciu
javenciu requested a review from cruisercohen October 10, 2026 15:37
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision e36ca2fac06d62a5e727cbc22e71123851a49393 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed e36ca2fac06d62a5e727cbc22e71123851a49393 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] e36ca2f

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🔴 BLOCK (blocking)

Design-level review of e36ca2fac06d62a5e727cbc22e71123851a49393 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: BLOCK

The problem isn't shown to exist: an existing reaper already runs on every periodic tick and kills a dead gateway's backends, with no spawn grace.

Blockers

The stated root cause is contradicted by cleanup_orphaned_session_roots

  • The description says "nothing else runs again until the next boot." But session_cleanup.py:1204 runs await self._sweep_session_roots() on every _cleanup_loop tick, ahead of _sweep_periodic_pids.
  • That calls session_pid.cleanup_orphaned_session_roots. Its docstring says it will "reap dead-gateway PID records." At session_pid.py:3067 it takes every entry whose gateway is PID_DEAD, checks the token and argv, then calls _kill_pid_tree. Nothing in it checks SWEEP_SPAWN_GRACE_SECONDS.
  • The red test misses this because it drives only _periodic_pid_sweep and _kill_confirmed_and_writeback, never the real tick.
  • What the PR actually adds is a second periodic reaper over the same entries. It sits behind the ownership gate, while the existing one does not. The result is two kill paths with different authority over one population, which the stated harm does not justify.
  • Fix: drive the real tick. If the backend still survives, fix whatever stops cleanup_orphaned_session_roots from reaping it, rather than adding a new path.

Clears when: a test that runs a full _cleanup_loop tick (including _sweep_session_roots) fails on main for a crashed gateway's young backend, or the PR is withdrawn

Watch

  • If the second path turns out to be needed, the reclaim section of docs/system-specs/modules/session.md (around line 4223) has to say which sweep owns dead-gateway entries, and why one asks the ownership gate and the other doesn't.

Clears when: session.md states which sweep owns dead-gateway entries, in the same commit

[DESIGN-REVIEWED] e36ca2f

Description read

[DESCRIPTION-READ] f6e6e6e16a67a62763777af1f37cc47676a12a89bbe5d7210b15706ad4d008af

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of e36ca2fac06d62a5e727cbc22e71123851a49393 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (5 items) — 5 justified

Intent: when a gateway crashes, the backends it spawned in its last 120 s are stopped soon after the next gateway starts, instead of running for that gateway's whole life. FIX.

  1. The periodic sweep now finds and kills backends whose owning gateway is gone, once the spawn grace has passed — justified
  2. The kill phase now checks a dead gateway's entries against their own recorded line and start token instead of skipping them — justified
  3. The periodic sweep now removes the file entries of a dead gateway's backends that have already exited — justified
  4. The boot reap and periodic sweep share one liveness probe; it wraps pid_exists, and the boot reap behaves as before — justified
  5. A test was renamed and its fake now keeps the other gateway alive, so it still checks that a live gateway's entries are kept — justified

[FIRST-PRINCIPLES-REVIEWED] e36ca2f

Description read

[DESCRIPTION-READ] 1957d6fa03175d230162fa3fb47728df9452afd663d4572cba4674cbc5e2a8a3

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed e36ca2fac06d62a5e727cbc22e71123851a49393 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] e36ca2f

@javenciu

Copy link
Copy Markdown
Contributor Author

Closing this one. The Design review is right: the problem it fixes does not exist on main.

_cleanup_loop calls _sweep_session_roots() on every tick (session_cleanup.py:1204), and cleanup_orphaned_session_roots() (session_pid.py:2989) kills the process tree of any tracked backend whose gateway is dead, after the start-token and argv checks and with no spawn grace (session_pid.py:3159). That covers the case this PR targets: a backend spawned just before its gateway was SIGKILLed, which the boot reap skips as too young.

I ran that case against main with this PR's own test harness. The boot reap left the backend alive, and the next real cleanup tick killed it and removed its tracking entry, in 4 runs out of 4. On this branch the roots sweep still runs first in the tick, so the new periodic path never reaches that backend. The description's premise, that nothing else reaps it until the next boot, is wrong, and the change would only add a second reaper.

@javenciu javenciu closed this Oct 10, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant