Skip to content

fix(apps): drain a backend's whole tree even when its launcher exited - #14292

Merged
iamwhatever merged 1 commit into
mainfrom
fix/windows-app-tree-pinned-drain-10143
Sep 27, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/windows-app-tree-pinned-drain-10143

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

What

Every exit that signals a spawned app backend now reaches its whole process tree, whether or not the launcher (the root process) is still alive. Three exits in src/kiro_crew/apps/backend.py change:

  • Startup-survival failure (_start_app_backend_body): the root's creation identity is read right after popen_limited returns and before _survived_spawn, and a new _drain_exited_root_tree drains whatever the dead launcher left behind before the spawn returns None.
  • Stop / restart of a tracked root that already exited (stop_app_backend, _restart_exited_backend): a launcher that died after startup while the server it forked kept serving used to be skipped by both (poll() is not None fell through to return True; the restart dropped the record and respawned beside the survivor). Both now call the same _drain_exited_root_tree; AppProcess carries spawn_instance beside pid_start_time so the POSIX vouch has a token once the pidfile row is gone.
  • Live stops (stop_app_backend, _terminate_retired_spawn): the SIGTERM step goes through a new _signal_backend_tree, which on Windows drains through platform_compat.kill_process_tree_pinned(root_pid, AppProcess.pid_start_time, SIGTERM) and falls back to the numeric taskkill only when the pinned path declines.
  • One spec paragraph in docs/system-specs/modules/app-kit-platform.md §7.

POSIX live stops (killpg via kill_process_tree), the SIGKILL escalation, adopted records (proc is None) and routes.py are untouched.

Why

A backend launcher is free to fork its real server and return -- the reported shape is a Python launcher that starts a Node process and exits 0. Crew then detected "backend exited immediately", returned None before any AppProcess or pidfile row existed, and nothing ever named the surviving Node process again. Disable, uninstall, update and gateway shutdown could not reach it either: on Windows kill_process_tree is taskkill /T /PID <root>, a walk FROM the root, which reaches nothing once the root has exited.

#12005 landed exactly the primitive this needs -- the exact-handle drain kill_process_tree_pinned -> terminate_windows_process_tree_owned, whose contract is that an exited root still anchors its descendants -- and wired it into the stale reaper and the ACP runtime, but not into the app backend's own exits. This is bounded reuse of that primitive, not a new ownership model: no Job Object, no kill-on-close (both settled against in #12005).

How

  • Identity before the check. root_start_time = _proc_start_time(proc.pid) runs on Windows only, right after the spawn, because an exited root's creation time cannot be probed once the Popen handle goes; POSIX does not need it (the group id IS the leader's pid) and pays no extra probe.
  • Dead root, Windows. kill_process_tree_pinned(proc.pid, root_start_time, SIGTERM). WindowsCleanupCapacityError and a drain that raises are logged and left to the cleanup registry's maintenance, as the stale reaper does at the same refusal. There is deliberately no numeric taskkill fallback here: the root it would walk from is the exited one.
  • Dead root, POSIX. One correction to the dispatch scope: kill_process_tree(proc.pid) cannot serve this branch because _survived_spawn has already reaped the leader and os.getpgid then raises ESRCH (measured on this host: getpgid -> ProcessLookupError, while killpg(pid, 0) still finds the group). So it takes the same route the stale reaper takes for a dead leader: signal_orphaned_spawn_group(proc.pid, SIGTERM, spawn_instance) -- members vouched by this spawn's KIROCREW_SPAWN_INSTANCE, signalled pinned to their own identity -- then SIGKILL after _REAP_SIGTERM_GRACE restricted to the members that took the SIGTERM. Off Linux the vouch cannot be read, nothing is signalled and the decline is logged (the documented trade). Never raises.
  • Live stops. _signal_backend_tree tries the pinned drain when IS_WINDOWS and an identity is recorded; a False (unpinnable), a capacity refusal or an OSError from the drain fall back to kill_process_tree, so a stop is never weaker than before. The SIGKILL escalation stays numeric: a drain that returned True has confirmed the root's exit, so the escalation is only reachable behind a fallback.

Tests

New test/test_app_backend_launcher_tree_drain.py (21 tests, CI only; Windows simulated by pointing platform_compat.IS_WINDOWS at True and closing the kernel boundary at kill_process_tree_pinned / kill_process_tree, the seam test_app_backend_stale_reap.py uses):

  • failed launcher on Windows: drained with the identity read BEFORE the survival check (the probe answers only until the check reports the exit); capacity refusal logged, no fallback, spawn still fails cleanly; a raising drain does not mask the failure; no identity -> nothing signalled;
  • failed launcher on POSIX: vouched members take SIGTERM, stubborn ones SIGKILL with expected = the SIGTERM takers, same instance token both passes, zero extra identity probes; a host that cannot vouch signals nothing; a raising vouch is a warning;
  • exited tracked root: Windows stop drains by the recorded identity (no spawn instance needed); POSIX stop drains by the record's spawn instance and signals nothing without one; the spawn records the same instance on AppProcess and in the pidfile row; the restart drains the dead root before spawning its replacement;
  • live stops on Windows: pinned drain on the recorded identity; unpinnable / capacity refusal / no identity fall back to the numeric taskkill; SIGKILL escalation stays numeric behind a fallback; adopted records are never tree-drained; retired spawn drains by its own identity;
  • POSIX stops and retired spawn: kill_process_tree calls byte-identical, pinned drain never called.

Red on base: 10/15 of the first round's tests fail on 47594b839 (the 5 that pass are the byte-identical / scope-safety guards). Mutation-verified: identity read at drain time instead of before the check (1 fail), drain call dropped (6), POSIX SIGKILL escalation dropped (1), stop back to kill_process_tree (3), pinned False treated as success (2), Windows failure branch using the numeric kill (3), exited-root stop branch removed (3), restart drain removed (1) -- all killed.

Sibling files run locally by path: test_apps_backend_coverage.py + test_app_backend_stale_reap.py + test_app_backend.py (463 passed). scripts/check_comment_history.py against main: passed. Static gates on the changed files: black (new file clean; backend.py is baseline-listed and its new hunks are black-clean), isort, flake8, mypy all 0.

Pattern harvest

Rule candidate: a tree-signalling helper that resolves the tree FROM a root pid (taskkill /T, getpgid) silently signals nothing once that root has exited; every exit that can run after the root's death must key on an identity captured while the root was still pinned (creation time on Windows, the spawn-instance vouch on POSIX).
Not generalizable: the exact fallback matrix (numeric taskkill for live stops, none for the failed-launcher branch) follows from which root the fallback would walk from and does not transfer to callers without a live/exited distinction.

Other

Out of scope, settled by #12005's design: a descendant that setsids out of the POSIX group, and Job-based kill-on-close. #10144 (update ordering, PR #14278) is a sibling in routes.py; this PR touches no line of it.

Closes #10143

@CrysisDeu
CrysisDeu requested review from a team and iamwhatever September 26, 2026 21:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of d9dde9b76f0791f1ac37297d67126e7e96a74063 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the #10143 repro on a real Windows host — the new tests simulate IS_WINDOWS by monkeypatch, so the exited-root pinned drain runs unproven there.

[DESIGN-REVIEWED] d9dde9b

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of d9dde9b76f0791f1ac37297d67126e7e96a74063 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Undeclared rider: a stop of an exited-root app can now return False under a withdrawn ceiling — verify callers of stop_app_backend(_retry_if_serving=…) expect that on this branch.

Not justified as shipped

  • Item 4 — undeclared: the description ("Both now call the same _drain_exited_root_tree") never mentions that the exited-root stop can now refuse, restore tracking and write a rejected_descendant_serving SEL row; the harm is the governance-ceiling boundary (a withdrawn ceiling told "stopped" while un-trusted code serves), nameable without the live-branch twin, and callers already handle False from that twin.

What this change ships

Inventory (9 items) — 8 justified

Intent: FIX (#10143, red-on-base tests) — every exit that signals a spawned app backend reaches its whole tree even after the launcher itself has exited.

  1. A failed launcher's surviving children are drained before the spawn reports failure — justified
  2. Windows spawns probe the root's creation identity before the survival check — justified
  3. Stopping an app whose tracked root already exited now drains the survivors instead of skipping them — justified
  4. That stop can refuse under a withdrawn ceiling (returns False, tracking restored, SEL row) — undeclared
  5. Live Windows stops try the identity-pinned drain first, numeric taskkill as fallback — justified
  6. Retired-spawn termination takes the same pinned path — justified
  7. Restart drains the dead root's tree before spawning the replacement — justified
  8. AppProcess.spawn_instance carries the vouch token (2 consumers counted: stop_app_backend, _restart_exited_backend; grep ap.spawn_instance) — justified
  9. Spec paragraph in app-kit-platform.md §7, same commit — justified

Reuse verified: the drain is kill_process_tree_pinned (platform_compat.py:6604) and signal_orphaned_spawn_group (session_pid.py:1791), both landed by #12005; the stale reaper's orchestration (_reap_orphaned_backend_group) is coupled to pidfile-row retention and batched grace windows, so the per-call helper is meaningfully different, not a second spelling.

[FIRST-PRINCIPLES-REVIEWED] d9dde9b

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of d9dde9b76f0791f1ac37297d67126e7e96a74063 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/apps/backend.py:1182 -- return False turns an inconclusive Windows drain error into a confirmed survivor, making ceiling enforcement skip its port probe and restore dead tracking -> Fix: return None.

FINDING -- src/kiro_crew/apps/backend.py:3031 -- publishing AppProcess without the captured root_start_time lets a concurrent Windows stop reap the launcher before identity is recorded, leaving its child running -> Fix: initialize pid_start_time=root_start_time. (origin: validation)

FINDING -- src/kiro_crew/apps/backend.py:4135 -- a confirmed surviving descendant is followed by an auto-port replacement, leaving two backend instances running -> Fix: restore the tracked record and continue the existing retry cadence when the drain returns False.

[GPT-REVIEWED] d9dde9b

False positive or not applicable? A repository writer can comment:
/ai-review override gpt d9dde9b76f0791f1ac37297d67126e7e96a74063: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed d9dde9b76f0791f1ac37297d67126e7e96a74063 — this comment is updated in place on each push.

Review details

The exited-root drain calls a tree it just SIGKILLed a positive survivor, because liveness is read with no settling window after the kill.

FINDING — src/kiro_crew/apps/backend.py:1239 — alive = [m for m in final_vouched if _pid_alive(m)] probes liveness immediately after the SIGKILL issued two lines above, and _pid_alive is os.kill(pid, 0) (a dying task or its unreaped zombie answers alive), so any descendant that ignored the SIGTERM and was then killed makes the drain return False — "a member positively survives" — which logs the unmeasured operator claim "a descendant outlived its exited backend root (pid %d) and survived the drain", makes _restart_exited_backend announce "the replacement is spawned beside it", and lets the gone is False disjunct at :3230 short-circuit the port probe so a ceiling revocation refuses the stop of a tree that is already gone → Fix: before computing alive, wait out the same _REAP_SIGTERM_GRACE/_REAP_POLL_INTERVAL poll over killed that the SIGTERM pass already runs over signalled, so False is only returned for a member still alive after that window. (origin: validation)

[OPUS-REVIEWED] d9dde9b

Verdict parsed from the review's SHA-scoped output markers for commit d9dde9b76f0791f1ac37297d67126e7e96a74063.

False positive or not applicable? A repository writer can comment:
/ai-review override fable d9dde9b76f0791f1ac37297d67126e7e96a74063: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the fix/windows-app-tree-pinned-drain-10143 branch from 7e9bf47 to 9f4c406 Compare September 26, 2026 22:00
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
On Windows an app backend launcher that forked its real server and
returned 0 left that server running with nothing able to stop it: the
startup-survival failure branch returned None without signalling, and
the live stops used kill_process_tree, i.e. taskkill /T /PID <root>,
which walks FROM the root and reaches nothing once the root has exited.
A tracked root that exited after startup was skipped entirely: the stop
and the restart both dropped the record without signalling.

Reuse the exact-handle drain that #12005 landed. Read the root's
creation identity right after popen_limited returns, before the
survival check, while the Popen still pins the process; drain a dead
root's tree (startup failure, stop of an exited root, restart of an
exited root) through kill_process_tree_pinned on that identity on
Windows and through the spawn-instance vouched group on POSIX, with
AppProcess now carrying the spawn instance beside pid_start_time; route
the live stops' Windows branch through the same pinned drain with the
numeric taskkill kept as the fallback when the pinned path declines.
POSIX live stops are unchanged. Adopted records are never tree-drained.

Closes #10143
@CrysisDeu
CrysisDeu force-pushed the fix/windows-app-tree-pinned-drain-10143 branch from 9f4c406 to d9dde9b Compare September 26, 2026 22:48
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 27, 2026 17:31
@iamwhatever
iamwhatever merged commit 9cb874b into main Sep 27, 2026
96 checks passed
@iamwhatever
iamwhatever deleted the fix/windows-app-tree-pinned-drain-10143 branch September 27, 2026 17:32
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 27, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Windows app backend descendants can survive after their launcher exits

2 participants