Repository navigation
fix(apps): drain a backend's whole tree even when its launcher exited - #14292
Conversation
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Verify the #10143 repro on a real Windows host — the new tests simulate [DESIGN-REVIEWED] d9dde9b |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Undeclared rider: a stop of an exited-root app can now return Not justified as shipped
What this change shipsInventory (9 items) — 8 justifiedIntent: FIX (#10143, red-on-base tests) — every exit that signals a spawned app backend reaches its whole tree even after the launcher itself has exited.
Reuse verified: the drain is [FIRST-PRINCIPLES-REVIEWED] d9dde9b |
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsFINDING -- src/kiro_crew/apps/backend.py:1182 -- FINDING -- src/kiro_crew/apps/backend.py:3031 -- publishing FINDING -- src/kiro_crew/apps/backend.py:4135 -- a confirmed surviving descendant is followed by an auto-port replacement, leaving two backend instances running -> Fix: restore the tracked record and continue the existing retry cadence when the drain returns [GPT-REVIEWED] d9dde9b False positive or not applicable? A repository writer can comment: |
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsThe exited-root drain calls a tree it just SIGKILLed a positive survivor, because liveness is read with no settling window after the kill. FINDING — src/kiro_crew/apps/backend.py:1239 — [OPUS-REVIEWED] d9dde9b Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
7e9bf47 to
9f4c406
Compare
On Windows an app backend launcher that forked its real server and returned 0 left that server running with nothing able to stop it: the startup-survival failure branch returned None without signalling, and the live stops used kill_process_tree, i.e. taskkill /T /PID <root>, which walks FROM the root and reaches nothing once the root has exited. A tracked root that exited after startup was skipped entirely: the stop and the restart both dropped the record without signalling. Reuse the exact-handle drain that #12005 landed. Read the root's creation identity right after popen_limited returns, before the survival check, while the Popen still pins the process; drain a dead root's tree (startup failure, stop of an exited root, restart of an exited root) through kill_process_tree_pinned on that identity on Windows and through the spawn-instance vouched group on POSIX, with AppProcess now carrying the spawn instance beside pid_start_time; route the live stops' Windows branch through the same pinned drain with the numeric taskkill kept as the fallback when the pinned path declines. POSIX live stops are unchanged. Adopted records are never tree-drained. Closes #10143
9f4c406 to
d9dde9b
Compare
What
Every exit that signals a spawned app backend now reaches its whole process tree, whether or not the launcher (the root process) is still alive. Three exits in
src/kiro_crew/apps/backend.pychange:_start_app_backend_body): the root's creation identity is read right afterpopen_limitedreturns and before_survived_spawn, and a new_drain_exited_root_treedrains whatever the dead launcher left behind before the spawn returnsNone.stop_app_backend,_restart_exited_backend): a launcher that died after startup while the server it forked kept serving used to be skipped by both (poll() is not Nonefell through toreturn True; the restart dropped the record and respawned beside the survivor). Both now call the same_drain_exited_root_tree;AppProcesscarriesspawn_instancebesidepid_start_timeso the POSIX vouch has a token once the pidfile row is gone.stop_app_backend,_terminate_retired_spawn): the SIGTERM step goes through a new_signal_backend_tree, which on Windows drains throughplatform_compat.kill_process_tree_pinned(root_pid, AppProcess.pid_start_time, SIGTERM)and falls back to the numerictaskkillonly when the pinned path declines.docs/system-specs/modules/app-kit-platform.md§7.POSIX live stops (
killpgviakill_process_tree), the SIGKILL escalation, adopted records (proc is None) androutes.pyare untouched.Why
A backend launcher is free to fork its real server and return -- the reported shape is a Python launcher that starts a Node process and exits 0. Crew then detected "backend exited immediately", returned
Nonebefore anyAppProcessor pidfile row existed, and nothing ever named the surviving Node process again. Disable, uninstall, update and gateway shutdown could not reach it either: on Windowskill_process_treeistaskkill /T /PID <root>, a walk FROM the root, which reaches nothing once the root has exited.#12005 landed exactly the primitive this needs -- the exact-handle drain
kill_process_tree_pinned->terminate_windows_process_tree_owned, whose contract is that an exited root still anchors its descendants -- and wired it into the stale reaper and the ACP runtime, but not into the app backend's own exits. This is bounded reuse of that primitive, not a new ownership model: no Job Object, no kill-on-close (both settled against in #12005).How
root_start_time = _proc_start_time(proc.pid)runs on Windows only, right after the spawn, because an exited root's creation time cannot be probed once thePopenhandle goes; POSIX does not need it (the group id IS the leader's pid) and pays no extra probe.kill_process_tree_pinned(proc.pid, root_start_time, SIGTERM).WindowsCleanupCapacityErrorand a drain that raises are logged and left to the cleanup registry's maintenance, as the stale reaper does at the same refusal. There is deliberately no numerictaskkillfallback here: the root it would walk from is the exited one.kill_process_tree(proc.pid)cannot serve this branch because_survived_spawnhas already reaped the leader andos.getpgidthen raisesESRCH(measured on this host:getpgid -> ProcessLookupError, whilekillpg(pid, 0)still finds the group). So it takes the same route the stale reaper takes for a dead leader:signal_orphaned_spawn_group(proc.pid, SIGTERM, spawn_instance)-- members vouched by this spawn'sKIROCREW_SPAWN_INSTANCE, signalled pinned to their own identity -- then SIGKILL after_REAP_SIGTERM_GRACErestricted to the members that took the SIGTERM. Off Linux the vouch cannot be read, nothing is signalled and the decline is logged (the documented trade). Never raises._signal_backend_treetries the pinned drain whenIS_WINDOWSand an identity is recorded; aFalse(unpinnable), a capacity refusal or anOSErrorfrom the drain fall back tokill_process_tree, so a stop is never weaker than before. The SIGKILL escalation stays numeric: a drain that returnedTruehas confirmed the root's exit, so the escalation is only reachable behind a fallback.Tests
New
test/test_app_backend_launcher_tree_drain.py(21 tests, CI only; Windows simulated by pointingplatform_compat.IS_WINDOWSat True and closing the kernel boundary atkill_process_tree_pinned/kill_process_tree, the seamtest_app_backend_stale_reap.pyuses):expected= the SIGTERM takers, same instance token both passes, zero extra identity probes; a host that cannot vouch signals nothing; a raising vouch is a warning;AppProcessand in the pidfile row; the restart drains the dead root before spawning its replacement;taskkill; SIGKILL escalation stays numeric behind a fallback; adopted records are never tree-drained; retired spawn drains by its own identity;kill_process_treecalls byte-identical, pinned drain never called.Red on base: 10/15 of the first round's tests fail on
47594b839(the 5 that pass are the byte-identical / scope-safety guards). Mutation-verified: identity read at drain time instead of before the check (1 fail), drain call dropped (6), POSIX SIGKILL escalation dropped (1), stop back tokill_process_tree(3), pinnedFalsetreated as success (2), Windows failure branch using the numeric kill (3), exited-root stop branch removed (3), restart drain removed (1) -- all killed.Sibling files run locally by path:
test_apps_backend_coverage.py+test_app_backend_stale_reap.py+test_app_backend.py(463 passed).scripts/check_comment_history.pyagainst main: passed. Static gates on the changed files: black (new file clean;backend.pyis baseline-listed and its new hunks are black-clean), isort, flake8, mypy all 0.Pattern harvest
Rule candidate: a tree-signalling helper that resolves the tree FROM a root pid (
taskkill /T,getpgid) silently signals nothing once that root has exited; every exit that can run after the root's death must key on an identity captured while the root was still pinned (creation time on Windows, the spawn-instance vouch on POSIX).Not generalizable: the exact fallback matrix (numeric
taskkillfor live stops, none for the failed-launcher branch) follows from which root the fallback would walk from and does not transfer to callers without a live/exited distinction.Other
Out of scope, settled by #12005's design: a descendant that
setsids out of the POSIX group, and Job-based kill-on-close. #10144 (update ordering, PR #14278) is a sibling inroutes.py; this PR touches no line of it.Closes #10143