Repository navigation
fix: stop the browser harness when graceful shutdown runs out of time - #36647
Conversation
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
7526e0a to
4a05c48
Compare
|
|
|
CI on 7526e0a, for the record:
|
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the exit path now reuses the existing Notes
[DESIGN-REVIEWED] 4ea644a Description read[DESCRIPTION-READ] 9692a1c7e0f071a3b2e97c89928b4be1b61e334e7569a37bfa0abf420b43fe6d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The exit path reuses the existing Notes
What this change shipsInventory (8 items) — 7 justifiedIntent: a stopped gateway leaves no browser-harness daemon (or instance tunnel) running after a graceful shutdown that timed out. FIX.
[FIRST-PRINCIPLES-REVIEWED] 4ea644a Description read[DESCRIPTION-READ] cc541b6fa6b50ca7123e27cd4502a45fa3f1849a5d6484b7545ec2a839f6258a That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Code change approver recommendation: A person should decideWhy: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 521 lines, more than the 400 it may approve on its own. The exit path now runs the existing pre-exec stop list once before os._exit. That list is bounded at 10 s and never raises, so a graceful shutdown that ran out of time still stops the browser harness. The harness and instance cleanup hooks now unregister their stops only after the stops return. A supervisor teardown cut partway keeps its handle aside, never reused as the live one, so the next stop or ensure reaches the children it missed and skips the tab close. Two stops at once are serialised by _ensure_lock. Tunnel shutdown removes a tunnel only after its stop returns and still holds the manager's lock. Every stop on the list is safe to repeat: sleep-inhibitor release, watchdog stop (the exit code is computed before this step), skill-usage flush. The restart path is unchanged. Tests are isolated per test and cover each changed path. No blocking defects found. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
CI on 4a05c48: every backend shard passed, Linux and Windows. All review lanes ran and posted (Design and First Principles PASS, GPT and Opus with no blocking lines). The only red is |
buluoray
left a comment
There was a problem hiding this comment.
Approve on 4a05c48. When a graceful shutdown runs out of time, the gateway now runs the stops that the browser harness and the instance tunnels registered for the in-place restart before it exits, so the harness daemon no longer outlives the gateway. The stop it runs is the same one that the cleanup hook and the restart already run. It closes only the tabs the agent opened and stops only child processes the gateway started itself, so a person's own Chrome and their tabs are untouched. Each cleanup hook now drops its registration only after its own stop returns. Since the last review, the only change is in the new test file: its fixed ten-second wait is replaced by the suite's shared wait helpers, which clears the wait-discipline ratchet; the product code is unchanged. Each production change still turns at least one new test red when reverted. Follow-ups that do not block: a hook cut in the middle of the supervisor's teardown has already dropped its handle, so the stop that runs again at exit cannot reach the worker or the browser it started; and the exit path now has more bounded steps after the graceful budget than the supervisor's stop deadline leaves room for, in the rare case where every one of them stalls. Decisive lines: src/kiro_crew/slack/gateway_runtime/exit_path.py:743, src/kiro_crew/dashboard/server_runtime/browser_lifecycle.py:78-90, src/kiro_crew/dashboard/server_runtime/instances_lifecycle.py:203-214, test/test_gateway_exit_pre_exec_stops.py (the hang guard now uses kiro_crew.testing.wait)
4a05c48 to
f3a02cc
Compare
|
Rebased onto |
f3a02cc to
f2a4609
Compare
|
self-added: no
|
f2a4609 to
baba48f
Compare
|
|
f619717 to
6c67fad
Compare
|
self-added: yes
|
6c67fad to
8bed575
Compare
|
|
buluoray
left a comment
There was a problem hiding this comment.
Approve on 8bed575. When a graceful shutdown runs out of time, the gateway now runs the stops that the browser harness and the instance tunnels registered for the in-place restart before it exits, so the harness daemon no longer outlives the gateway. Since the last review, this head also closes the gap we noted as a follow-up. A harness stop that is cut in the middle of its teardown now keeps the half-stopped handle aside, and the next stop finishes stopping the worker, daemon, relay and browser that the cut one never reached. That retry skips the agent-tab close, and the tab close itself is now bounded at three seconds, so a stalled browser connection cannot use up the stop's time before the children stop. The tunnel manager drops each tunnel only after its stop returns, so the exit path also reaches a tunnel that a cut shutdown left mid-stop. The retry only stops child processes that the gateway itself started, through the same process handles as before, and a child that already exited is skipped, so nothing else can be stopped by mistake. Each new change turns a new test red when reverted, and the new tests wait on events with bounded waits, not sleeps. Follow-ups that do not block: a stop cut inside the relay's own close leaves the relay's local listener open until the process exits, because the relay marks itself torn down before its awaits; and the exit path still has more bounded steps after the graceful budget than the service manager's stop deadline leaves room for, in the rare case where every one of them stalls. Decisive lines: src/kiro_crew/browser_harness/supervisor.py:286 (_cut_handle), :441-446 (holds_browser counts it), :771-799 (_teardown / _stop_handle), :1264-1276 (_stop_process skips a reaped child, stops via platform_compat.terminate_and_reap); src/kiro_crew/instances/ssh_tunnel_manager_lifecycle.py:866-876 (pop after stop, under self._lock); src/kiro_crew/slack/gateway_runtime/exit_path.py (seam call, unchanged since 4a05c48)
|
|
8bed575 to
b44a89a
Compare
|
Rebased onto main |
|
|
A gateway stopped while a long request is open (the dashboard Logs page) runs out of its graceful shutdown budget before the web server reaches the dashboard's cleanup hooks, and its os._exit then leaves the browser harness daemon running under PID 1. The exit path now runs the pre-exec stops before its hard exit. The harness, the instance tunnels and the sleep inhibitor already register their stops there for the in-place restart, which skips the same hooks. The harness and instance cleanup hooks now unregister their stop only once it has returned, so a hook the graceful bound cancels mid-stop leaves the stop for the exit path. Fixes #36555 Part of #26967
|
Rebased onto main |
b44a89a to
4ea644a
Compare
Problem / Motivation
Goal: A gateway whose graceful shutdown runs out of time still stops the browser harness daemon before it exits.
Stop a gateway while the dashboard Logs page is open, and the browser harness daemon keeps running after it exits. Reproduced on main with an isolated home and a loopback-only gateway (
gateway --test-mode --no-crons --no-tunnel), after the startup auto-install had warmed the harness: withGET /api/logsheld open, SIGTERM makes the gateway logGraceful shutdown timed outand exit after 10 seconds, andpython -m browser_harness.daemonis still running, reparented to PID 1.Why it matters
A stopped gateway should leave nothing running. The daemon keeps its IPC socket and its pid file while no gateway owns it, until the next boot reclaims it. Any operator who stops the gateway while a long request is open (the Logs page, a streamed reply) is hit.
Not a goal
_force_exit), which ends the process from its own thread without the exit path's stops.What changed (motivation → approach → change)
Symptom → cause. The harness supervisor is stopped only by a dashboard
on_cleanuphook. The web server runs cleanup hooks only after its in-flight requests drain, and an open/api/logsstream holds that drain past the 10-second graceful budget, so the hook never runs andos._exitleaves the daemon behind.Approach. The in-place restart skips the same hooks, so the harness (and the instance tunnels and the sleep inhibitor) already register the same stop with the pre-exec seam,
pre_exec_stops. The exit path now runs that seam once before its hard exit, instead of naming any owner.Change.
slack/gateway_runtime/exit_path.py_shutdown_and_exitawaitsrun_pre_exec_stops()after the orphan sweep and the shared crew release, before the drains andos._exit. It is bounded inside the seam (10 s) and never raises.dashboard/server_runtime/browser_lifecycle.py,instances_lifecycle.pybrowser_harness/supervisor.py,instances/ssh_tunnel_manager_lifecycle.pyshutdown()drops a tunnel only after that tunnel'sstop()returns.pre_exec_stops.pyrun_pre_exec_stopstakes anoccasion("restart", the default, or"exit") so a stop that fails or overruns at exit is not logged as "before restart". The module docstring names the exit path as a caller, and that each stop must be safe to repeat.After a shutdown that did reach the hooks, the harness and instance stops are unregistered. What is left (the skill-usage flush and the loop-stall watchdog stop) is idempotent. Signals still go through the supervisor's existing
platform_compat.terminate_and_reap; nothing new is signalled. The slack-gateway, browser and instances specs describe the exit-path call and the repeatable stops.Backwards compatibility
Compatible: a shutdown that finished in time finds only idempotent stops left. One that ran out of time takes as long as the harness stop (bounded at 10 s) and stops a daemon that was left running before.
Removes nothing: it adds a call to the exit path and moves two unregister calls and one tunnel removal after their stops; no path, record or key is taken away.
Tests
New
test/test_gateway_exit_pre_exec_stops.py, driving the real startup and cleanup hooks with the supervisor and tunnel manager replaced:test_a_timed_out_shutdown_still_stops_the_browser_harness: no cleanup hook ran; the exit path stops the harness before its hard exit.test_a_cleanup_hook_cut_mid_stop_leaves_the_stop_to_the_exit_path: the harness hook is cancelled mid-stop; the exit path runs the stop again.test_a_cleanup_hook_that_finished_leaves_nothing_to_stop_again: a hook that finished leaves nothing registered, and the exit path does not stop the harness twice.test_an_instance_teardown_cut_mid_stop_leaves_it_to_the_exit_path: the same as the second test, for the instance tunnels.test_a_failing_stop_is_logged_as_before_the_exit_and_the_exit_still_runs: a stop that raises is logged as before the exit, and the hard exit still runs.Owner-level tests run the real stop:
test/test_browser_harness_supervisor_cov.py::test_a_stop_cut_mid_teardown_leaves_the_rest_to_the_next_stop: a supervisorstop()cancelled while it stops the worker sets its handle aside (notconnected, stillholds_browser), and the nextstop()stops the worker, daemon and browser and closes the relay.test_a_stop_cut_in_a_stalled_tab_close_skips_it_on_the_next_stopandtest_a_stalled_tab_close_is_bounded_and_the_children_are_still_stopped(same file): a stop cut in a tab close that never answers does not wait on it again, and a stalled tab close alone still lets the worker, daemon and browser stop.test/test_instances.py::TestSshTunnelManager::test_a_shutdown_cut_mid_stop_leaves_the_tunnel_to_the_next_shutdown: ashutdown()cancelled inside a tunnel'sstop()leaves that tunnel in the manager, and the nextshutdown()stops it.Proof: PROVEN test/test_browser_harness_supervisor_cov.py::test_a_stop_cut_in_a_stalled_tab_close_skips_it_on_the_next_stop, test/test_gateway_exit_pre_exec_stops.py::test_a_timed_out_shutdown_still_stops_the_browser_harness, test/test_gateway_exit_pre_exec_stops.py::test_a_cleanup_hook_cut_mid_stop_leaves_the_stop_to_the_exit_path, test/test_gateway_exit_pre_exec_stops.py::test_an_instance_teardown_cut_mid_stop_leaves_it_to_the_exit_path, test/test_gateway_exit_pre_exec_stops.py::test_a_failing_stop_is_logged_as_before_the_exit_and_the_exit_still_runs (head 6c67fad)
Each production hunk was also reverted on its own: without the exit-path call four tests fail, without either reorder the matching cut-mid-stop test fails, without the supervisor's set-aside handle, its retry skipping the tab close, its tab-close bound, or the tunnel manager's later removal the matching owner-level test fails, and without
occasion="exit"the logging test fails. The restart's own log lines are unchanged; their tests intest_restart_stops_instance_tunnels.pypass.Determinism: every wait in the new file awaits the signal the code under test sets, through
kiro_crew.testing.wait(await_until,finished), so no fixed short timeout decides a result and the wait-discipline ratchet stays level. The file passed 20 of 20 runs at-n0, in reverse order, and in a seeded shuffled order. The owner-level tests hold their cut stop on an event the test cancels, set the tab-close bound to 0 instead of waiting it out, and wait throughawait_untilandfinishedtoo; each passed 20 of 20 runs at-n0, and their two files passed in a seeded shuffled order at-n0(89 passed) and at-n 4 --dist loadgroup. The related backend set (exit path, harness lifecycle and auto-install, instance restart, composition contracts, crash dump, SEL hard-exit flush, the wait-discipline ratchet) passed at-n 4 --dist loadgrouptogether with the supervisor and tunnel-manager tests and the wait-discipline ratchet: 2597 passed. The flake-pattern ratchet, flake8, isort, mypy and the black, comment-history and brand gates passed.Manual verification
The reproduction above, on main and then on this branch, with the same isolated home and
/api/logsheld open:Graceful shutdown timed outloggedbrowser_harness.daemonafter the exitEvery gateway and leftover process from these runs was stopped, and the home was deleted.
Related Issues
Fixes #36555
Part of #26967
Sibling sweep
Swept: git grep -n "register_pre_exec_stop(" -- src/ and git grep -n "on_cleanup.append" -- src/kiro_crew/dashboard/server_runtime/
Sibling: src/kiro_crew/dashboard/server_runtime/instances_lifecycle.py:203 -- changed here
Sibling: src/kiro_crew/browser_harness/supervisor.py:757 -- changed here
Sibling: src/kiro_crew/instances/ssh_tunnel_manager_lifecycle.py:865 -- changed here
Sibling: src/kiro_crew/dashboard/server_runtime/prevent_sleep.py:118 -- not affected -- its hook unregisters and releases with no await in between, so it cannot be cut mid-stop
Sibling: src/kiro_crew/dashboard/server_runtime/heartbeat.py:104 -- not affected -- never unregistered, and the watchdog stop is idempotent
Sibling: src/kiro_crew/slack/gateway_runtime/service_boot.py:319 -- not affected -- never unregistered, and the ledger flush is idempotent
Pattern harvest
Rule candidate: review-prompt
Pattern: a cleanup hook that unregisters its fallback stop before awaiting the stop loses both when the hook is cancelled mid-stop, and so does a stop that forgets its child before that child's stop returns.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)