Skip to content

fix: stop the browser harness when graceful shutdown runs out of time - #36647

Merged
bolichen97 merged 1 commit into
mainfrom
fix/36555-harness-survives-forced-exit
Nov 14, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/36555-harness-survives-forced-exit

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway whose graceful shutdown runs out of time still stops the browser harness daemon before it exits.

Stop a gateway while the dashboard Logs page is open, and the browser harness daemon keeps running after it exits. Reproduced on main with an isolated home and a loopback-only gateway (gateway --test-mode --no-crons --no-tunnel), after the startup auto-install had warmed the harness: with GET /api/logs held open, SIGTERM makes the gateway log Graceful shutdown timed out and exit after 10 seconds, and python -m browser_harness.daemon is still running, reparented to PID 1.

Why it matters

A stopped gateway should leave nothing running. The daemon keeps its IPC socket and its pid file while no gateway owns it, until the next boot reclaims it. Any operator who stops the gateway while a long request is open (the Logs page, a streamed reply) is hit.

Not a goal

  • Making the graceful shutdown fit its budget, for example by bounding the web server's request drain.
  • The app backends the same timed-out shutdown leaves running. fix: stop spawned app backends when graceful shutdown runs out of time #36558 stops them from the exit path.
  • The second-signal force exit (_force_exit), which ends the process from its own thread without the exit path's stops.
  • Other processes a timed-out shutdown leaves behind that no cleanup hook owns.

What changed (motivation → approach → change)

Symptom → cause. The harness supervisor is stopped only by a dashboard on_cleanup hook. The web server runs cleanup hooks only after its in-flight requests drain, and an open /api/logs stream holds that drain past the 10-second graceful budget, so the hook never runs and os._exit leaves the daemon behind.

Approach. The in-place restart skips the same hooks, so the harness (and the instance tunnels and the sleep inhibitor) already register the same stop with the pre-exec seam, pre_exec_stops. The exit path now runs that seam once before its hard exit, instead of naming any owner.

Change.

Where What
slack/gateway_runtime/exit_path.py _shutdown_and_exit awaits run_pre_exec_stops() after the orphan sweep and the shared crew release, before the drains and os._exit. It is bounded inside the seam (10 s) and never raises.
dashboard/server_runtime/browser_lifecycle.py, instances_lifecycle.py Each cleanup hook unregisters its stop only once the stop has returned. The graceful bound can cancel a hook mid-stop; the stop then stays registered and the exit path runs it again.
browser_harness/supervisor.py, instances/ssh_tunnel_manager_lifecycle.py That second run reaches what the cut one did not. A supervisor teardown cut before its last child sets that handle aside, never to be reused as the live one, so the next stop or start stops the daemon, relay and browser it never got to, without retrying the tab close. The best-effort tab close is bounded at 3 s, so a stalled CDP connection cannot spend the seam's bound first. The tunnel manager's shutdown() drops a tunnel only after that tunnel's stop() returns.
pre_exec_stops.py run_pre_exec_stops takes an occasion ("restart", the default, or "exit") so a stop that fails or overruns at exit is not logged as "before restart". The module docstring names the exit path as a caller, and that each stop must be safe to repeat.

After a shutdown that did reach the hooks, the harness and instance stops are unregistered. What is left (the skill-usage flush and the loop-stall watchdog stop) is idempotent. Signals still go through the supervisor's existing platform_compat.terminate_and_reap; nothing new is signalled. The slack-gateway, browser and instances specs describe the exit-path call and the repeatable stops.

Backwards compatibility

Compatible: a shutdown that finished in time finds only idempotent stops left. One that ran out of time takes as long as the harness stop (bounded at 10 s) and stops a daemon that was left running before.

Removes nothing: it adds a call to the exit path and moves two unregister calls and one tunnel removal after their stops; no path, record or key is taken away.

Tests

New test/test_gateway_exit_pre_exec_stops.py, driving the real startup and cleanup hooks with the supervisor and tunnel manager replaced:

  • test_a_timed_out_shutdown_still_stops_the_browser_harness: no cleanup hook ran; the exit path stops the harness before its hard exit.
  • test_a_cleanup_hook_cut_mid_stop_leaves_the_stop_to_the_exit_path: the harness hook is cancelled mid-stop; the exit path runs the stop again.
  • test_a_cleanup_hook_that_finished_leaves_nothing_to_stop_again: a hook that finished leaves nothing registered, and the exit path does not stop the harness twice.
  • test_an_instance_teardown_cut_mid_stop_leaves_it_to_the_exit_path: the same as the second test, for the instance tunnels.
  • test_a_failing_stop_is_logged_as_before_the_exit_and_the_exit_still_runs: a stop that raises is logged as before the exit, and the hard exit still runs.

Owner-level tests run the real stop:

  • test/test_browser_harness_supervisor_cov.py::test_a_stop_cut_mid_teardown_leaves_the_rest_to_the_next_stop: a supervisor stop() cancelled while it stops the worker sets its handle aside (not connected, still holds_browser), and the next stop() stops the worker, daemon and browser and closes the relay.
  • test_a_stop_cut_in_a_stalled_tab_close_skips_it_on_the_next_stop and test_a_stalled_tab_close_is_bounded_and_the_children_are_still_stopped (same file): a stop cut in a tab close that never answers does not wait on it again, and a stalled tab close alone still lets the worker, daemon and browser stop.
  • test/test_instances.py::TestSshTunnelManager::test_a_shutdown_cut_mid_stop_leaves_the_tunnel_to_the_next_shutdown: a shutdown() cancelled inside a tunnel's stop() leaves that tunnel in the manager, and the next shutdown() stops it.

Proof: PROVEN test/test_browser_harness_supervisor_cov.py::test_a_stop_cut_in_a_stalled_tab_close_skips_it_on_the_next_stop, test/test_gateway_exit_pre_exec_stops.py::test_a_timed_out_shutdown_still_stops_the_browser_harness, test/test_gateway_exit_pre_exec_stops.py::test_a_cleanup_hook_cut_mid_stop_leaves_the_stop_to_the_exit_path, test/test_gateway_exit_pre_exec_stops.py::test_an_instance_teardown_cut_mid_stop_leaves_it_to_the_exit_path, test/test_gateway_exit_pre_exec_stops.py::test_a_failing_stop_is_logged_as_before_the_exit_and_the_exit_still_runs (head 6c67fad)

Each production hunk was also reverted on its own: without the exit-path call four tests fail, without either reorder the matching cut-mid-stop test fails, without the supervisor's set-aside handle, its retry skipping the tab close, its tab-close bound, or the tunnel manager's later removal the matching owner-level test fails, and without occasion="exit" the logging test fails. The restart's own log lines are unchanged; their tests in test_restart_stops_instance_tunnels.py pass.

Determinism: every wait in the new file awaits the signal the code under test sets, through kiro_crew.testing.wait (await_until, finished), so no fixed short timeout decides a result and the wait-discipline ratchet stays level. The file passed 20 of 20 runs at -n0, in reverse order, and in a seeded shuffled order. The owner-level tests hold their cut stop on an event the test cancels, set the tab-close bound to 0 instead of waiting it out, and wait through await_until and finished too; each passed 20 of 20 runs at -n0, and their two files passed in a seeded shuffled order at -n0 (89 passed) and at -n 4 --dist loadgroup. The related backend set (exit path, harness lifecycle and auto-install, instance restart, composition contracts, crash dump, SEL hard-exit flush, the wait-discipline ratchet) passed at -n 4 --dist loadgroup together with the supervisor and tunnel-manager tests and the wait-discipline ratchet: 2597 passed. The flake-pattern ratchet, flake8, isort, mypy and the black, comment-history and brand gates passed.

Manual verification

The reproduction above, on main and then on this branch, with the same isolated home and /api/logs held open:

main this branch
Graceful shutdown timed out logged yes yes
exit after SIGTERM 10.0 s 10.1 s
browser_harness.daemon after the exit running, parent PID 1 gone

Every gateway and leftover process from these runs was stopped, and the home was deleted.

Related Issues

Fixes #36555
Part of #26967

Sibling sweep

Swept: git grep -n "register_pre_exec_stop(" -- src/ and git grep -n "on_cleanup.append" -- src/kiro_crew/dashboard/server_runtime/
Sibling: src/kiro_crew/dashboard/server_runtime/instances_lifecycle.py:203 -- changed here
Sibling: src/kiro_crew/browser_harness/supervisor.py:757 -- changed here
Sibling: src/kiro_crew/instances/ssh_tunnel_manager_lifecycle.py:865 -- changed here
Sibling: src/kiro_crew/dashboard/server_runtime/prevent_sleep.py:118 -- not affected -- its hook unregisters and releases with no await in between, so it cannot be cut mid-stop
Sibling: src/kiro_crew/dashboard/server_runtime/heartbeat.py:104 -- not affected -- never unregistered, and the watchdog stop is idempotent
Sibling: src/kiro_crew/slack/gateway_runtime/service_boot.py:319 -- not affected -- never unregistered, and the ledger flush is idempotent

Pattern harvest

Rule candidate: review-prompt
Pattern: a cleanup hook that unregisters its fallback stop before awaiting the stop loses both when the hook is cancelled mid-stop, and so does a stop that forgets its child before that child's stop returns.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 13, 2026 14:06
@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 14:06
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 4ea644a04c00c2c89211bbcae5e6d0d0200452cd and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4ea644a

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 4ea644a04c00c2c89211bbcae5e6d0d0200452cd false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Nov 13, 2026
@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from 7526e0a to 4a05c48 Compare November 13, 2026 19:22
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Retrying an interrupted stop loses the children it must terminate (span=07083213b206)

Rebutted: the goal's daemon is reached at every cut point. HarnessSupervisor._teardown clears _handle first, so the exit-path retry takes the _shutdown_stray_daemon branch, which pings the daemon over IPC and shuts it down; a cut after _ipc_shutdown has already sent that shutdown.
What a cut stop can leave is the one child mid-teardown (a Windows Chromium or worker, or the one tunnel SshTunnelManager.shutdown already popped); every tunnel not yet popped and the hop-port guard are still stopped by the retry.
Not a regression: before this PR the hook unregistered first, so the same cut left everything with no retry. The next boot's reclaim (_reclaim_then_auto_setup) and hop_port_guard recover the residue.
Making the supervisor and tunnel teardown resumable changes two owners' handle lifetimes outside the frozen goal; the adjudicator downgraded it as disproportionate. A resource leak, not security or data loss.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Startup auto-setup can launch a daemon after the final stop (span=07083213b206)

Rebutted: the race needs a shutdown during the startup auto-install AND a held request drain; the registered stop and the startup task it races are unchanged by this PR, and the in-place restart runs the same stop with the same race.
Not a regression: before this PR the exit path ran no stop at all, so a daemon auto-setup started was left the same way; this PR only adds a stop before the exit.
The next gateway start runs _stop_supervisor first (browser_lifecycle.py), which shuts a stray daemon down over IPC, so the leak clears itself.
Cancelling auto-setup and the install job inside the shared stop adds a cross-module cancel order to the restart path too, which is new scope; the adjudicator downgraded it as disproportionate. A resource leak, not security or data loss.

@bolichen97

Copy link
Copy Markdown
Contributor Author

CI on 7526e0a, for the record:

  • Backend Tests (3.12, 5) and Backend Tests (Windows) (2): test_wait_discipline_ratchet.py (test_no_file_gains_a_short_fixed_wait, test_the_scan_matches_known_sites) counted five 10 s asyncio.wait_for ceilings in the new test file. That was this PR's; 4a05c48 has the file await its signals through kiro_crew.testing.wait (await_until, finished) instead. Both ratchet tests pass locally on the new head.
  • Backend Tests (3.12, 7): "The self-hosted runner lost communication with the server", with no test failure. That shard's known runner trouble is tracked in Flaky: test_ssrf_guard_fail_closed node subprocess tests time out on Linux shard 7 #36261 and test: run the lenient-read ratchet's tree scan on one worker #36653, not this diff.
  • Coverage Gate failed only because those shards failed.
  • Design, First Principles, Opus, UX, CodeQL Analyze (actions) and Publish readiness signal hit the installation's exhausted API rate limit (0/15000 remaining) and posted no verdict. They run again on the new head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 4ea644a04c00c2c89211bbcae5e6d0d0200452cd touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 4ea644a04c00c2c89211bbcae5e6d0d0200452cd — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the exit path now reuses the existing pre_exec_stops seam instead of naming each owner, and moving each unregister to after its stop is what lets that second run happen.

Notes

  • The skill-usage flush and the loop-stall watchdog stop were registered for the restart path, and now also run on every exit. The skill-usage flush is already followed by the exit path's own ledger flush. That's harmless today, but the module docstring's "each stop must be safe to repeat" is now a requirement any future registrant has to meet.

[DESIGN-REVIEWED] 4ea644a

Description read

[DESCRIPTION-READ] 9692a1c7e0f071a3b2e97c89928b4be1b61e334e7569a37bfa0abf420b43fe6d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 4ea644a04c00c2c89211bbcae5e6d0d0200452cd — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The exit path reuses the existing pre_exec_stops seam instead of naming owners, which works on the cause (cleanup hooks skipped before os._exit), and every reorder is needed for the cut-mid-stop case.

Notes

  • Item 5 — inherited: the 3 s tab-close bound rests only on the claim that "a stalled CDP connection cannot spend the seam's bound first". The description gives no reproduction for a stall during exit, and the test sets the bound to 0. Subtraction: none needed if the stall was actually seen. Otherwise drop the _TAB_CLOSE_SECS wrapper and rely on the retry already skipping the tab close (close_tabs=not retry).

What this change ships

Inventory (8 items) — 7 justified

Intent: a stopped gateway leaves no browser-harness daemon (or instance tunnel) running after a graceful shutdown that timed out. FIX.

  1. Gateway exit now runs every registered pre-exec stop before its hard exit — justified
  2. Browser harness cleanup hook unregisters its stop only after the stop returns — justified
  3. Instances cleanup hook unregisters its stop only after the teardown returns — justified
  4. Tunnel manager forgets a tunnel only after its stop returns — justified
  5. Harness agent-tab close at teardown now gives up after 3 s — rides along, unverified stall premise
  6. A harness stop cut mid-teardown keeps its handle, and the next stop finishes it without retrying the tab close — justified
  7. Pre-exec stop failure logs now say "before exit" when the exit path runs them — justified
  8. Skill-usage flush, loop-stall watchdog and sleep-inhibitor stops now also run at every gateway exit — justified

[FIRST-PRINCIPLES-REVIEWED] 4ea644a

Description read

[DESCRIPTION-READ] cc541b6fa6b50ca7123e27cd4502a45fa3f1849a5d6484b7545ec2a839f6258a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 4ea644a04c00c2c89211bbcae5e6d0d0200452cd — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4ea644a

Verdict parsed from the review's SHA-scoped output markers for commit 4ea644a04c00c2c89211bbcae5e6d0d0200452cd.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 4ea644a04c00c2c89211bbcae5e6d0d0200452cd false-positive: <reason, 20 to 500 characters>

@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 12% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 521 lines, more than the 400 it may approve on its own.

The exit path now runs the existing pre-exec stop list once before os._exit. That list is bounded at 10 s and never raises, so a graceful shutdown that ran out of time still stops the browser harness. The harness and instance cleanup hooks now unregister their stops only after the stops return. A supervisor teardown cut partway keeps its handle aside, never reused as the live one, so the next stop or ensure reaches the children it missed and skips the tab close. Two stops at once are serialised by _ensure_lock. Tunnel shutdown removes a tunnel only after its stop returns and still holds the manager's lock. Every stop on the list is safe to repeat: sleep-inhibitor release, watchdog stop (the exit code is computed before this step), skill-usage flush. The restart path is unchanged. Tests are isolated per test and cover each changed path. No blocking defects found.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 8bed575f33 · 521 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@bolichen97

Copy link
Copy Markdown
Contributor Author

CI on 4a05c48: every backend shard passed, Linux and Windows. All review lanes ran and posted (Design and First Principles PASS, GPT and Opus with no blocking lines).

The only red is Coverage Gate: 75.7% src/kiro_crew/agent_ipc_client.py (87/115), which is under the 80% per-file floor. This PR does not touch that file. The red comes from main, after #36619 stubbed _set_non_dumpable in the autouse fixture, and it shows on every PR based on current main. #36804 fixes it. This PR waits for that fix and will rebase onto it; no override is posted for this red.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
buluoray
buluoray previously approved these changes Nov 13, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 4a05c48. When a graceful shutdown runs out of time, the gateway now runs the stops that the browser harness and the instance tunnels registered for the in-place restart before it exits, so the harness daemon no longer outlives the gateway. The stop it runs is the same one that the cleanup hook and the restart already run. It closes only the tabs the agent opened and stops only child processes the gateway started itself, so a person's own Chrome and their tabs are untouched. Each cleanup hook now drops its registration only after its own stop returns. Since the last review, the only change is in the new test file: its fixed ten-second wait is replaced by the suite's shared wait helpers, which clears the wait-discipline ratchet; the product code is unchanged. Each production change still turns at least one new test red when reverted. Follow-ups that do not block: a hook cut in the middle of the supervisor's teardown has already dropped its handle, so the stop that runs again at exit cannot reach the worker or the browser it started; and the exit path now has more bounded steps after the graceful budget than the supervisor's stop deadline leaves room for, in the rare case where every one of them stalls. Decisive lines: src/kiro_crew/slack/gateway_runtime/exit_path.py:743, src/kiro_crew/dashboard/server_runtime/browser_lifecycle.py:78-90, src/kiro_crew/dashboard/server_runtime/instances_lifecycle.py:203-214, test/test_gateway_exit_pre_exec_stops.py (the hang guard now uses kiro_crew.testing.wait)

@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from 4a05c48 to f3a02cc Compare November 13, 2026 22:53
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main at 6d61070 to pick up #36804, which fixes the main-side Coverage Gate red (src/kiro_crew/agent_ipc_client.py 75.7% under the 80% per-file floor) that 4a05c48 hit without touching that file. The diff is unchanged; the Proof line was re-run on the new head f3a02cc (PROVEN). CI re-running.

@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from f3a02cc to f2a4609 Compare November 14, 2026 00:19
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: no
mechanism: a cut stop keeps the child it has not stopped yet, so the exit path's repeat reaches it

  • A stop cut mid-teardown leaves the cut handle's children unreachable for the repeat span=0fb1bfbeb7ff

fixed in f2a4609: HarnessSupervisor._teardown puts the handle back when it is cut before its last stop, so the exit path's stop() reaches the daemon, relay and browser it never got to; already-stopped children are skipped on the repeat.
Same class, second site: SshTunnelManager.shutdown drops a tunnel only after that tunnel's stop() returns, the order _SshTunnel.stop's docstring already states for every other caller.
Swept every register_pre_exec_stop owner: the sleep inhibitor releases with no await, the watchdog and ledger stops keep their references; neither is in the class.
Pinned: test_a_stop_cut_mid_teardown_leaves_the_rest_to_the_next_stop and test_a_shutdown_cut_mid_stop_leaves_the_tunnel_to_the_next_shutdown run the real stop twice and fail with either hunk reverted.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 14, 2026
@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from f2a4609 to baba48f Compare November 14, 2026 01:01
@bolichen97

bolichen97 commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 5) on f2a4609 was this PR's own red: test_wait_discipline_ratchet.py::test_no_file_gains_a_short_fixed_wait and ::test_the_scan_matches_known_sites counted the asyncio.wait_for(..., 5) waits the two owner-level tests added. baba48f waits through kiro_crew.testing.wait.await_until / finished instead; both ratchet tests pass locally, and Coverage Gate was red only because that shard failed. Rebased onto 342409d on the way; CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 14, 2026
@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from f619717 to 6c67fad Compare November 14, 2026 01:41
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: the retry of a cut supervisor teardown skips the tab close, and the tab close is bounded

  • Retry repeats stalled tab closure before stopping the daemon span=e756ee25f7ae

fixed in 6c67fad: _teardown passes close_tabs=False when it is finishing a set-aside cut handle, so the exit path's retry goes straight to the IPC shutdown and the worker, daemon and browser stops.
Same class at the other site: a first stop on the exit path (the hook never ran) met the same stalled close; the best-effort close is now bounded at _TAB_CLOSE_SECS (3 s), under the seam's 10 s, and its timeout is swallowed like any other tab-close failure.
Pinned: test_a_stop_cut_in_a_stalled_tab_close_skips_it_on_the_next_stop (red with the skip removed) and test_a_stalled_tab_close_is_bounded_and_the_children_are_still_stopped (hangs with the bound removed).

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from 6c67fad to 8bed575 Compare November 14, 2026 03:32
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/slack-gateway.md since this head went green (538bad7 -> f9ebf3a); CI re-running.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision labels Nov 14, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 6) on 8bed575 is main's red, not this diff: test/test_search_docs.py::test_search_docs_ranks_the_page_the_query_is_about_first[...-how do I run a task in the background] (assert 'task-runner' == 'subagents') fails the same way on a clean origin/main worktree at f9ebf3a, and this PR touches nothing under src/kiro_crew/docs/. Tracked in #37060. Waiting for that fix to land, then rebasing onto it; no rerun or override for this red.

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve on 8bed575. When a graceful shutdown runs out of time, the gateway now runs the stops that the browser harness and the instance tunnels registered for the in-place restart before it exits, so the harness daemon no longer outlives the gateway. Since the last review, this head also closes the gap we noted as a follow-up. A harness stop that is cut in the middle of its teardown now keeps the half-stopped handle aside, and the next stop finishes stopping the worker, daemon, relay and browser that the cut one never reached. That retry skips the agent-tab close, and the tab close itself is now bounded at three seconds, so a stalled browser connection cannot use up the stop's time before the children stop. The tunnel manager drops each tunnel only after its stop returns, so the exit path also reaches a tunnel that a cut shutdown left mid-stop. The retry only stops child processes that the gateway itself started, through the same process handles as before, and a child that already exited is skipped, so nothing else can be stopped by mistake. Each new change turns a new test red when reverted, and the new tests wait on events with bounded waits, not sleeps. Follow-ups that do not block: a stop cut inside the relay's own close leaves the relay's local listener open until the process exits, because the relay marks itself torn down before its awaits; and the exit path still has more bounded steps after the graceful budget than the service manager's stop deadline leaves room for, in the rare case where every one of them stalls. Decisive lines: src/kiro_crew/browser_harness/supervisor.py:286 (_cut_handle), :441-446 (holds_browser counts it), :771-799 (_teardown / _stop_handle), :1264-1276 (_stop_process skips a reaped child, stops via platform_compat.terminate_and_reap); src/kiro_crew/instances/ssh_tunnel_manager_lifecycle.py:866-876 (pop after stop, under self._lock); src/kiro_crew/slack/gateway_runtime/exit_path.py (seam call, unchanged since 4a05c48)

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests (Windows) (6) on 8bed575 is the same main red as the Linux shard above, not this diff: test/test_search_docs.py::test_search_docs_ranks_the_page_the_query_is_about_first[...-how do I run a task in the background] fails with assert 'task-runner' == 'subagents' (https://github.com/kirodotdev/KiroCrew/actions/runs/43552166516/job/132333553402). It reproduces on this head locally with pytest -n0 test/test_search_docs.py -k ranks, and this PR touches nothing under src/kiro_crew/docs/ or the guide search. Tracked in #37060; fixes in flight: #37063 (auto-merge armed), #37065. No rerun (the failure is deterministic) and no override: this PR rebases onto the fix once it lands.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main 205d08d0 to pick up #37063. The shard 6 red on 8bed575f (Linux and Windows, plus the Coverage Gate that depends on it) was test/test_search_docs.py::test_search_docs_ranks_the_page_the_query_is_about_first[...how do I run a task in the background], a main red that #37063 fixes. The patch did not change (git range-diff shows =). Related tests pass locally on the rebased head b44a89a9 (479 passed). CI is running again.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 14, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Backend Tests (3.12, 2) on b44a89a9 is red in test/tracing/test_tracing_otlp.py::test_the_exit_flush_sends_spans_still_queued (assert 5 == 4) and ::test_a_traced_turn_reaches_a_loopback_receiver_with_its_vault_header (an extra kirocrew.step span). This is main's red, tracked in #37169: #37005 added the span, and the two wire tests still expect four spans. This PR does not touch tracing. Both tests pass on this head without the otlp extra (they skip). The fix is #37171. This PR waits for it to merge and then rebases onto it. No rerun and no override for this red.

A gateway stopped while a long request is open (the dashboard Logs
page) runs out of its graceful shutdown budget before the web server
reaches the dashboard's cleanup hooks, and its os._exit then leaves
the browser harness daemon running under PID 1.

The exit path now runs the pre-exec stops before its hard exit. The
harness, the instance tunnels and the sleep inhibitor already register
their stops there for the in-place restart, which skips the same hooks.
The harness and instance cleanup hooks now unregister their stop only
once it has returned, so a hook the graceful bound cancels mid-stop
leaves the stop for the exit path.

Fixes #36555
Part of #26967
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto main 6a61c846 to pick up #37171, the fix for main's shard 2 red (#37169). Every other shard on b44a89a9 passed, and all review lanes passed there too. The patch did not change (git range-diff shows =), and main did not touch any file this PR changes. The gates and related tests pass on the new head 4ea644a0 (469 passed). CI is running again.

@bolichen97
bolichen97 force-pushed the fix/36555-harness-survives-forced-exit branch from b44a89a to 4ea644a Compare November 14, 2026 11:15
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 14, 2026
@bolichen97
bolichen97 merged commit 7d7b3f9 into main Nov 14, 2026
99 checks passed
@bolichen97
bolichen97 deleted the fix/36555-harness-survives-forced-exit branch November 14, 2026 12:59
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] A timed-out graceful shutdown skips the dashboard cleanup hooks and leaves the browser harness daemon running

3 participants