Repository navigation
fix: wait for a pod's whole stop before pod down reports - #33272
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the stop budget now stacks consistently from the unit up to the MCP client, and an existing pod picks up the new Notes
[DESIGN-REVIEWED] 2eab5ba Description read[DESCRIPTION-READ] 158e2ca2feed1790227b170a74c60f44e130c30ed15449e003c36141ea29aedd That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS The fix targets the cause: the stop waits were shorter than systemd's stop budget. The one stop path that bypasses Notes
What this change shipsInventory (7 items) — 6 justifiedIntent: stopping a busy pod reports success once it finishes stopping, not a false failure while it is still going down. FIX.
[FIRST-PRINCIPLES-REVIEWED] 2eab5ba Description read[DESCRIPTION-READ] 671fb02ecbe672125288f6ad2928ae3115215b8c3fa9401247c24cc489c5f8e6 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
f6c57a8 to
5130b37
Compare
5130b37 to
b11db3f
Compare
b11db3f to
1831802
Compare
|
span=d74a20561d1d
|
`systemctl --user stop` returns when the stop job ends. The pod unit set no TimeoutStopSec, so systemd allowed its 90 s default, while pod down waited 15 s for systemctl and Dev Fleet 30 s for pod down. A pod whose gateway took longer than 15 s to exit was reported as a failed stop and then stopped anyway. The unit now sets TimeoutStopSec to the live gateway unit's shutdown budget, and is stale without it so the next start re-renders it. The systemctl stop wait covers SIGTERM and SIGKILL, Dev Fleet's pod down bound covers that plus the drain, and the pod_down MCP timeout is above Dev Fleet's. Closes #33270
1831802 to
2eab5ba
Compare
|
macOS Tests / Backend Tests (macOS) (8) on 2eab5ba fails in test/test_dashboard_server_composition_contract_owners.py::test_the_watchdog_budget_falls_back_to_the_managed_service_grace (the watchdog now gets loop_major_faults and paging_grace_max on macOS). This PR touches neither the dashboard watchdog nor that test; main has the same failure, tracked in #33311 with the fix in #33322. The (5) job was cancelled after that failure. |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix that sets TimeoutStopSec on the pod unit and makes each wait above systemctl stop longer than the one below, so a pod still stopping is not reported as a failed pod down. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/dev-fleet.md.
`systemctl --user stop` returns when the stop job ends. The pod unit set no TimeoutStopSec, so systemd allowed its 90 s default, while pod down waited 15 s for systemctl and Dev Fleet 30 s for pod down. A pod whose gateway took longer than 15 s to exit was reported as a failed stop and then stopped anyway. The unit now sets TimeoutStopSec to the live gateway unit's shutdown budget, and is stale without it so the next start re-renders it. The systemctl stop wait covers SIGTERM and SIGKILL, Dev Fleet's pod down bound covers that plus the drain, and the pod_down MCP timeout is above Dev Fleet's. Closes #33270 (cherry picked from commit 8509138)
Kiro Crew's built-in Dev Fleet moved on after this tree was cut from main at 6d07df11. An adopted host runs this tree, so each of those changes is carried here first, up to main at 46f62dea: - pod_down waits for the pod's whole stop, and the tool waits 120s (kirodotdev/KiroCrew#33272) - pod_up names a worktree hidden in a session's scratch, and its description says where to create one (kirodotdev/KiroCrew#33254) - the how-to sentence is one message per language (kirodotdev/KiroCrew#33327), rendered by i18nNodes, the bundle's stand-in for the dashboard's <Trans components> - Create worktree from latest stable, with Refresh, in a labelled More actions menu (kirodotdev/KiroCrew#33391), on the page's own MenuBtn - pod_up's test-double env override (kirodotdev/KiroCrew#35199) - provision checks the build input (kirodotdev/KiroCrew#22686), a clean closed worktree can prune (kirodotdev/KiroCrew#31124), the secret comes from the spawn environment (kirodotdev/KiroCrew#34762), every bad-body 400 carries a code (kirodotdev/KiroCrew#32503), and the demo skill's Windows tool lookup (kirodotdev/KiroCrew#33556, #33779) app.json now carries the wheel copy's agentTools block byte for byte, which an adopted record needs for its tools to stay listed (kirodotdev/KiroCrew#33141). The suite runs against Kiro Crew main at 46f62dea, the fixtures match that wheel copy, and PORTING.md records each port. Co-authored-by: Bolin Chen <bolichen@amazon.com>
Problem / Motivation
Goal:
pod_downwaits for a pod to finish stopping and reports success, instead of reporting a failure while the pod is still stopping.A
pod_downon a pod that had served a real model turn failed insystemctl --user stop; the pod stopped a little later. A fresh pod stopped in time.Why it matters
Every agent and dashboard user who stops a busy pod can get a false failure, then retries or reports a bug for a pod that is already going away. Dev Fleet's worktree removal stops the pod through the same call, so a removal can fail the same way.
Not a goal
What changed (motivation → approach → change)
Symptom:
pod_downfailed whilesystemctl --user stopwas still running.Cause:
pod.runtime.systemctlwaits 15 s by default, andhalt_pod/stop_podused that forstop. The pod unit set noTimeoutStopSec, so systemd allowed its 90 s default before SIGKILL. Above that, Dev Fleet's_pod_downboundedkirocrew pod downat 30 s, less than a stop plus the 15 s cgroup drain.Change: one stop budget, and each wait above it longer than the one below.
TimeoutStopSecPOD_STOP_TIMEOUT_SECS= live unit budget (20 s)systemctl stopinhalt_pod/stop_podSYSTEMCTL_STOP_WAIT_SECS= 2 × 20 + 5 = 45 s (SIGTERM, then SIGKILL)kirocrew pod down_POD_DOWN_TIMEOUT_SECS= 45 + 45 = 90 s (unit refresh, drain, CLI start)pod_downMCP clientTimeoutStopSec=joinsunit._REQUIRED_DIRECTIVES, so an installed unit without it is stale and the nextpod upre-renders it (the same upgrade pathLimitCORE=uses).halt_podandstop_podtake the existing pre-stop refresh (_refresh_stale_unit: re-render,daemon-reload) when the unit file is stale as well as when a teardown hook is loaded, so a pod started before the upgrade gets the new stop timeout before its stop. A failed refresh still refuses the stop. The 20 s budget isgateway_shutdown_budget.TOTAL_SHUTDOWN_BUDGET_SECS, which the live gateway unit already uses: a pod runs the same gateway.dev_fleet/worktree_ops.pynames the two new names its facade serves in itsTYPE_CHECKINGmanifests, as the facade guard tests require.docs/system-specs/modules/dev-fleet.mdlists the new required directive and adds a Pod stop timeout section.builder-security checked: no listener, bind, auth or sandbox change. Only wait bounds and one unit directive change.
Backwards compatibility
Compatible: only timeouts grow, and the unit gains one directive. An installed unit without it is re-rendered on the next start, as for the earlier required directives.
Removes nothing: no route, field, tool or behaviour is taken away.
Tests
Proof: not provable -- other: prove.py reverts every production hunk at once, including the constants the new test imports, so the mutated run fails at collection; with the source reverted and the constants stubbed in the test, 5 of the 6 new tests fail
test/test_pod_stop_timeout.py:TimeoutStopSec=<POD_STOP_TIMEOUT_SECS>and it is a required directive;stop_podandhalt_podpassSYSTEMCTL_STOP_WAIT_SECStosystemctl stop;_pod_downhands_run_cmdthat bound;stopin bothstop_podandhalt_pod, and a current one is not (these fail with theruntime_lifecycle.pychange reverted).test_pod_teardown.py: the two no-refresh ordering tests pin a current unit file.stop_pod/halt_podtests are POSIX-only (requires_posix_pod_lifecycle), like the existing teardown tests.test_pod.py,test_pod_teardown.py,test_pod_systemd.py,test_pod_limits.py,test_pod_refusals.py,test_pod_windows.py,test_dev_fleet_pod_down_orphan.py,test_dev_fleet_pod_agent_api.py,test_dev_fleet_app_guards.py,test_dev_fleet_app_context.py,test_dev_fleet_server_coverage.py,test_flake_pattern_ratchet.py: 712 passed, 4 skipped (earlier revision); on the current head the pod, launchd, Windows, Dev Fleet pod and ratchet set passes 592, 3 skipped, andtest_worktree_ops_live_facades.pyplustest_facade_mirror_guards.pypass 77.Manual verification
Not run against a live pod: this host's shell cannot reach the systemd user bus, and the live gateway runs
main, so the pod tools would exercise the old waits. The unit tests drive the exactsystemctland_run_cmdcalls.Screenshots / video
N/A - no frontend change.
Related Issues
Closes #33270
Sibling sweep
Swept: git grep -n 'systemctl("stop"' -- src/kiro_crew/pod
Sibling: src/kiro_crew/pod/runtime_lifecycle.py -- changed here -- halt_pod and stop_pod are the only systemctl stop calls
Sibling: src/kiro_crew/apps/builtins/dev_fleet/worktree_ops_removal.py -- not affected -- it stops the pod in-process through stop_pod, so it gets the new systemctl wait
Pattern harvest
Not generalizable: one unit kept systemd's default stop timeout while its caller assumed a shorter one; the live unit already pins its own.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.