Skip to content

fix: wait for a pod's whole stop before pod down reports - #33272

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
crew/slack/pod-down-stop-timeout
Nov 8, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
crew/slack/pod-down-stop-timeout

Conversation

@buluoray

@buluoray buluoray commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: pod_down waits for a pod to finish stopping and reports success, instead of reporting a failure while the pod is still stopping.

A pod_down on a pod that had served a real model turn failed in systemctl --user stop; the pod stopped a little later. A fresh pod stopped in time.

Why it matters

Every agent and dashboard user who stops a busy pod can get a false failure, then retries or reports a bug for a pod that is already going away. Dev Fleet's worktree removal stops the pod through the same call, so a removal can fail the same way.

Not a goal

  • No change to how a pod shuts down, or to the drain, reclaim and verify steps after the stop.
  • No change to the macOS (launchd) or Windows pod backends.
  • No change to the live gateway unit.

What changed (motivation → approach → change)

Symptom: pod_down failed while systemctl --user stop was still running.

Cause: pod.runtime.systemctl waits 15 s by default, and halt_pod / stop_pod used that for stop. The pod unit set no TimeoutStopSec, so systemd allowed its 90 s default before SIGKILL. Above that, Dev Fleet's _pod_down bounded kirocrew pod down at 30 s, less than a stop plus the 15 s cgroup drain.

Change: one stop budget, and each wait above it longer than the one below.

Wait Before After
unit TimeoutStopSec systemd default (90 s) POD_STOP_TIMEOUT_SECS = live unit budget (20 s)
systemctl stop in halt_pod / stop_pod 15 s SYSTEMCTL_STOP_WAIT_SECS = 2 × 20 + 5 = 45 s (SIGTERM, then SIGKILL)
Dev Fleet kirocrew pod down 30 s _POD_DOWN_TIMEOUT_SECS = 45 + 45 = 90 s (unit refresh, drain, CLI start)
pod_down MCP client 60 s 120 s, so the gateway's bound fires first and names the cause

TimeoutStopSec= joins unit._REQUIRED_DIRECTIVES, so an installed unit without it is stale and the next pod up re-renders it (the same upgrade path LimitCORE= uses). halt_pod and stop_pod take the existing pre-stop refresh (_refresh_stale_unit: re-render, daemon-reload) when the unit file is stale as well as when a teardown hook is loaded, so a pod started before the upgrade gets the new stop timeout before its stop. A failed refresh still refuses the stop. The 20 s budget is gateway_shutdown_budget.TOTAL_SHUTDOWN_BUDGET_SECS, which the live gateway unit already uses: a pod runs the same gateway.

dev_fleet/worktree_ops.py names the two new names its facade serves in its TYPE_CHECKING manifests, as the facade guard tests require. docs/system-specs/modules/dev-fleet.md lists the new required directive and adds a Pod stop timeout section.

builder-security checked: no listener, bind, auth or sandbox change. Only wait bounds and one unit directive change.

Backwards compatibility

Compatible: only timeouts grow, and the unit gains one directive. An installed unit without it is re-rendered on the next start, as for the earlier required directives.

Removes nothing: no route, field, tool or behaviour is taken away.

Tests

Proof: not provable -- other: prove.py reverts every production hunk at once, including the constants the new test imports, so the mutated run fails at collection; with the source reverted and the constants stubbed in the test, 5 of the 6 new tests fail

  • New test/test_pod_stop_timeout.py:
    • the rendered unit carries TimeoutStopSec=<POD_STOP_TIMEOUT_SECS> and it is a required directive;
    • stop_pod and halt_pod pass SYSTEMCTL_STOP_WAIT_SECS to systemctl stop;
    • the stop wait covers SIGTERM plus SIGKILL;
    • Dev Fleet's bound exceeds the stop wait plus the drain, and the MCP timeout exceeds Dev Fleet's;
    • _pod_down hands _run_cmd that bound;
    • with no hook loaded, a stale unit is re-rendered and reloaded before stop in both stop_pod and halt_pod, and a current one is not (these fail with the runtime_lifecycle.py change reverted).
  • test_pod_teardown.py: the two no-refresh ordering tests pin a current unit file.
  • The stop_pod / halt_pod tests are POSIX-only (requires_posix_pod_lifecycle), like the existing teardown tests.
  • test_pod.py, test_pod_teardown.py, test_pod_systemd.py, test_pod_limits.py, test_pod_refusals.py, test_pod_windows.py, test_dev_fleet_pod_down_orphan.py, test_dev_fleet_pod_agent_api.py, test_dev_fleet_app_guards.py, test_dev_fleet_app_context.py, test_dev_fleet_server_coverage.py, test_flake_pattern_ratchet.py: 712 passed, 4 skipped (earlier revision); on the current head the pod, launchd, Windows, Dev Fleet pod and ratchet set passes 592, 3 skipped, and test_worktree_ops_live_facades.py plus test_facade_mirror_guards.py pass 77.
  • flake8, black, isort, mypy on the changed files, and the comment-history, brand-name and file-line-cap checks pass.

Manual verification

Not run against a live pod: this host's shell cannot reach the systemd user bus, and the live gateway runs main, so the pod tools would exercise the old waits. The unit tests drive the exact systemctl and _run_cmd calls.

Screenshots / video

N/A - no frontend change.

Related Issues

Closes #33270

Sibling sweep

Swept: git grep -n 'systemctl("stop"' -- src/kiro_crew/pod
Sibling: src/kiro_crew/pod/runtime_lifecycle.py -- changed here -- halt_pod and stop_pod are the only systemctl stop calls
Sibling: src/kiro_crew/apps/builtins/dev_fleet/worktree_ops_removal.py -- not affected -- it stops the pod in-process through stop_pod, so it gets the new systemctl wait

Pattern harvest

Not generalizable: one unit kept systemd's default stop timeout while its caller assumed a shorter one; the live unit already pins its own.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@buluoray
buluoray requested review from a team and KiroCrewCaptain as code owners November 8, 2026 16:03
@buluoray
buluoray requested a review from tim-finnigan November 8, 2026 16:03
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 2eab5ba9e776cd40ca787cf8b58110061515e929 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 2eab5ba9e776cd40ca787cf8b58110061515e929 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the stop budget now stacks consistently from the unit up to the MCP client, and an existing pod picks up the new TimeoutStopSec= through the refresh path already used for required directives.

Notes

  • This adds a new refusal case. A stale unit with no teardown hook used to stop even when daemon-reload failed; now that pod down is refused. Consider letting a no-hook stale refresh failure log and go ahead, since no HOME-wiping hook is at stake there.

[DESIGN-REVIEWED] 2eab5ba

Description read

[DESCRIPTION-READ] 158e2ca2feed1790227b170a74c60f44e130c30ed15449e003c36141ea29aedd

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 2eab5ba9e776cd40ca787cf8b58110061515e929 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix targets the cause: the stop waits were shorter than systemd's stop budget. The one stop path that bypasses halt_pod, restart_pod, already runs through it. One framing tension is noted below.

Notes

  • Item 1 — inherited: ## Not a goal says "No change to how a pod shuts down", but TimeoutStopSec={stop_timeout} moves SIGKILL for a busy pod from systemd's 90 s to 20 s. The stated reason is symmetry: "it gets the live unit's budget". Issue pod_down reports failure while the pod is still stopping #33270 only shows a stop that ran past 15 s, not one that needs less than 20 s. Subtraction: drop TimeoutStopSec=, its _REQUIRED_DIRECTIVES entry and the pre-stop stale-unit refresh (items 1, 5 and 6), and size SYSTEMCTL_STOP_WAIT_SECS from systemd's default instead.

What this change ships

Inventory (7 items) — 6 justified

Intent: stopping a busy pod reports success once it finishes stopping, not a false failure while it is still going down. FIX.

  1. A pod's gateway is now SIGKILLed after 20 s instead of systemd's 90 s default — inherited (live-unit symmetry)
  2. systemctl stop in pod down/halt/restart waits 45 s instead of 15 s — justified
  3. Dev Fleet's pod down subprocess bound goes from 30 s to 90 s — justified
  4. The pod_down MCP client timeout goes from 60 s to 120 s — justified
  5. A stop now re-renders and reloads a stale unit even with no teardown hook loaded, and refuses to stop if that fails — justified
  6. An installed unit without TimeoutStopSec= reads as stale and is re-rendered on the next pod up — justified
  7. New constants POD_STOP_TIMEOUT_SECS, SYSTEMCTL_STOP_WAIT_SECS and _POD_DOWN_TIMEOUT_SECS — justified

[FIRST-PRINCIPLES-REVIEWED] 2eab5ba

Description read

[DESCRIPTION-READ] 671fb02ecbe672125288f6ad2928ae3115215b8c3fa9401247c24cc489c5f8e6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 2eab5ba9e776cd40ca787cf8b58110061515e929 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2eab5ba

Verdict parsed from the review's SHA-scoped output markers for commit 2eab5ba9e776cd40ca787cf8b58110061515e929.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 2eab5ba9e776cd40ca787cf8b58110061515e929 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 2eab5ba9e776cd40ca787cf8b58110061515e929 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2eab5ba

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 2eab5ba9e776cd40ca787cf8b58110061515e929 false-positive: <reason, 20 to 500 characters>

@buluoray
buluoray force-pushed the crew/slack/pod-down-stop-timeout branch from f6c57a8 to 5130b37 Compare November 8, 2026 16:14
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/pod-down-stop-timeout branch from 5130b37 to b11db3f Compare November 8, 2026 16:56
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray
buluoray force-pushed the crew/slack/pod-down-stop-timeout branch from b11db3f to 1831802 Compare November 8, 2026 17:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 8, 2026
@buluoray

buluoray commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

span=d74a20561d1d

  • Refresh existing units before applying the new stop wait

fixed in 1831802: halt_pod and stop_pod take the existing pre-stop refresh (_refresh_stale_unit: re-render + daemon-reload) when the unit file is stale (unit_is_current false, e.g. it lacks TimeoutStopSec=), not only when a teardown hook is loaded, so a pod started before the upgrade gets the new stop timeout before systemctl stop. A failed refresh still refuses the stop. Tests in test/test_pod_stop_timeout.py cover both functions and fail with the change reverted.

`systemctl --user stop` returns when the stop job ends. The pod unit
set no TimeoutStopSec, so systemd allowed its 90 s default, while pod
down waited 15 s for systemctl and Dev Fleet 30 s for pod down. A pod
whose gateway took longer than 15 s to exit was reported as a failed
stop and then stopped anyway.

The unit now sets TimeoutStopSec to the live gateway unit's shutdown
budget, and is stale without it so the next start re-renders it. The
systemctl stop wait covers SIGTERM and SIGKILL, Dev Fleet's pod down
bound covers that plus the drain, and the pod_down MCP timeout is
above Dev Fleet's.

Closes #33270
@buluoray
buluoray force-pushed the crew/slack/pod-down-stop-timeout branch from 1831802 to 2eab5ba Compare November 8, 2026 17:52
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026
@buluoray

buluoray commented Nov 8, 2026

Copy link
Copy Markdown
Contributor Author

macOS Tests / Backend Tests (macOS) (8) on 2eab5ba fails in test/test_dashboard_server_composition_contract_owners.py::test_the_watchdog_budget_falls_back_to_the_managed_service_grace (the watchdog now gets loop_major_faults and paging_grace_max on macOS). This PR touches neither the dashboard watchdog nor that test; main has the same failure, tracked in #33311 with the fix in #33322. The (5) job was cancelled after that failure.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 8, 2026 19:05

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (11 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix that sets TimeoutStopSec on the pod unit and makes each wait above systemctl stop longer than the one below, so a pod still stopping is not reported as a failed pod down. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/dev-fleet.md.

@KiroCrewCaptain
KiroCrewCaptain merged commit 8509138 into main Nov 8, 2026
103 of 105 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the crew/slack/pod-down-stop-timeout branch November 8, 2026 19:05
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
bolichen97 pushed a commit that referenced this pull request Nov 9, 2026
`systemctl --user stop` returns when the stop job ends. The pod unit
set no TimeoutStopSec, so systemd allowed its 90 s default, while pod
down waited 15 s for systemctl and Dev Fleet 30 s for pod down. A pod
whose gateway took longer than 15 s to exit was reported as a failed
stop and then stopped anyway.

The unit now sets TimeoutStopSec to the live gateway unit's shutdown
budget, and is stale without it so the next start re-renders it. The
systemctl stop wait covers SIGTERM and SIGKILL, Dev Fleet's pod down
bound covers that plus the drain, and the pod_down MCP timeout is
above Dev Fleet's.

Closes #33270

(cherry picked from commit 8509138)
KiroCrewCaptain added a commit to KiroCrewCaptain/kirocrew-app-dev-fleet that referenced this pull request Nov 14, 2026
Kiro Crew's built-in Dev Fleet moved on after this tree was cut from
main at 6d07df11. An adopted host runs this tree, so each of those
changes is carried here first, up to main at 46f62dea:

- pod_down waits for the pod's whole stop, and the tool waits 120s
  (kirodotdev/KiroCrew#33272)
- pod_up names a worktree hidden in a session's scratch, and its
  description says where to create one (kirodotdev/KiroCrew#33254)
- the how-to sentence is one message per language
  (kirodotdev/KiroCrew#33327), rendered by i18nNodes, the bundle's
  stand-in for the dashboard's <Trans components>
- Create worktree from latest stable, with Refresh, in a labelled More
  actions menu (kirodotdev/KiroCrew#33391), on the page's own MenuBtn
- pod_up's test-double env override (kirodotdev/KiroCrew#35199)
- provision checks the build input (kirodotdev/KiroCrew#22686), a clean
  closed worktree can prune (kirodotdev/KiroCrew#31124), the secret
  comes from the spawn environment (kirodotdev/KiroCrew#34762), every
  bad-body 400 carries a code (kirodotdev/KiroCrew#32503), and the
  demo skill's Windows tool lookup (kirodotdev/KiroCrew#33556, #33779)

app.json now carries the wheel copy's agentTools block byte for byte,
which an adopted record needs for its tools to stay listed
(kirodotdev/KiroCrew#33141). The suite runs against Kiro Crew main at
46f62dea, the fixtures match that wheel copy, and PORTING.md records
each port.

Co-authored-by: Bolin Chen <bolichen@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pod_down reports failure while the pod is still stopping

2 participants