Repository navigation
fix(watchdog): decide the loop-stall exit on the monotonic clock - #14302
Conversation
GPT 5.6 Review — ✅ no blocking findingsGPT 5.6 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Fable 5) — ✅ PASSDesign-level review of Design-Verdict: PASS Run the described two-minute Linux suspend/resume check before merge — the suspend-pausing alarm semantic is CI-untestable and only the maintainer recipe exercises it. [DESIGN-REVIEWED] 7c85798 |
First Principles Review (Fable 5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify the macOS premise on real hardware: that Not justified as shipped
What this change shipsInventory (10 items) — 8 justifiedIntent: the gateway survives a laptop suspend instead of dying on resume with a false crash dump — a FIX.
Root-cause note: the fix sits at CAUSE level (the wall-clock deadline itself); grep [FIRST-PRINCIPLES-REVIEWED] 7c85798 |
|
Opus 5 Review — ✅ no blocking findingsReviewed Review detailsAdvisory only: the exec seams cancel FINDING — src/kiro_crew/platform_compat.py:87 — [OPUS-REVIEWED] 7c85798 Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
a9922e2 to
1ac54c0
Compare
1ac54c0 to
aed0ddb
Compare
|
|
|
|
aed0ddb to
6e90657
Compare
|
|
|
c2cb179 to
3e253d3
Compare
The gateway died on every laptop resume. LoopStallWatchdog re-armed
faulthandler's dump_traceback_later(exit=True) timer on each 5 s
heartbeat, and that timer waits on an interpreter lock whose deadline is
an absolute CLOCK_REALTIME instant on any CPython built without
sem_clockwait (portable builds link an old glibc; every macOS build). A
suspend advances CLOCK_REALTIME by its whole length while
CLOCK_MONOTONIC stands still, so on resume the 25 s deadline was already
past and faulthandler dumped and _exit(1)'d before the loop thread,
which had missed no beat on the monotonic clock, could re-arm it. No
budget fixes a wall-clock deadline: it fires on any suspend longer than
itself, and four times the 25 s desktop budget is still shorter than
the reporter's own two-minute suspend.
The dump-then-exit now belongs to the kernel's per-process alarm. Each
beat re-arms setitimer(ITIMER_REAL) for exit_after seconds
(platform_compat.arm_process_alarm) and faulthandler.register(SIGALRM,
chain=True) on the crash-dump file makes the alarm dump every thread's
stack from inside the signal handler, in C with no GIL, then hand
SIGALRM to its default disposition, which ends the process. Linux runs
ITIMER_REAL on CLOCK_MONOTONIC (copy_signal initialises the process's
real_timer on that clock) and macOS schedules it on the absolute mach
timebase, so the alarm pauses while the host sleeps. It fires whether
the loop thread is blocked in a syscall or holding the GIL inside a
long C call. Windows has no process alarm, and a process that already
handles SIGALRM from Python (pytest-timeout in a test worker, an
embedding host) owns ITIMER_REAL too; in both cases faulthandler's
timer carries the exit at the same budget and the alarm is never armed
or cancelled. On Windows nothing new is lost: its time.monotonic()
counts a sleep as well.
What changes about a real stall: the process ends by SIGALRM instead of
status 1 and the dump carries no "Timeout (" preamble line. No consumer
of either exists (the desktop supervisor logs code and signal
generically and its stale-bundle check matches only its own code;
systemd Restart=always and launchd KeepAlive respawn on either; every
dump reader treats the preamble as optional). The dump, the file and
the next-boot replay are unchanged.
The daemon thread keeps the observability layer on the monotonic clock:
enrichment, then the soft dump, on its 5 s poll (15 s lands on the
15-20 s tick, ahead of the 25 s exit), and a resume is logged once
at INFO when platform_compat.boottime_now (CLOCK_BOOTTIME on Linux, the
wall clock on macOS; moved from acp/liveness, which re-exports it, so
dashboard code imports no ACP module) advances 2 s or more beyond the
monotonic clock across one poll. Each alarm arm releases and renews
faulthandler's SIGALRM registration: a repeat register reinstalls
nothing while faulthandler believes it still holds the signal, so a
temporary owner that handed SIGALRM back with SIG_DFL would otherwise
leave the next alarm to end the process without a dump.
The alarm belongs to the image that armed it. execve preserves
ITIMER_REAL while it resets a caught SIGALRM to its default disposition,
so an in-app restart or update would hand the successor gateway the last
heartbeat's deadline as a lethal signal it never armed, during its own
boot. Both exec seams (platform_compat.reexec_launcher and
reexec_python_module) cancel the alarm immediately before os.execv, and
the gateway entrypoint clears any deadline that still arrived
(loop_watchdog.disarm_inherited_alarm), only while SIGALRM is at its
default disposition -- the ownership rule exit_mechanism applies.
Alternatives not taken: re-arming the old timer from the heartbeat
before faulthandler's thread runs (both wake on the same resume; the C
thread needs no GIL and wins); a daemon-thread exit decision with the
alarm as a wider last resort (two deciders for one question; the alarm
alone answers it); dropping exit=True for the soft dump plus the
Electron/systemd probes (a foreground `kirocrew gateway` has no probe;
a real stall would wedge it for good).
dashboard.loop_stall_exit_after_secs keeps its meaning; no config key
is added.
Fixes #14273
Related: #13918 (a real 25 s stall on a foreground gateway, a stat walk
on the loop thread; the alarm ends that stall at the same budget)
3e253d3 to
7c85798
Compare
|
|
|
|
bolichen97
left a comment
There was a problem hiding this comment.
Reviewed on behalf of the maintainers (sweep 2026-09-28): readiness passed, required checks green on 7c85798, no blocking review threads, security posture unchanged. Approving and merging.
…15634) Backport of #13308, #14302, #14860, #14961, #15070 to release/0.8.0, cherry-picked in main's merge order. * fix(sel): anchor the chain lock on the log directory (#13308) (cherry picked from commit 998b045) * fix(watchdog): decide the loop-stall exit on the monotonic clock (#14302) (cherry picked from commit 1d37ab1) * fix(ops-mission-control): strict read in ledger resolve_conflict (#14860) (cherry picked from commit b8ea16b) * fix(env): skip a mise candidate whose probe is denied instead of aborting boot (#14961) (cherry picked from commit e054c95) * fix(lessons): retire only a finding on a contradiction verdict (#15070) (cherry picked from commit dd5e840)
…hen execv fails (#22651) Two follow-ups deferred from #14302. 1. cli.py imports disarm_inherited_alarm at module scope. The watchdog's import closure (dashboard.stall_enrichment -> acp.liveness, and platform_compat) contains no module-scope import of cli, so this creates no cycle, and acp is already loaded when cli imports session, so only the two small watchdog modules are new. The gateway dispatch keeps its order (faulthandler.enable() before disarm_inherited_alarm()). 2. The two exec seams re-arm a pre-existing ITIMER_REAL alarm when os.execv fails. Cancelling the alarm before execv is correct only when the exec succeeds: a successor image never armed it. When execv raises and the process keeps running, a foreign owner's pending deadline must be put back. _disarm_process_alarm_before_exec now returns the pre-cancel itimer value, and reexec_launcher / reexec_python_module restore it via _rearm_process_alarm_after_failed_exec before re-raising the OSError. Fixes #14358
Problem / Motivation
Goal: The gateway survives a laptop suspend of any length instead of dying on resume with a crash dump that was never a crash.
The gateway dies on every laptop resume. After a two-minute suspend,
kirocrew gatewayis gone and the crash-dump file shows the loop-stall watchdog'sTimeout (0:00:25)!with the main thread inside_default_cancel_later— alive and petting the timer at the very moment the timer killed it.Why it matters
Every Linux laptop user restarts the gateway by hand after every sleep, and every one of those restarts reports a "prior crash dump" that was never a crash. The same mechanism applies to every macOS build.
Not a goal
disarm_inherited_alarmimport, and re-arming a foreign owner's alarm whenexecvfails.exit=Truefor the probed gateway.What changed (motivation → approach → change)
setitimer(ITIMER_REAL)), re-armed by every heartbeat, with faulthandler'sSIGALRMhandler dumping every thread from C_exit(1)CLOCK_MONOTONIC; macOS: absolute mach timebase) → nothing fires; one INFO line names the sleep_exit(1)atexit_afterexit_after, then the process ends bySIGALRM— same file, same budgetRoot cause.
faulthandler.dump_traceback_laterwaits on an interpreter lock whose deadline clock is fixed when CPython is built:sem_clockwait(CLOCK_MONOTONIC)only when the build's libc offered it, otherwisesem_timedwaitonCLOCK_REALTIME. Portable builds (this fleet's 3.12.13 and the desktop bundle:HAVE_SEM_CLOCKWAITundefined) and every macOS build take the realtime branch. A suspend movesCLOCK_REALTIMEby its whole length whileCLOCK_MONOTONIC, the loop's clock, stands still, so the timer fires the instant the host wakes. No budget fixes a wall-clock deadline: it fires on any suspend longer than itself.The change (
src/kiro_crew/dashboard/loop_watchdog.py). Each heartbeat re-arms the kernel alarm forexit_afterseconds througharm_process_alarminsrc/kiro_crew/platform_compat.py(new), andfaulthandler.register(SIGALRM, chain=True)on the crash-dump file makes the alarm dump every thread's stack from inside the signal handler (C, no GIL) and then handSIGALRMto its default disposition, which ends the process. Linux runsITIMER_REALonCLOCK_MONOTONIC, so a suspend leaves the deadline where it was. One mechanism covers a syscall-blocked loop thread and a GIL-holding C call. Windows has no process alarm, and a process already handlingSIGALRMfrom Python (pytest-timeout, an embedding host) ownsITIMER_REALtoo; in both cases faulthandler's timer carries the exit at the same budget. The startup line names it:exit_after=25s (alarm). The daemon thread keeps enrichment (15 s) and the soft dump (30 s) on the monotonic clock, on its 5 s poll; each poll also readsplatform_compat.boottime_now(), and an advance of 2 s or more beyond the monotonic advance is logged once at INFO as a suspend (or forward clock step).server.py's heartbeat comment,docs/system-specs/modules/slack-gateway.mdanddocs/system-specs/common/platform-compat.mdcarry the contract.dashboard.loop_stall_exit_after_secskeeps its meaning; no new config key.Round 1: the exit mechanism is latched at arm time and cancelled by that latch (Design); the child runner decodes UTF-8 (Opus, CI); the threshold-precise daemon wake is deleted, the 5 s cadence comment restored (First Principles).
Round 2:
boottime_nowmoves fromsrc/kiro_crew/acp/liveness.py(now a re-export) intoplatform_compat.py, where the watchdog imports it: dashboard code reaches no ACP module (agent-SDK boundary gate); each alarm arm releases and renews faulthandler'sSIGALRMregistration, so a temporary owner that handed the signal back withSIG_DFLcannot cost the next alarm its dump (GPT).Round 3:
reexec_launcher/reexec_python_modulecancel the alarm beforeos.execv(execvekeepsITIMER_REALand resets a caughtSIGALRMtoSIG_DFL: the successor would die of a deadline it never armed), and thegatewayentrypoint (src/kiro_crew/cli.py) clears a deadline that arrived whileSIGALRMis stillSIG_DFL(disarm_inherited_alarm) (Opus).Round 4: the alarm and restart tests run on every platform through seams, no skips (GPT) — skips replaced by seams to clear the ratchet rule; whether capability skips are allowed at all is #14455's call.
Alternatives not taken. Re-arming the old timer from the heartbeat first (both wake on the same resume; the C thread needs no GIL and wins); a daemon-thread exit with the alarm as last resort (two deciders for one question); dropping
exit=Truefor the Electron/systemd probes (#13918: a foreground gateway has no probe).What a real stall looks like now, and who reads it
The process ends by
SIGALRMinstead of status 1, and the dump has noTimeout (preamble line. I found no consumer of either: the desktop supervisor logs code and signal generically and its stale-bundle check matches only its own exit code; systemdRestart=alwaysand launchdKeepAliverespawn on either;doctor, the startup replay and stall attribution treat the preamble as optional and prefer theCurrent threadmarker, which the signal handler puts on the main thread. The dump, the file and the next-boot replay are unchanged.Backwards compatibility
Compatible: no config key, CLI flag or API changes; a real stall still ends in the same crash-dump file at the same budget. The exit shape changes from status 1 to
SIGALRMand theTimeout (preamble line disappears; no consumer of either exists (see above). An in-app restart or update execs its successor with the alarm cancelled and the successor's entrypoint clears any deadline that still arrived; the restart is otherwise unchanged.Tests
Red on
main, green here (test/test_loop_watchdog.py,-n0):test_real_stall_ends_the_process_by_the_alarm: a child interpreter whose main thread blocks intime.sleepand never beats ends with-signal.SIGALRMand its dump carriesThread 0xand the module frame.test_real_gil_holding_stall_ends_the_process_by_the_alarm: the same withre.match(r"(a+)+$", "a" * 64 + "b")holding the GIL on the main thread.test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl(red on the round-1 head): a child installs a PythonSIGALRMhandler, restoresSIG_DFL, beats, and stalls; it still ends with-signal.SIGALRMand a dump carryingThread 0x— before the fix the file stayed header-only.test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarmpinsunregisterbeforeregisteron each arm.test_restart_exec_does_not_hand_the_successor_the_alarm(red on the round-2 head: the successor died-SIGALRM): a child arms the watchdog at 0.2 s and execs a successor throughreexec_launcher; the successor sleeps 1 s and exits 7 on its own.test/test_platform_compat_process_alarm.py::test_reexec_cancels_the_process_alarm_before_the_exec(both seams) andtest/test_managed_launcher_restart.py::test_the_restart_reaches_its_exec_with_the_stall_alarm_cancelled(the dashboard restart path end to end) read a stand-inITIMER_REALof(0.0, 0.0)at the stubbedos.execvafterarm_process_alarm(25.0).test_disarm_inherited_alarm_*pin the successor's half (clears atSIG_DFL, leaves a foreign owner's timer alone, no-op without a process alarm);test/test_cli.py::test_gateway_dispatch_clears_an_inherited_stall_alarm_before_the_gateway_startspins the entrypoint order.test_suspend_is_not_a_stall: suspend-inclusive clock +125 s, monotonic +5 s → exactly one INFO line containing"suspend"and"120",assert fx.dumps == [],assert fx.dump_file.written == [],assert (fx.arms, fx.cancels) == (arms_before, cancels_before)(the daemon touches no timer), and the nextbeat()re-arms25.0.test_overnight_suspend_is_still_not_a_stall,test_skew_below_the_floor_is_not_reported_as_a_suspend,test_suspend_clock_none_disables_the_comparison.test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarm/..._falls_back_to_faulthandler_timer_without_a_process_alarm/test_alarm_yields_to_a_foreign_sigalrm_owner/test_a_process_that_owns_sigalrm_keeps_its_own_deadline(the owner is constructed through the seam and its deadline read back off a stand-inITIMER_REAL): the arm/cancel wiring on every branch, never touching a deadline another handler owns;test_cancel_targets_the_mechanism_the_last_arm_used: a foreignSIGALRMhandler installed after an arm still has the next beat cancel the itimer that arm started (0.0), with faulthandler's timer untouched, and the re-arm lands on faulthandler's timer;test_armed_line_names_the_exit_mechanism. The alarm-branch tests stub the platform bindings (signal.setitimer/ITIMER_REAL/SIGALRM,faulthandler.register/unregister) through one_AlarmSeamhelper, so they run on every runner, Windows included; the child-process tests assert the platform's own exit (SIGALRMwhere the alarm exists, faulthandler's_exit(1)where it does not) and the restart test writes its stub launcher instead of copying a system binary. No test in this PR skips.test/test_platform_compat_process_alarm.py:arm_process_alarmarms and cancels throughsetitimer, reports unavailability without it.test/test_acp_liveness_darwin.pykeeps readingboottime_nowthroughlivenessand passes unchanged.start()arms[25.0], each beat re-arms,stop()cancels);test/test_crash_dump_store.py::test_watchdog_rearm_failure_restores_discoverable_soft_dumpandtest/test_loop_stall_surfacing.py::test_watchdog_arms_with_the_managed_service_budget([90]) are unchanged in contract.Local gate: the change-related backend set ran in full on an earlier head of this same content (67,838 passed; 110 reds and 9 setup errors all reproduce against
main's sources on this host or pass in isolation, none in the watchdog's vicinity); the touched suites re-ran green on the final head.Windows (1) red is #13215, contended-write flake.
Manual verification
Pending a real suspend/resume by a maintainer: CI cannot suspend a host and none was performed here (the child-process tests exercise the real alarm and signal handler end to end). Two-minute laptop check, Linux:
git fetch origin pull/14302/head && git checkout FETCH_HEAD.~/.kiro/crew/config.jsonset"dashboard": {"loop_stall_exit_after_secs": 25}, then runkirocrew -v gatewayin a terminal (-vshows INFO lines; the default level is WARNING).kirocrew statusanswers) and one INFO line in the terminal (or~/.kiro/crew/gateway.logfor a detached gateway) beginninghost was suspended (or its clock stepped forward) for about— its number is roughly the sleep length. Before this PR, on a portable or macOS interpreter build, the same sleep ended the process on resume.macOS: the same steps on a macOS build confirm the mach-timebase half, or a maintainer defers it; if that reading were wrong, macOS would keep today's behaviour, not gain a new failure.
Screenshots / video
Why no screenshot: backend-only change; no rendered surface is touched.
Related Issues
Fixes #14273
Related: #13918 (a real 25 s stall on a foreground gateway; the alarm ends it at the same budget)
Pattern harvest
Rule candidate: review-prompt
Pattern: a deadline armed on
CLOCK_REALTIME(anysem_timedwait/pthread_cond_timedwait-backed timeout, includingfaulthandler.dump_traceback_lateron portable and macOS builds) is fired by a host suspend; a liveness deadline must live on a clock that pauses with the host (CLOCK_MONOTONIC, the kernel'sITIMER_REAL), and a "counts the sleep" clock is read only to explain a gap.