Skip to content

fix(watchdog): decide the loop-stall exit on the monotonic clock - #14302

Merged
bolichen97 merged 1 commit into
mainfrom
fix/loop-watchdog-suspend-is-not-a-stall
Sep 29, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/loop-watchdog-suspend-is-not-a-stall

Conversation

@pepmach

@pepmach pepmach commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The gateway survives a laptop suspend of any length instead of dying on resume with a crash dump that was never a crash.

The gateway dies on every laptop resume. After a two-minute suspend, kirocrew gateway is gone and the crash-dump file shows the loop-stall watchdog's Timeout (0:00:25)! with the main thread inside _default_cancel_later — alive and petting the timer at the very moment the timer killed it.

Why it matters

Every Linux laptop user restarts the gateway by hand after every sleep, and every one of those restarts reports a "prior crash dump" that was never a crash. The same mechanism applies to every macOS build.

Not a goal

  • Proving the fix on a real suspend/resume here, on Linux or a macOS build: CI cannot suspend a host; the steps in Manual verification are for a maintainer to run or defer.
  • The follow-ups deferred to chore(platform): follow-ups deferred from #14302 (module-scope disarm import; re-arm a foreign alarm when execv fails) #14358: a module-scope disarm_inherited_alarm import, and re-arming a foreign owner's alarm when execv fails.
  • Changing what a real stall costs: the 25 s budget, the crash-dump file and its next-boot replay stay as they are.
  • A second exit decider beside the alarm (re-arming the old timer from the heartbeat, a daemon-thread exit) or dropping exit=True for the probed gateway.

What changed (motivation → approach → change)

Piece Before After
What ends a stalled gateway faulthandler's C timer, re-armed by every heartbeat, on a wall-clock deadline the kernel's per-process alarm (setitimer(ITIMER_REAL)), re-armed by every heartbeat, with faulthandler's SIGALRM handler dumping every thread from C
A laptop suspend wall clock jumps by the whole sleep → deadline already past on resume → dump and _exit(1) the alarm pauses with the host (Linux: CLOCK_MONOTONIC; macOS: absolute mach timebase) → nothing fires; one INFO line names the sleep
A real stall dump + _exit(1) at exit_after dump at exit_after, then the process ends by SIGALRM — same file, same budget
A thread holding the GIL the same C timer the same alarm: a signal handler needs no GIL

Root cause. faulthandler.dump_traceback_later waits on an interpreter lock whose deadline clock is fixed when CPython is built: sem_clockwait(CLOCK_MONOTONIC) only when the build's libc offered it, otherwise sem_timedwait on CLOCK_REALTIME. Portable builds (this fleet's 3.12.13 and the desktop bundle: HAVE_SEM_CLOCKWAIT undefined) and every macOS build take the realtime branch. A suspend moves CLOCK_REALTIME by its whole length while CLOCK_MONOTONIC, the loop's clock, stands still, so the timer fires the instant the host wakes. No budget fixes a wall-clock deadline: it fires on any suspend longer than itself.

The change (src/kiro_crew/dashboard/loop_watchdog.py). Each heartbeat re-arms the kernel alarm for exit_after seconds through arm_process_alarm in src/kiro_crew/platform_compat.py (new), and faulthandler.register(SIGALRM, chain=True) on the crash-dump file makes the alarm dump every thread's stack from inside the signal handler (C, no GIL) and then hand SIGALRM to its default disposition, which ends the process. Linux runs ITIMER_REAL on CLOCK_MONOTONIC, so a suspend leaves the deadline where it was. One mechanism covers a syscall-blocked loop thread and a GIL-holding C call. Windows has no process alarm, and a process already handling SIGALRM from Python (pytest-timeout, an embedding host) owns ITIMER_REAL too; in both cases faulthandler's timer carries the exit at the same budget. The startup line names it: exit_after=25s (alarm). The daemon thread keeps enrichment (15 s) and the soft dump (30 s) on the monotonic clock, on its 5 s poll; each poll also reads platform_compat.boottime_now(), and an advance of 2 s or more beyond the monotonic advance is logged once at INFO as a suspend (or forward clock step). server.py's heartbeat comment, docs/system-specs/modules/slack-gateway.md and docs/system-specs/common/platform-compat.md carry the contract. dashboard.loop_stall_exit_after_secs keeps its meaning; no new config key.

Round 1: the exit mechanism is latched at arm time and cancelled by that latch (Design); the child runner decodes UTF-8 (Opus, CI); the threshold-precise daemon wake is deleted, the 5 s cadence comment restored (First Principles).

Round 2: boottime_now moves from src/kiro_crew/acp/liveness.py (now a re-export) into platform_compat.py, where the watchdog imports it: dashboard code reaches no ACP module (agent-SDK boundary gate); each alarm arm releases and renews faulthandler's SIGALRM registration, so a temporary owner that handed the signal back with SIG_DFL cannot cost the next alarm its dump (GPT).

Round 3: reexec_launcher / reexec_python_module cancel the alarm before os.execv (execve keeps ITIMER_REAL and resets a caught SIGALRM to SIG_DFL: the successor would die of a deadline it never armed), and the gateway entrypoint (src/kiro_crew/cli.py) clears a deadline that arrived while SIGALRM is still SIG_DFL (disarm_inherited_alarm) (Opus).

Round 4: the alarm and restart tests run on every platform through seams, no skips (GPT) — skips replaced by seams to clear the ratchet rule; whether capability skips are allowed at all is #14455's call.

Alternatives not taken. Re-arming the old timer from the heartbeat first (both wake on the same resume; the C thread needs no GIL and wins); a daemon-thread exit with the alarm as last resort (two deciders for one question); dropping exit=True for the Electron/systemd probes (#13918: a foreground gateway has no probe).

What a real stall looks like now, and who reads it

The process ends by SIGALRM instead of status 1, and the dump has no Timeout ( preamble line. I found no consumer of either: the desktop supervisor logs code and signal generically and its stale-bundle check matches only its own exit code; systemd Restart=always and launchd KeepAlive respawn on either; doctor, the startup replay and stall attribution treat the preamble as optional and prefer the Current thread marker, which the signal handler puts on the main thread. The dump, the file and the next-boot replay are unchanged.

Backwards compatibility

Compatible: no config key, CLI flag or API changes; a real stall still ends in the same crash-dump file at the same budget. The exit shape changes from status 1 to SIGALRM and the Timeout ( preamble line disappears; no consumer of either exists (see above). An in-app restart or update execs its successor with the alarm cancelled and the successor's entrypoint clears any deadline that still arrived; the restart is otherwise unchanged.

Tests

Red on main, green here (test/test_loop_watchdog.py, -n0):

  • test_real_stall_ends_the_process_by_the_alarm: a child interpreter whose main thread blocks in time.sleep and never beats ends with -signal.SIGALRM and its dump carries Thread 0x and the module frame.
  • test_real_gil_holding_stall_ends_the_process_by_the_alarm: the same with re.match(r"(a+)+$", "a" * 64 + "b") holding the GIL on the main thread.
  • test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl (red on the round-1 head): a child installs a Python SIGALRM handler, restores SIG_DFL, beats, and stalls; it still ends with -signal.SIGALRM and a dump carrying Thread 0x — before the fix the file stayed header-only. test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarm pins unregister before register on each arm.
  • test_restart_exec_does_not_hand_the_successor_the_alarm (red on the round-2 head: the successor died -SIGALRM): a child arms the watchdog at 0.2 s and execs a successor through reexec_launcher; the successor sleeps 1 s and exits 7 on its own. test/test_platform_compat_process_alarm.py::test_reexec_cancels_the_process_alarm_before_the_exec (both seams) and test/test_managed_launcher_restart.py::test_the_restart_reaches_its_exec_with_the_stall_alarm_cancelled (the dashboard restart path end to end) read a stand-in ITIMER_REAL of (0.0, 0.0) at the stubbed os.execv after arm_process_alarm(25.0). test_disarm_inherited_alarm_* pin the successor's half (clears at SIG_DFL, leaves a foreign owner's timer alone, no-op without a process alarm); test/test_cli.py::test_gateway_dispatch_clears_an_inherited_stall_alarm_before_the_gateway_starts pins the entrypoint order.
  • test_suspend_is_not_a_stall: suspend-inclusive clock +125 s, monotonic +5 s → exactly one INFO line containing "suspend" and "120", assert fx.dumps == [], assert fx.dump_file.written == [], assert (fx.arms, fx.cancels) == (arms_before, cancels_before) (the daemon touches no timer), and the next beat() re-arms 25.0.
  • test_overnight_suspend_is_still_not_a_stall, test_skew_below_the_floor_is_not_reported_as_a_suspend, test_suspend_clock_none_disables_the_comparison.
  • test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarm / ..._falls_back_to_faulthandler_timer_without_a_process_alarm / test_alarm_yields_to_a_foreign_sigalrm_owner / test_a_process_that_owns_sigalrm_keeps_its_own_deadline (the owner is constructed through the seam and its deadline read back off a stand-in ITIMER_REAL): the arm/cancel wiring on every branch, never touching a deadline another handler owns; test_cancel_targets_the_mechanism_the_last_arm_used: a foreign SIGALRM handler installed after an arm still has the next beat cancel the itimer that arm started (0.0), with faulthandler's timer untouched, and the re-arm lands on faulthandler's timer; test_armed_line_names_the_exit_mechanism. The alarm-branch tests stub the platform bindings (signal.setitimer / ITIMER_REAL / SIGALRM, faulthandler.register / unregister) through one _AlarmSeam helper, so they run on every runner, Windows included; the child-process tests assert the platform's own exit (SIGALRM where the alarm exists, faulthandler's _exit(1) where it does not) and the restart test writes its stub launcher instead of copying a system binary. No test in this PR skips.
  • test/test_platform_compat_process_alarm.py: arm_process_alarm arms and cancels through setitimer, reports unavailability without it. test/test_acp_liveness_darwin.py keeps reading boottime_now through liveness and passes unchanged.
  • Existing armed-timer, soft-dump and enrichment tests keep their assertions (start() arms [25.0], each beat re-arms, stop() cancels); test/test_crash_dump_store.py::test_watchdog_rearm_failure_restores_discoverable_soft_dump and test/test_loop_stall_surfacing.py::test_watchdog_arms_with_the_managed_service_budget ([90]) are unchanged in contract.

Local gate: the change-related backend set ran in full on an earlier head of this same content (67,838 passed; 110 reds and 9 setup errors all reproduce against main's sources on this host or pass in isolation, none in the watchdog's vicinity); the touched suites re-ran green on the final head.

Windows (1) red is #13215, contended-write flake.

Manual verification

Pending a real suspend/resume by a maintainer: CI cannot suspend a host and none was performed here (the child-process tests exercise the real alarm and signal handler end to end). Two-minute laptop check, Linux:

  1. Check out this branch: git fetch origin pull/14302/head && git checkout FETCH_HEAD.
  2. Start a dev gateway with a short exit budget: in ~/.kiro/crew/config.json set "dashboard": {"loop_stall_exit_after_secs": 25}, then run kirocrew -v gateway in a terminal (-v shows INFO lines; the default level is WARNING).
  3. Suspend the laptop for more than 25 s.
  4. Resume.
  5. Expect the gateway process still alive (kirocrew status answers) and one INFO line in the terminal (or ~/.kiro/crew/gateway.log for a detached gateway) beginning host was suspended (or its clock stepped forward) for about — its number is roughly the sleep length. Before this PR, on a portable or macOS interpreter build, the same sleep ended the process on resume.

macOS: the same steps on a macOS build confirm the mach-timebase half, or a maintainer defers it; if that reading were wrong, macOS would keep today's behaviour, not gain a new failure.

Screenshots / video

Why no screenshot: backend-only change; no rendered surface is touched.

Related Issues

Fixes #14273

Related: #13918 (a real 25 s stall on a foreground gateway; the alarm ends it at the same budget)

Pattern harvest

Rule candidate: review-prompt
Pattern: a deadline armed on CLOCK_REALTIME (any sem_timedwait/pthread_cond_timedwait-backed timeout, including faulthandler.dump_traceback_later on portable and macOS builds) is fired by a host suspend; a liveness deadline must live on a clock that pauses with the host (CLOCK_MONOTONIC, the kernel's ITIMER_REAL), and a "counts the sleep" clock is read only to explain a gap.

@pepmach
pepmach requested review from a team and CrysisDeu September 26, 2026 22:42
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 26, 2026
@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 7c857988ce1eff4f8df1c298157fc70c33d6c6e6 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7c85798

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7c857988ce1eff4f8df1c298157fc70c33d6c6e6: <one-sentence reason>

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 7c857988ce1eff4f8df1c298157fc70c33d6c6e6 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Run the described two-minute Linux suspend/resume check before merge — the suspend-pausing alarm semantic is CI-untestable and only the maintainer recipe exercises it.

[DESIGN-REVIEWED] 7c85798

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 7c857988ce1eff4f8df1c298157fc70c33d6c6e6 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify the macOS premise on real hardware: that ITIMER_REAL pauses across sleep ("absolute mach timebase") is the one claim CI never exercises, and the description itself defers it to a maintainer.

Not justified as shipped

  • Item 6 — rides along: the resume INFO line (_note_resume, SUSPEND_SKEW_MIN_SECS, suspend_clock seam) is observability, not the fix; declared, and it is the instrument the deferred manual verification reads.
  • Item 7 — rides along: the startup line renaming to exit_after=25s (alarm) is a relabel the fix does not need; declared, trivial surface.

What this change ships

Inventory (10 items) — 8 justified

Intent: the gateway survives a laptop suspend instead of dying on resume with a false crash dump — a FIX.

  1. Stall exit decided by the kernel alarm (suspend-paused) instead of faulthandler's wall-clock timer — justified
  2. A real stall now ends by SIGALRM with no Timeout ( preamble, not _exit(1) — justified (I verified crash_dump_store._split_stack_content treats the preamble as optional and _wedged_thread_block prefers the Current thread marker the signal handler adds; grep Timeout \( in src found no other consumer)
  3. Windows and foreign-SIGALRM-owner processes keep the old timer at the same budget — justified
  4. Restart/update exec seams cancel the alarm; the successor entrypoint clears an inherited deadline — justified
  5. Each arm re-registers faulthandler's SIGALRM handler; cancel targets the latched mechanism — justified
  6. One INFO line names a resume (boot-clock vs monotonic skew ≥ 2 s) — rides along (declared observability)
  7. Startup line names the exit mechanism — rides along (declared relabel)
  8. boottime_now moved to platform_compat with a liveness re-export — justified (agent-SDK boundary gate; 2 counted importers, subagent.py:31, acp/session_handle.py:95, keep working)
  9. New arm_process_alarm / process_alarm_available helpers — justified (consumers counted: watchdog arm/cancel, both exec seams, entrypoint disarm)
  10. Specs and comments (platform-compat.md, slack-gateway.md, server.py) updated in the same commit — justified

Root-cause note: the fix sits at CAUSE level (the wall-clock deadline itself); grep dump_traceback_later in src shows the single arming site, now the deliberate fallback — no unfixed siblings.

[FIRST-PRINCIPLES-REVIEWED] 7c85798

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 26, 2026
@pepmach

pepmach commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author
  • evidence-limit — span=9feafbec0590 — the macOS alarm-during-sleep reading is source-build evidence; a Mac verifies it at review

Source-build evidence: the alarm exit is proven by the child-process tests on Linux CI (test/test_loop_watchdog.py::test_real_stall_ends_the_process_by_the_alarm, ::test_real_gil_holding_stall_ends_the_process_by_the_alarm, ending with -SIGALRM and a dump); the macOS half rests on XNU arming setitimer on the absolute mach timebase, which does not advance during sleep. No Mac is available here, so that reading is not exercised.
The failure direction is benign: if the reading is wrong, macOS keeps today's behaviour (a wall-clock deadline fired on resume) and gains no new failure; Linux, where the reporter's suspend kill occurs, is proven.
Declared in the body's ## Manual verification section as its Not in this PR item, in these words. A maintainer with a Mac clears it at review by running test/test_platform_compat_process_alarm.py and the two child-process tests above there, then sleeping the host longer than exit_after with an armed gateway and confirming it stays alive and logs the resume line.

@github-actions

github-actions Bot commented Sep 26, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 7c857988ce1eff4f8df1c298157fc70c33d6c6e6 — this comment is updated in place on each push.

Review details

Advisory only: the exec seams cancel ITIMER_REAL without the ownership check the rest of the PR applies.

FINDING — src/kiro_crew/platform_compat.py:87 — _disarm_process_alarm_before_exec's bare arm_process_alarm(0.0) zeroes the real ITIMER_REAL for whoever owns it, and the process only survives the seam when os.execv is mocked: test_managed_launcher_restart.py:119, test_platform_compat.py:167,189 and test_wheel_engine.py:927,940 patch os.execv alone, so on POSIX the call cancels pytest-timeout's signal-method deadline (setup.cfg --timeout=120, no timeout_method, the SIGALRM timer conftest.py:1791 and test_cli_lazy_imports.py:153 both build on) for the remainder of that item, leaving any later hang in it unbounded — while the PR's own new tests stub setitimer precisely to avoid this and its spec states the alarm "is never armed or cancelled" when another owner holds SIGALRM → Fix: apply the same rule inside _disarm_process_alarm_before_exec — if process_alarm_available() and signal.getsignal(signal.SIGALRM) == signal.SIG_DFL: arm_process_alarm(0.0), which still cancels the gateway's own deadline because faulthandler.register leaves getsignal(SIGALRM) at SIG_DFL.

[OPUS-REVIEWED] 7c85798

Verdict parsed from the review's SHA-scoped output markers for commit 7c857988ce1eff4f8df1c298157fc70c33d6c6e6.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7c857988ce1eff4f8df1c298157fc70c33d6c6e6: <one-sentence reason>

@pepmach
pepmach force-pushed the fix/loop-watchdog-suspend-is-not-a-stall branch from a9922e2 to 1ac54c0 Compare September 26, 2026 23:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 26, 2026
@pepmach
pepmach force-pushed the fix/loop-watchdog-suspend-is-not-a-stall branch from 1ac54c0 to aed0ddb Compare September 26, 2026 23:55
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 26, 2026
@pepmach

pepmach commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author
  • fixed — span=344211ed2d47 — the exit mechanism is latched at arm time and the cancel targets that latch

Fixed in aed0ddb64cffded8acdf0d7c162800cd7dd990b0. _default_arm_later asks exit_mechanism() once, arms that mechanism, and records it in the module-level _armed_mechanism; _default_cancel_later cancels exactly the latched one (setitimer(ITIMER_REAL, 0) or cancel_dump_traceback_later()) and clears the latch only once the cancel has happened. Nothing latched cancels nothing, so no derived guess ever reaches a timer another owner holds.
The race is pinned red-first by test/test_loop_watchdog.py::test_cancel_targets_the_mechanism_the_last_arm_used: armed under the alarm, a foreign SIGALRM handler installed, then a beat — the itimer is cancelled (armed == [25.0, 0.0]), cancel_dump_traceback_later is not called, and the re-arm lands on faulthandler's timer, so exactly one exit is pending on the mechanism the foreign owner leaves free. Red on the judged head: the beat left the itimer pending ([0.0, 25.0], no cancel) and armed faulthandler's timer beside it.
The owning spec sentence in docs/system-specs/modules/slack-gateway.md names the latch.

@pepmach

pepmach commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author
  • fixed — span=a706dff03bba — the tests that force the alarm branch are POSIX-only; the no-alarm branch test still runs on Windows

Fixed in aed0ddb64cffded8acdf0d7c162800cd7dd990b0. _NEEDS_ALARM (skipif(not hasattr(signal, "setitimer"))) is defined above the alarm-wiring section and decorates every test that patches process_alarm_available to True and so reaches signal.SIGALRM (test_armed_line_names_the_exit_mechanism, test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarm, test_alarm_yields_to_a_foreign_sigalrm_owner, test_alarm_is_used_while_sigalrm_is_at_its_default, test_a_test_worker_that_owns_sigalrm_keeps_its_own_deadline, and the new test_cancel_targets_the_mechanism_the_last_arm_used).
Windows shard 8 on the judged head names a fifth failure the finding does not: test_default_arm_falls_back_to_faulthandler_timer_without_a_process_alarm fails at its faulthandler.register guard, because that attribute does not exist on Windows either. It is the Windows branch, so it keeps running there with raising=False on the guard rather than a skip. The Backend Lint & Type Check job on the same head flags _run_child's text=True (subprocess-encoding gate); it now decodes with **UTF8_TEXT.

@pepmach

pepmach commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author
  • evidence-limit — span=b21f80e98982 — item 1: the suspend-during-alarm claim is source-build evidence; a laptop confirms it at review

Source-build evidence: the alarm exit is proven by the child-process tests on Linux CI (test/test_loop_watchdog.py::test_real_stall_ends_the_process_by_the_alarm, ::test_real_gil_holding_stall_ends_the_process_by_the_alarm); the suspend half rests on Linux arming ITIMER_REAL on CLOCK_MONOTONIC (copy_signal, kernel/fork.c) and on XNU arming setitimer on the absolute mach timebase. No real suspend/resume is performed here, on either platform.
The failure direction on macOS is benign: if the mach-timebase reading is wrong, macOS keeps today's behaviour (a deadline fired on resume) and gains no new failure. On Linux the clock semantics are documented (clock_gettime(2)), and the reporter's kill is the wall-clock timer this PR retires.
The body's ## Manual verification section now carries both as its Not in this PR item: one real suspend/resume on a laptop — Linux, and a macOS build — longer than exit_after, confirming the armed gateway stays alive and logs the resume line. A maintainer with a laptop may confirm it at review; that is what clears this item.

@pepmach

pepmach commented Sep 26, 2026

Copy link
Copy Markdown
Contributor Author
  • fixed — span=75070d13eb7f — item 6, subtraction taken: the threshold-precise daemon wake is deleted and the cadence comment restored

Deleted in aed0ddb64cffded8acdf0d7c162800cd7dd990b0: _next_wait, _MIN_POLL_WAIT_SECS, the two test_daemon_wait_* tests and every sentence describing the wake (module docstring, poll_interval, the owning spec sentence, the body). _run waits poll_interval again, as on main, and the enrich_after docstring carries main's cadence sentence again: with the 5s poll cadence, 15s triggers on the 15–20s tick, ahead of the 25s exit.
No shipped budget is missed by the poll wait: enrichment lands at 15–20 s against the 25 s desktop exit, the soft dump at 30–35 s against the 90 s managed exit. The only budgets the cadence can miss are an explicit dashboard.loop_stall_exit_after_secs of 16–19 s (enrichment) or 31–34 s (soft dump); no default selects them, the field's own guidance points operators upward, and below 20 s the desktop probe ends the process first. Not worth a mechanism, so the wake goes rather than a comment naming that band.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@pepmach
pepmach force-pushed the fix/loop-watchdog-suspend-is-not-a-stall branch from aed0ddb to 6e90657 Compare September 27, 2026 01:42
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 27, 2026
@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • fixed — span=6112b3ae01c8 — each alarm arm releases and renews faulthandler's SIGALRM registration

Fixed in 6e9065755260309c5d8a8f1fb2bf5d78af63f5d4: _default_arm_later calls faulthandler.unregister(signal.SIGALRM) before faulthandler.register(signal.SIGALRM, file=target, all_threads=True, chain=True) on every alarm arm. register installs its C handler only while it holds no registration for the signal, so after a temporary Python owner restored SIG_DFL a bare re-register left the disposition at SIG_DFL and the next alarm ended the process with no dump.
Red-first in a child process: test/test_loop_watchdog.py::test_alarm_dump_survives_a_temporary_sigalrm_owner_that_restores_sig_dfl installs a Python SIGALRM handler, restores SIG_DFL, beats, and stalls — on the judged head the process died by SIGALRM with a header-only dump file; on the fix it ends with -SIGALRM and a dump carrying Thread 0x. test_default_arm_registers_faulthandler_on_sigalrm_and_starts_the_alarm pins the unregister → register order; the faulthandler-branch tests pin that neither is touched when a foreign owner holds the signal.
Every caller cancels the pending deadline before it arms (beat, start), so no alarm of ours can land in the unregister/register gap. Spec sentence added to docs/system-specs/modules/slack-gateway.md and the alarm row of docs/system-specs/common/platform-compat.md.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • evidence-limit — span=faa571c234c1 — the kernel-claim premise is source-build evidence; a laptop confirms it at review

Source-build evidence: the alarm exit is proven by the child-process tests on Linux CI (test/test_loop_watchdog.py::test_real_stall_ends_the_process_by_the_alarm, ::test_real_gil_holding_stall_ends_the_process_by_the_alarm, ending with -SIGALRM and a dump); the suspend half rests on Linux arming ITIMER_REAL on CLOCK_MONOTONIC (copy_signal, kernel/fork.c) and on XNU arming setitimer on the absolute mach timebase. No real suspend/resume was performed here, on either platform, and no Mac was available.
The failure direction on macOS is benign: if the mach-timebase reading is wrong, macOS keeps today's behaviour (a deadline fired on resume) and gains no new failure. On Linux the clock semantics are documented (clock_gettime(2)), and the reporter's kill is the wall-clock timer this PR retires.
Declared in the body's ## Manual verification section as its Not in this PR item: one real suspend/resume on a laptop — Linux, and a macOS build — longer than exit_after, confirming the armed gateway stays alive and logs the resume line. A maintainer with a laptop may confirm it at review; that is what clears this item, as it does the same-premise items on the prior head (design 9feafbec0590, first-principles b21f80e98982).

@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • declared-and-kept — span=02dadbc66fc5 — item 3: the resume INFO line is a declared observability rider that decides nothing

Kept as shipped: _note_resume compares the suspend-inclusive clock's advance against the monotonic clock's across one poll and logs one INFO line when the host slept; it arms, cancels and dumps nothing (test_suspend_is_not_a_stall asserts (fx.arms, fx.cancels) unchanged and fx.dumps == [] across the resume). It exists so a resume that once killed the gateway is visible in the log as a resume, which is the one artifact a maintainer has when a laptop user reports the fix.
Declared as such in the body's What changed section ("an advance of 2 s or more beyond the monotonic advance is logged once at INFO as a suspend (or forward clock step)") and in the owning spec sentence ("it decides no exit"). The reader it reuses is now platform_compat.boottime_now (6e9065755260309c5d8a8f1fb2bf5d78af63f5d4): one definition, moved out of acp/liveness.py so dashboard code imports no ACP module.

@pepmach
pepmach force-pushed the fix/loop-watchdog-suspend-is-not-a-stall branch from c2cb179 to 3e253d3 Compare September 27, 2026 10:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 27, 2026
The gateway died on every laptop resume. LoopStallWatchdog re-armed
faulthandler's dump_traceback_later(exit=True) timer on each 5 s
heartbeat, and that timer waits on an interpreter lock whose deadline is
an absolute CLOCK_REALTIME instant on any CPython built without
sem_clockwait (portable builds link an old glibc; every macOS build). A
suspend advances CLOCK_REALTIME by its whole length while
CLOCK_MONOTONIC stands still, so on resume the 25 s deadline was already
past and faulthandler dumped and _exit(1)'d before the loop thread,
which had missed no beat on the monotonic clock, could re-arm it. No
budget fixes a wall-clock deadline: it fires on any suspend longer than
itself, and four times the 25 s desktop budget is still shorter than
the reporter's own two-minute suspend.

The dump-then-exit now belongs to the kernel's per-process alarm. Each
beat re-arms setitimer(ITIMER_REAL) for exit_after seconds
(platform_compat.arm_process_alarm) and faulthandler.register(SIGALRM,
chain=True) on the crash-dump file makes the alarm dump every thread's
stack from inside the signal handler, in C with no GIL, then hand
SIGALRM to its default disposition, which ends the process. Linux runs
ITIMER_REAL on CLOCK_MONOTONIC (copy_signal initialises the process's
real_timer on that clock) and macOS schedules it on the absolute mach
timebase, so the alarm pauses while the host sleeps. It fires whether
the loop thread is blocked in a syscall or holding the GIL inside a
long C call. Windows has no process alarm, and a process that already
handles SIGALRM from Python (pytest-timeout in a test worker, an
embedding host) owns ITIMER_REAL too; in both cases faulthandler's
timer carries the exit at the same budget and the alarm is never armed
or cancelled. On Windows nothing new is lost: its time.monotonic()
counts a sleep as well.

What changes about a real stall: the process ends by SIGALRM instead of
status 1 and the dump carries no "Timeout (" preamble line. No consumer
of either exists (the desktop supervisor logs code and signal
generically and its stale-bundle check matches only its own code;
systemd Restart=always and launchd KeepAlive respawn on either; every
dump reader treats the preamble as optional). The dump, the file and
the next-boot replay are unchanged.

The daemon thread keeps the observability layer on the monotonic clock:
enrichment, then the soft dump, on its 5 s poll (15 s lands on the
15-20 s tick, ahead of the 25 s exit), and a resume is logged once
at INFO when platform_compat.boottime_now (CLOCK_BOOTTIME on Linux, the
wall clock on macOS; moved from acp/liveness, which re-exports it, so
dashboard code imports no ACP module) advances 2 s or more beyond the
monotonic clock across one poll. Each alarm arm releases and renews
faulthandler's SIGALRM registration: a repeat register reinstalls
nothing while faulthandler believes it still holds the signal, so a
temporary owner that handed SIGALRM back with SIG_DFL would otherwise
leave the next alarm to end the process without a dump.

The alarm belongs to the image that armed it. execve preserves
ITIMER_REAL while it resets a caught SIGALRM to its default disposition,
so an in-app restart or update would hand the successor gateway the last
heartbeat's deadline as a lethal signal it never armed, during its own
boot. Both exec seams (platform_compat.reexec_launcher and
reexec_python_module) cancel the alarm immediately before os.execv, and
the gateway entrypoint clears any deadline that still arrived
(loop_watchdog.disarm_inherited_alarm), only while SIGALRM is at its
default disposition -- the ownership rule exit_mechanism applies.

Alternatives not taken: re-arming the old timer from the heartbeat
before faulthandler's thread runs (both wake on the same resume; the C
thread needs no GIL and wins); a daemon-thread exit decision with the
alarm as a wider last resort (two deciders for one question; the alarm
alone answers it); dropping exit=True for the soft dump plus the
Electron/systemd probes (a foreground `kirocrew gateway` has no probe;
a real stall would wedge it for good).

dashboard.loop_stall_exit_after_secs keeps its meaning; no config key
is added.

Fixes #14273
Related: #13918 (a real 25 s stall on a foreground gateway, a stat walk
on the loop thread; the alarm ends that stall at the same budget)
@pepmach
pepmach force-pushed the fix/loop-watchdog-suspend-is-not-a-stall branch from 3e253d3 to 7c85798 Compare September 27, 2026 11:42
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • fixed — span=148fdb64bde7 — newly added skip paths loosen the test ratchet

Fixed in 7c857988ce1eff4f8df1c298157fc70c33d6c6e6: skips replaced by seams to clear the ratchet rule; whether capability skips are allowed at all is #14455's call. No pytest.mark.skipif and no pytest.skip( remains in this PR's tests.
_NEEDS_ALARM is gone: an _AlarmSeam helper in test/test_loop_watchdog.py stubs the platform bindings the alarm branch reaches — a stand-in ITIMER_REAL (signal.setitimer / getitimer), a SIGALRM number where the platform has none, signal.getsignal answering the owner the test sets, recording faulthandler.register / unregister — all with raising=False, so the real process_alarm_available and arm_process_alarm take the alarm branch on every runner against the fake timer. test_a_process_that_owns_sigalrm_keeps_its_own_deadline constructs the foreign owner through the seam and reads its 7.0 s deadline back unchanged after an arm/cancel cycle and disarm_inherited_alarm(), instead of skipping when the runner's timeout method left SIGALRM at SIG_DFL. The child-process tests run everywhere and assert the platform's own dump-then-exit (-SIGALRM where the alarm exists, faulthandler's _exit(1) where it does not); the SIG_DFL-restore child installs its owner only where SIGALRM exists; the exec child asserts the successor's exit status only where the alarm exists and the exec's effects everywhere.
test/test_managed_launcher_restart.py::test_the_restart_reaches_its_exec_with_the_stall_alarm_cancelled writes its own stub launcher (kirocrew, kirocrew.exe on Windows) instead of copying /bin/true: os.execv is stubbed, and the resolver checks only for an absolute executable file.
Evidence on the fix head: test/test_loop_watchdog.py 36 passed, test/test_managed_launcher_restart.py 21 passed, test/test_platform_compat_process_alarm.py 4 passed (-n0, one file at a time); the 32 in-process watchdog tests also pass under a Windows-surface simulation that deletes setitimer / getitimer / ITIMER_REAL / SIGALRM / faulthandler.register / unregister before collection, and the launcher test passes with IS_WINDOWS forced.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • declared-and-kept — span=7c85bb2b9b2b — item 6: the resume INFO line is declared diagnostics beside the fix

Kept as shipped: _note_resume compares the suspend-inclusive clock's advance with the monotonic clock's across one poll and logs one INFO line when the host slept; it arms, cancels and dumps nothing (test_suspend_is_not_a_stall pins (fx.arms, fx.cancels) unchanged and fx.dumps == [] across a resume). It replaces the false artifact a suspend left before this change (a stall dump and an exit) with one true line, and it is the line the body's Manual verification asks a maintainer to look for after a real suspend — what makes the fix observable outside CI.
Declared in the body's What changed section ("logged once at INFO as a suspend (or forward clock step)") and in the owning spec sentence ("it decides no exit"); the reader it uses is platform_compat.boottime_now.

@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author
  • declared-and-kept — span=7050c635a55d — item 7: the (alarm|faulthandler) label on the armed line is declared diagnostics

Kept as shipped: the armed line reads exit_after=25s (alarm) or (faulthandler), naming which of the two mechanisms this process holds its exit on. The fallback is new in this PR (Windows, and a process whose SIGALRM a Python handler already owns), so the one line an operator reads at boot says which decider is live; without it a faulthandler-carried exit on a Linux host would be indistinguishable from the alarm in the log. No consumer parses the line.
Declared in the body's What changed section ("The startup line names it: exit_after=25s (alarm)") and the owning spec sentence ("the startup line says exit_after=<budget> (alarm|faulthandler)"); pinned by test_armed_line_names_the_exit_mechanism.

@pepmach

pepmach commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

Not a defect as filed: the proposed gate (getsignal(SIGALRM) == SIG_DFL) is the successor's ownership rule, and it is wrong for the image about to exec. execve preserves interval timers and resets every caught signal to SIG_DFL, so a pending ITIMER_REAL crosses the exec while its owner's handler does not — for pytest-timeout's timer as much as for the watchdog's — and the successor image dies of a default-action SIGALRM it never armed (test_restart_exec_does_not_hand_the_successor_the_alarm shows exactly that death, -SIGALRM, on the head before the seam cancel). Gating the cancel on a Python handler being installed would keep that kill for precisely the owners the gate is meant to protect; a timer nobody can catch in the successor is not one to preserve.
The residue is real and small: an exec that FAILS returns to the caller (OSError), and the process then runs on with a foreign owner's alarm gone. The in-process seam tests that stub only os.execv (test/test_platform_compat.py, test/test_wheel_engine.py, test/test_managed_launcher_restart.py) sit on that path, so their remaining in-process assertions run without pytest-timeout's per-item deadline; no test hangs or fails on it. Fix tracked in #14358 (label deferred-finding, assignee pepmach, Due: 2026-10-09): save signal.getitimer(ITIMER_REAL) before the cancel and re-arm it when os.execv raises, red-first with a stubbed exec that raises.
This head is frozen at readiness, so the residue is not pushed as another round; the successful-exec behaviour it would change is the one the PR's tests prove.

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed on behalf of the maintainers (sweep 2026-09-28): readiness passed, required checks green on 7c85798, no blocking review threads, security posture unchanged. Approving and merging.

@bolichen97
bolichen97 merged commit 1d37ab1 into main Sep 29, 2026
104 of 110 checks passed
@bolichen97
bolichen97 deleted the fix/loop-watchdog-suspend-is-not-a-stall branch September 29, 2026 00:00
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 29, 2026
bolichen97 added a commit that referenced this pull request Oct 1, 2026
…15634)

Backport of #13308, #14302, #14860, #14961, #15070 to release/0.8.0, cherry-picked in main's merge order.

* fix(sel): anchor the chain lock on the log directory (#13308)
  (cherry picked from commit 998b045)
* fix(watchdog): decide the loop-stall exit on the monotonic clock (#14302)
  (cherry picked from commit 1d37ab1)
* fix(ops-mission-control): strict read in ledger resolve_conflict (#14860)
  (cherry picked from commit b8ea16b)
* fix(env): skip a mise candidate whose probe is denied instead of aborting boot (#14961)
  (cherry picked from commit e054c95)
* fix(lessons): retire only a finding on a contradiction verdict (#15070)
  (cherry picked from commit dd5e840)
KiroCrewCaptain pushed a commit that referenced this pull request Nov 1, 2026
…hen execv fails (#22651)

Two follow-ups deferred from #14302.

1. cli.py imports disarm_inherited_alarm at module scope. The watchdog's
   import closure (dashboard.stall_enrichment -> acp.liveness, and
   platform_compat) contains no module-scope import of cli, so this creates
   no cycle, and acp is already loaded when cli imports session, so only the
   two small watchdog modules are new. The gateway dispatch keeps its order
   (faulthandler.enable() before disarm_inherited_alarm()).

2. The two exec seams re-arm a pre-existing ITIMER_REAL alarm when os.execv
   fails. Cancelling the alarm before execv is correct only when the exec
   succeeds: a successor image never armed it. When execv raises and the
   process keeps running, a foreign owner's pending deadline must be put back.
   _disarm_process_alarm_before_exec now returns the pre-cancel itimer value,
   and reexec_launcher / reexec_python_module restore it via
   _rearm_process_alarm_after_failed_exec before re-raising the OSError.

Fixes #14358
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] crash after suspend/resume

2 participants