From ec9414e75b19bc781e94540b4ba74e7456900309 Mon Sep 17 00:00:00 2001 From: Tribhuwan Kandpal Date: Sun, 18 Oct 2026 01:52:42 -0700 Subject: [PATCH 1/2] feat(watchdog): add macOS reader for the loop-stall paging deferral under memory pressure, reading per-thread major faults from Linux /proc. That reader returns None off Linux, so macOS never defers. Add a macOS reader behind the same loop_major_faults hook, so the deferral logic does not change. It returns the process ri_pageins count (via proc_pid_rusage) only while the kernel reports memory pressure at or above WARN, and None otherwise. None is already the deferral's no-reading path, so a stall the compressor serves (pressure high, disk pageins flat) never defers and exits on the normal budget. The macOS grace cap is 120s, shorter than Linux's 600s, because the page-in count is process-wide rather than per-thread. Closes #21139. --- docs/system-specs/modules/slack-gateway.md | 2 +- src/kiro_crew/dashboard/loop_watchdog.py | 108 ++++++++- src/kiro_crew/dashboard/server.py | 4 + .../dashboard/server_runtime/heartbeat.py | 16 +- src/kiro_crew/platform_compat.py | 3 + src/kiro_crew/platform_compat_proctable.py | 45 ++++ test/test_loop_watchdog.py | 229 ++++++++++++++++++ test/test_platform_compat_refactor_facade.py | 4 +- 8 files changed, 402 insertions(+), 9 deletions(-) diff --git a/docs/system-specs/modules/slack-gateway.md b/docs/system-specs/modules/slack-gateway.md index a914b036e2f5..3be1e56f0607 100644 --- a/docs/system-specs/modules/slack-gateway.md +++ b/docs/system-specs/modules/slack-gateway.md @@ -954,7 +954,7 @@ that follow it, and the signal, do not yield. The gateway runs a single asyncio loop, so any blocking call on the loop thread freezes the whole backend. App Home skill loader construction and listing run together in a worker: listing can initialize/read the persistent SQLite metadata index. Two mechanisms contain this (see `dashboard/loop_watchdog.py`, `executors.py`): - **Lag enrichment** — when the heartbeat measures lag over 1 s it claims one lag enrichment per episode (`claim_lag_enrichment`: skipped when `check()` already enriched or a capture is in flight, with a 60 s cooldown) and writes it through `log_lag_enrichment` to the logger only, never to the dump file. -- **`LoopStallWatchdog`** — armed only when `faulthandler.is_enabled()` (the real `gateway` entrypoint; not `chat`/`tui`). The async heartbeat (`dashboard/server_runtime/heartbeat.py`, 5s interval) `beat()`s it each tick, re-arming the kernel's per-process alarm (`setitimer(ITIMER_REAL)` for `exit_after` seconds, `platform_compat.arm_process_alarm`) with `faulthandler.register(SIGALRM, chain=True)` on the crash-dump file: if the loop goes silent, the alarm dumps all thread stacks from inside the signal handler — in C, with no GIL, so it fires whether the loop thread is blocked in a syscall or holding the GIL inside a long C call — and then hands `SIGALRM` to its default disposition, which ends the process. **A suspend is not a stall:** the alarm pauses while the host sleeps (Linux runs `ITIMER_REAL` on `CLOCK_MONOTONIC`; macOS schedules it on the absolute mach timebase), and the loop's own monotonic clock stands still too, so a laptop resume misses no beat and fires no deadline. faulthandler's own `dump_traceback_later` timer cannot be that decider on every platform: it waits on an interpreter lock whose deadline clock is fixed when CPython is built (`sem_clockwait(CLOCK_MONOTONIC)` with `HAVE_SEM_CLOCKWAIT`, otherwise `sem_timedwait` on `CLOCK_REALTIME`, which jumps by the whole suspend on resume and fires any pending deadline the instant the host wakes, whatever its budget — the branch every portable interpreter build and every macOS build takes). Windows has no process alarm, and a process that already handles `SIGALRM` from Python (pytest-timeout in a test worker, an embedding host) owns `ITIMER_REAL` too; in both cases that timer carries the exit at the same budget and the alarm is never armed or cancelled (`exit_mechanism()`; the startup line says `exit_after= (alarm|faulthandler)`). The mechanism is decided once per arm and latched (`_armed_mechanism`), and each beat's cancel targets the latched one, so a `SIGALRM` owner that appears between two beats moves the exit onto faulthandler's timer at the next re-arm instead of leaving the pending alarm to fire beside it. Each alarm arm releases faulthandler's `SIGALRM` registration and registers it afresh: a repeat `faulthandler.register` reinstalls nothing while faulthandler believes it still holds the signal, so a temporary owner that handed `SIGALRM` back with `SIG_DFL` would otherwise leave the next alarm to end the process without a dump. On Windows, whose `time.monotonic()` counts a sleep, the heartbeat's lag and the daemon thread's silence are read on the unbiased interrupt time instead (`platform_compat.awake_now`), so a resume is neither lag nor silence there either; faulthandler's timer, re-armed by every beat, still carries the exit. The exit is by `SIGALRM` rather than status 1 and the dump carries no `Timeout (` preamble line; no consumer of either exists. `SIGALRM` and `ITIMER_REAL` belong to the watchdog in the gateway process, and to no successor image: the exec seams cancel the alarm before `os.execv` and the successor's entrypoint clears any deadline that still arrived (see "Restart after update"). A daemon thread measures the silence since the last beat on the clock a suspend does not advance (`platform_compat.awake_now`: `time.monotonic()` on Linux and macOS, the unbiased interrupt time on Windows) for the observability layer — enrichment, then the soft dump — on its 5s poll; each poll also samples the suspend-inclusive clock `platform_compat.boottime_now` (`CLOCK_BOOTTIME` on Linux, the wall clock on macOS, `None` where none exists) and an advance there of `SUSPEND_SKEW_MIN_SECS` (2s) or more beyond the monotonic advance is logged once at INFO as a resume; it decides no exit. Desktop/foreground launches automatically use 25s; managed systemd/launchd gateways automatically use 90s because they have no Electron probe and WSL, VM, or heavy disk pressure can suspend scheduling long enough to make 25s a false death. The config value is nullable/automatic so an unrelated full config save cannot pin either launch-class default; any explicit `dashboard.loop_stall_exit_after_secs` value, including 25, overrides both. Older full-config saves may have materialized the former 25-second default; Kiro Crew reports that through the read-only superseded-default warning and `doctor` rather than guessing whether the value was deliberate. The managed path emits a non-fatal all-thread dump to stderr at `stall_after=30s`, never to the fatal crash-sentinel file, then exits at its service budget if the loop has not recovered. **The loop thread is always in the fatal dump:** faulthandler lists at most 100 threads (`FAULTHANDLER_MAX_THREADS`), newest first, so a gateway with more threads loses the oldest one, the event-loop thread. So at enrichment, while the alarm is armed, the daemon thread writes the loop thread's own stack into the crash-dump file from `sys._current_frames()`, under a `Loop thread 0x…` header and one `#` line giving the live thread count (`_write_loop_stack`); the alarm's dump lands after it. The daemon thread's first poll after a recovery, or its exit after an orderly `stop()` (including the pre-exec stop `_open_loop_watchdog` registers, so an in-place restart right after a recovered stall leaves no block), truncates the file back to where it was (`_drop_loop_stack`), so a recovered stall leaves it header-only; `beat()` and `stop()` do no file I/O on the loop thread, and a writer that sees a beat or a stop land during its write drops its own block. When faulthandler's dump also lists the loop thread (as `Current thread` on the alarm path, `Thread` on its timer path), the readers take that newer stack over the block. A daemon thread that cannot run in time (the loop thread holds the GIL in a C call, or the host is frozen) writes nothing, and the readers then say the loop thread is missing (below). **A paging loop thread is not a wedge:** on a host short of memory the kernel swaps out part of the gateway, and the loop thread stops beating while it faults those pages back in. So on Linux, once a silence reaches two poll intervals, each poll reads the loop thread's major-fault count (field 12 of `/proc/self/task//stat`, parsed by `diag.threads._parse_stat`); a rise since the previous poll of the same silence re-arms the alarm, never past `exit_after + PAGING_GRACE_MAX_SECS` (600s) of total silence, and logs one WARNING per silence saying the exit is deferred. A beat that lands after the poll's sample wins, so a deferral never shortens a recovered loop's fresh deadline. A thread blocked in a syscall or holding the GIL in a C call takes no major faults and still exits on the normal budget; a thread wedged in Python code that keeps touching swapped-out memory looks the same as paging and can stay down for up to the full grace. A budget below `stall_after` (the desktop default) defers nothing, because the desktop app's own liveness probe ends a silent gateway within a minute anyway. If the alarm is off (`exit_after=None`) or fails to arm or re-arm, no fatal capture can follow, so that soft-only dump is written to the dedicated dump file as well as stderr to remain discoverable. `KIROCREW_SERVICE_MANAGED=1` in the generated systemd unit or launchd plist is the sole managed-launch authority; inherited systemd metadata is deliberately ignored because descendants receive it too. `kirocrew doctor` detects an installed definition without the marker and tells the operator to run `kirocrew service install` once to regenerate it and adopt the managed-service default. **Sustained lag** is judged on the same daemon thread: `beat()` records the gap it closed, and each poll that sees a new beat takes that gap minus the 5s heartbeat sleep as one sample. Three samples in a row of 2s or more open an episode: one WARNING (`event loop lagging`), `lagging` set and `sustained_lag_episodes` counted; an on-time beat ends it with one INFO. This catches a loop that keeps answering seconds late without ever going silent long enough for the stall path. `/api/health` carries `loop: {lagging, sustained_lag_episodes}` behind the same direct-local gate as `version`. +- **`LoopStallWatchdog`** — armed only when `faulthandler.is_enabled()` (the real `gateway` entrypoint; not `chat`/`tui`). The async heartbeat (`dashboard/server_runtime/heartbeat.py`, 5s interval) `beat()`s it each tick, re-arming the kernel's per-process alarm (`setitimer(ITIMER_REAL)` for `exit_after` seconds, `platform_compat.arm_process_alarm`) with `faulthandler.register(SIGALRM, chain=True)` on the crash-dump file: if the loop goes silent, the alarm dumps all thread stacks from inside the signal handler — in C, with no GIL, so it fires whether the loop thread is blocked in a syscall or holding the GIL inside a long C call — and then hands `SIGALRM` to its default disposition, which ends the process. **A suspend is not a stall:** the alarm pauses while the host sleeps (Linux runs `ITIMER_REAL` on `CLOCK_MONOTONIC`; macOS schedules it on the absolute mach timebase), and the loop's own monotonic clock stands still too, so a laptop resume misses no beat and fires no deadline. faulthandler's own `dump_traceback_later` timer cannot be that decider on every platform: it waits on an interpreter lock whose deadline clock is fixed when CPython is built (`sem_clockwait(CLOCK_MONOTONIC)` with `HAVE_SEM_CLOCKWAIT`, otherwise `sem_timedwait` on `CLOCK_REALTIME`, which jumps by the whole suspend on resume and fires any pending deadline the instant the host wakes, whatever its budget — the branch every portable interpreter build and every macOS build takes). Windows has no process alarm, and a process that already handles `SIGALRM` from Python (pytest-timeout in a test worker, an embedding host) owns `ITIMER_REAL` too; in both cases that timer carries the exit at the same budget and the alarm is never armed or cancelled (`exit_mechanism()`; the startup line says `exit_after= (alarm|faulthandler)`). The mechanism is decided once per arm and latched (`_armed_mechanism`), and each beat's cancel targets the latched one, so a `SIGALRM` owner that appears between two beats moves the exit onto faulthandler's timer at the next re-arm instead of leaving the pending alarm to fire beside it. Each alarm arm releases faulthandler's `SIGALRM` registration and registers it afresh: a repeat `faulthandler.register` reinstalls nothing while faulthandler believes it still holds the signal, so a temporary owner that handed `SIGALRM` back with `SIG_DFL` would otherwise leave the next alarm to end the process without a dump. On Windows, whose `time.monotonic()` counts a sleep, the heartbeat's lag and the daemon thread's silence are read on the unbiased interrupt time instead (`platform_compat.awake_now`), so a resume is neither lag nor silence there either; faulthandler's timer, re-armed by every beat, still carries the exit. The exit is by `SIGALRM` rather than status 1 and the dump carries no `Timeout (` preamble line; no consumer of either exists. `SIGALRM` and `ITIMER_REAL` belong to the watchdog in the gateway process, and to no successor image: the exec seams cancel the alarm before `os.execv` and the successor's entrypoint clears any deadline that still arrived (see "Restart after update"). A daemon thread measures the silence since the last beat on the clock a suspend does not advance (`platform_compat.awake_now`: `time.monotonic()` on Linux and macOS, the unbiased interrupt time on Windows) for the observability layer — enrichment, then the soft dump — on its 5s poll; each poll also samples the suspend-inclusive clock `platform_compat.boottime_now` (`CLOCK_BOOTTIME` on Linux, the wall clock on macOS, `None` where none exists) and an advance there of `SUSPEND_SKEW_MIN_SECS` (2s) or more beyond the monotonic advance is logged once at INFO as a resume; it decides no exit. Desktop/foreground launches automatically use 25s; managed systemd/launchd gateways automatically use 90s because they have no Electron probe and WSL, VM, or heavy disk pressure can suspend scheduling long enough to make 25s a false death. The config value is nullable/automatic so an unrelated full config save cannot pin either launch-class default; any explicit `dashboard.loop_stall_exit_after_secs` value, including 25, overrides both. Older full-config saves may have materialized the former 25-second default; Kiro Crew reports that through the read-only superseded-default warning and `doctor` rather than guessing whether the value was deliberate. The managed path emits a non-fatal all-thread dump to stderr at `stall_after=30s`, never to the fatal crash-sentinel file, then exits at its service budget if the loop has not recovered. **The loop thread is always in the fatal dump:** faulthandler lists at most 100 threads (`FAULTHANDLER_MAX_THREADS`), newest first, so a gateway with more threads loses the oldest one, the event-loop thread. So at enrichment, while the alarm is armed, the daemon thread writes the loop thread's own stack into the crash-dump file from `sys._current_frames()`, under a `Loop thread 0x…` header and one `#` line giving the live thread count (`_write_loop_stack`); the alarm's dump lands after it. The daemon thread's first poll after a recovery, or its exit after an orderly `stop()` (including the pre-exec stop `_open_loop_watchdog` registers, so an in-place restart right after a recovered stall leaves no block), truncates the file back to where it was (`_drop_loop_stack`), so a recovered stall leaves it header-only; `beat()` and `stop()` do no file I/O on the loop thread, and a writer that sees a beat or a stop land during its write drops its own block. When faulthandler's dump also lists the loop thread (as `Current thread` on the alarm path, `Thread` on its timer path), the readers take that newer stack over the block. A daemon thread that cannot run in time (the loop thread holds the GIL in a C call, or the host is frozen) writes nothing, and the readers then say the loop thread is missing (below). **A paging loop thread is not a wedge:** on a host short of memory the kernel swaps out part of the gateway, and the loop thread stops beating while it faults those pages back in. So on Linux, once a silence reaches two poll intervals, each poll reads the loop thread's major-fault count (field 12 of `/proc/self/task//stat`, parsed by `diag.threads._parse_stat`); a rise since the previous poll of the same silence re-arms the alarm, never past `exit_after + PAGING_GRACE_MAX_SECS` (600s) of total silence, and logs one WARNING per silence saying the exit is deferred. A beat that lands after the poll's sample wins, so a deferral never shortens a recovered loop's fresh deadline. A thread blocked in a syscall or holding the GIL in a C call takes no major faults and still exits on the normal budget; a thread wedged in Python code that keeps touching swapped-out memory looks the same as paging and can stay down for up to the full grace. On macOS the per-thread major-fault reader returns `None` (procfs is Linux-only), so a macOS managed gateway swaps in `read_loop_pageins_under_pressure`: it returns the whole process's `ri_pageins` count (`proc_pid_rusage`) only while the kernel reports memory pressure at or above WARN (`memory_pressure_level`), and `None` otherwise, so the deferral fires only under real pressure and a compressor-served stall with flat disk page-ins still exits on the normal budget. Because that signal is process-wide, not per-thread, it cannot tell the loop thread's paging from another thread's, so macOS uses a shorter `PAGING_GRACE_MAX_SECS_MACOS` (120s) cap and logs a message that names the process page-in count without claiming the loop thread is paging or that the loop is not wedged. A budget below `stall_after` (the desktop default) defers nothing, because the desktop app's own liveness probe ends a silent gateway within a minute anyway. If the alarm is off (`exit_after=None`) or fails to arm or re-arm, no fatal capture can follow, so that soft-only dump is written to the dedicated dump file as well as stderr to remain discoverable. `KIROCREW_SERVICE_MANAGED=1` in the generated systemd unit or launchd plist is the sole managed-launch authority; inherited systemd metadata is deliberately ignored because descendants receive it too. `kirocrew doctor` detects an installed definition without the marker and tells the operator to run `kirocrew service install` once to regenerate it and adopt the managed-service default. **Sustained lag** is judged on the same daemon thread: `beat()` records the gap it closed, and each poll that sees a new beat takes that gap minus the 5s heartbeat sleep as one sample. Three samples in a row of 2s or more open an episode: one WARNING (`event loop lagging`), `lagging` set and `sustained_lag_episodes` counted; an on-time beat ends it with one INFO. This catches a loop that keeps answering seconds late without ever going silent long enough for the stall path. `/api/health` carries `loop: {lagging, sustained_lag_episodes}` behind the same direct-local gate as `version`. - **Cautious boot** (`dashboard/cautious_boot.py`, `dashboard.cautious_boot`, default on, restart-only) — when startup finds a loop-stall crash dump with thread stacks under 30 min old (`RECENT_DUMP_MAX_AGE_SECS`), `pause_before(group)` staggers the startup battery (MCP servers, cron scheduler, app backends, session restores): `MAX_DELAY_SECS` (10 s) on a `tight`/`critical` `resource_status` posture, `MILD_DELAY_SECS` (2 s) otherwise. A third signal downgrades it one step: both sites that publish `DashboardState.ready` dispatch (never await, so a stalled data home cannot hold `KIROCREW_READY`) `crash_dump_store.record_healthy_boot`, which writes this process's `(pid, domain, start_id)` to `/last-healthy-boot` through `atomic_write`; `dump_owner_reached_healthy` matches it against the dump header, and when the dump's writer had reached ready the next boot runs un-staggered on a calm host and mildly staggered on a pressured one. The match fails closed toward staggering: a missing marker, a start identity absent on either side, a recycled PID or another host's marker all answer no, and the read uses `O_NOFOLLOW`/`O_NONBLOCK` plus an `S_ISREG` check, bounded to 256 bytes, so a planted link or FIFO reads as absent. Any evaluation error boots normally (#16954). - **Crash-dump retention** (`dashboard/crash_dump_store.py`) — every boot runs `sweep_stale_dumps` then `rotate_dumps` before it opens its own dump. A header-only dump says only that faulthandler never fired, so an orderly end marks it: `record_clean_exit` creates an empty `.clean` beside it (`O_EXCL`, never the dump's own bytes), through `mark_clean_exit` (off the loop, bounded to 1 s) as the last step of each orderly exit (`_shutdown_and_exit` before its final force-exit park, and the owner's `/kirocrew restart` before `hard_exit`), and just before an in-place restart's exec, which skips that exit; `unmark_clean_exit` withdraws it when that restart is skipped or its exec fails, and a write still in flight then leaves nothing. Marking earlier would vouch for a teardown that can still wedge and be killed; `open_dump_file` clears any marker its new dump's name already had, and writes `_CLEAN_EXIT_AWARE_TOKEN` into the dump's first header line: a header-only dump without it was written by a build before the marker, so it reads as a clean exit and an upgrade does not report the last pre-upgrade session as unclean. The sweep removes a header-only dump only when its owner is confirmed dead AND it reads as a clean exit (the marker is present, or its header lacks the token), and deletes the marker with it; a dead owner's token-bearing header-only dump without a marker is an unexplained exit (killed or crashed without the alarm firing) and is kept, as the only trace of that gateway, for the `kirocrew doctor` `unclean:` row (`unexplained_exits`). Under the `max_dumps` cap rotation sacrifices marked or unattributable header-only dumps first, then unexplained exits, then unreadable files, then dumps with stacks, oldest first within each; live-owner and foreign-domain dumps are never victims. Startup replay and cautious boot read stack-bearing dumps only, and the diagnostics bundle leaves markers out and gives unexplained exits only the crash-dump slots nothing else fills (#14724). - **Bounded executors** — blocking work is offloaded off the default executor (which the loop uses for DNS) into dedicated bounded pools defined in `executors.py`; for example `maintenance_executor()` (`mc-maint`, fast orphan-reaping sweeps + agent-overlay rewrites) and `cron_executor()` (`mc-cron`, long/concurrent cron command & script jobs), kept separate so a burst of cron jobs cannot starve the orphan sweeps. MCP `probe_all()` fan-out is bounded by `PROBE_MAX_CONCURRENCY` (5). diff --git a/src/kiro_crew/dashboard/loop_watchdog.py b/src/kiro_crew/dashboard/loop_watchdog.py index dca84b72cfdb..1a43411243da 100644 --- a/src/kiro_crew/dashboard/loop_watchdog.py +++ b/src/kiro_crew/dashboard/loop_watchdog.py @@ -120,9 +120,12 @@ from kiro_crew.dashboard.stall_enrichment import collect_stall_enrichment, describe_loop_thread from kiro_crew.diag.threads import _parse_stat, _read_text, _task_dir from kiro_crew.platform_compat import ( + MEMORY_PRESSURE_WARN, arm_process_alarm, awake_now, boottime_now, + memory_pressure_level, + proc_pageins_for_pid, process_alarm_available, ) @@ -139,6 +142,31 @@ #: paging still ends the process. PAGING_GRACE_MAX_SECS: float = 600.0 +#: The macOS grace cap, shorter than the Linux one on purpose. The macOS +#: page-in signal (:func:`read_loop_pageins_under_pressure`) is process-wide, +#: not per-thread, so it cannot tell "the loop thread is paging" from "some +#: other thread in the process is paging". A shorter cap bounds how long a +#: genuinely wedged loop can be kept alive by an unrelated thread's paging. +PAGING_GRACE_MAX_SECS_MACOS: float = 120.0 + +#: The WARNING a paging deferral logs, once per silence, in two forms selected +#: by the grace cap -- already how the macOS watchdog is distinguished without a +#: platform branch (see :data:`PAGING_GRACE_MAX_SECS_MACOS`). The Linux signal +#: is the loop thread's own major-fault count, so its message names the thread +#: and rules out a wedge. The macOS signal is the whole process's page-in count, +#: which cannot attribute paging to the loop thread, so its message makes no +#: such claim. Each takes three %-args: silence seconds, count delta, budget. +_PAGING_MESSAGE_LINUX: str = ( + "event loop silent %.1fs while its thread is paging (%d major faults " + "since the last poll): the host is short of memory, not the loop " + "wedged, so the stall exit is deferred for up to %.0fs of silence" +) +_PAGING_MESSAGE_MACOS: str = ( + "event loop silent %.1fs while the process is paging (%d page-ins " + "under memory pressure since the last poll): the stall exit is " + "deferred for up to %.0fs of silence" +) + def read_thread_major_faults(native_thread_id: int | None) -> int | None: """Major page faults taken so far by one thread of this process. @@ -156,6 +184,48 @@ def read_thread_major_faults(native_thread_id: int | None) -> int | None: return major_faults if isinstance(major_faults, int) else None +def read_loop_pageins_under_pressure( + _native_thread_id: int | None, + *, + pressure_level: "Callable[[], int | None]" = memory_pressure_level, + pageins: "Callable[[int], int | None]" = proc_pageins_for_pid, + warn_level: int = MEMORY_PRESSURE_WARN, +) -> int | None: + """macOS analogue of :func:`read_thread_major_faults` for the paging deferral. + + Plugs into the same ``loop_major_faults`` hook, so :meth:`LoopStallWatchdog. + _defer_exit_while_paging` needs no change. It returns the process's + ``ri_pageins`` count **only while the host is under memory pressure** + (``memory_pressure_level`` at or above WARN); otherwise it returns ``None``. + + The ``None``-when-not-pressured choice is deliberate and load-bearing. The + deferral acts on a *rise* between two readings, and ``None`` is already its + "no reading -- reset the baseline, defer nothing" path. Returning a flat + ``0`` instead would manufacture a huge false rise on the first pressured poll + after an unpressured one (``0`` -> true count) and defer a genuinely wedged + loop. So the pressure gate folds into the reader as: no pressure -> ``None`` + -> no baseline -> no deferral. + + This is also why a stall the compressor serves (pressure high, disk pageins + flat) still exits on the normal budget: the count does not rise, so nothing + defers. The signal only ever *extends* the budget on real disk-pagein + progress; it can never keep a flat-pagein stall alive. + + The signal is process-wide, not per-thread -- macOS exposes no public + per-thread page-in count -- so ``_native_thread_id`` is accepted to match + :func:`read_thread_major_faults`'s signature and ignored. That coarseness + is bounded by the shorter :data:`PAGING_GRACE_MAX_SECS_MACOS`. + + Off macOS, or on any unreadable signal, returns ``None``. The two readers + are injectable so the deferral can be driven deterministically in tests + without ever inducing real memory pressure. + """ + level = pressure_level() + if level is None or level < warn_level: + return None + return pageins(os.getpid()) + + #: The mechanism the last :func:`_default_arm_later` armed the dump-then-exit #: on — ``"alarm"`` or ``"faulthandler"`` — or ``None`` while it holds no #: pending deadline. Process-wide like the two deadlines it names @@ -371,7 +441,13 @@ class LoopStallWatchdog: :func:`kiro_crew.dashboard.stall_enrichment.collect_stall_enrichment`. loop_major_faults: Reads the loop thread's major-fault count from its native thread id, ``None`` when unknown; injectable for tests. - Defaults to :func:`read_thread_major_faults`. + Defaults to :func:`read_thread_major_faults`. macOS passes + :func:`read_loop_pageins_under_pressure` instead. + paging_grace_max: Longest the exit may be deferred past ``exit_after`` + while the loop keeps paging. ``None`` resolves to the module + :data:`PAGING_GRACE_MAX_SECS` at construction; macOS passes the + shorter :data:`PAGING_GRACE_MAX_SECS_MACOS` because its page-in + signal is process-wide rather than per-thread. log: Logger, injectable for tests. """ @@ -399,6 +475,7 @@ def __init__( enrich_after: float = 15.0, enrich: "Callable[[float], list[str]] | None" = None, loop_major_faults: Callable[[int | None], int | None] = read_thread_major_faults, + paging_grace_max: float | None = None, log: logging.Logger | None = None, ) -> None: self._stall_after = stall_after @@ -432,6 +509,27 @@ def __init__( # The kernel's id for that same thread, which names its procfs row. self._loop_native_thread_id: int | None = None self._loop_major_faults = loop_major_faults + # The grace cap for THIS instance, resolved from the module default at + # construction when not given. Reading the module name here (not as a + # default-arg value, which binds once at def time) keeps a + # ``monkeypatch.setattr(loop_watchdog, "PAGING_GRACE_MAX_SECS", ...)`` + # done before construction effective, and lets the macOS watchdog pass + # the shorter ``PAGING_GRACE_MAX_SECS_MACOS`` without a platform branch + # inside the hot path. + self._paging_grace_max = ( + PAGING_GRACE_MAX_SECS if paging_grace_max is None else paging_grace_max + ) + # The WARNING a paging deferral logs, once per silence. Selected from + # the grace cap, the same signal that already distinguishes the macOS + # watchdog without a platform branch: the macOS cap pairs with the + # process-wide page-in message, every other cap with the Linux + # per-thread major-fault message. Takes three %-args: silence seconds, + # the count delta, total budget. + self._paging_log_message = ( + _PAGING_MESSAGE_MACOS + if self._paging_grace_max == PAGING_GRACE_MAX_SECS_MACOS + else _PAGING_MESSAGE_LINUX + ) # Counts beats, so a paging deferral can tell that the loop beat after # it sampled and must leave the beat's fresh deadline alone. self._beat_generation = 0 @@ -755,7 +853,7 @@ def _defer_exit_while_paging(self, silence: float) -> None: return if major_faults <= baseline[1]: return - remaining_budget = exit_after + PAGING_GRACE_MAX_SECS - silence + remaining_budget = exit_after + self._paging_grace_max - silence if remaining_budget <= 0: return if not self._rearm_alarm(min(exit_after, remaining_budget), unless_beat_since=generation): @@ -763,12 +861,10 @@ def _defer_exit_while_paging(self, silence: float) -> None: if self._paging_logged_generation != generation: self._paging_logged_generation = generation self._log.warning( - "event loop silent %.1fs while its thread is paging (%d major faults " - "since the last poll): the host is short of memory, not the loop " - "wedged, so the stall exit is deferred for up to %.0fs of silence", + self._paging_log_message, silence, major_faults - baseline[1], - exit_after + PAGING_GRACE_MAX_SECS, + exit_after + self._paging_grace_max, ) def claim_lag_enrichment(self, lag: float) -> bool: diff --git a/src/kiro_crew/dashboard/server.py b/src/kiro_crew/dashboard/server.py index 4f556ae33a86..4353deac257d 100644 --- a/src/kiro_crew/dashboard/server.py +++ b/src/kiro_crew/dashboard/server.py @@ -158,6 +158,10 @@ release_site, ) from kiro_crew.dashboard.loop_watchdog import LoopStallWatchdog # noqa: F401 +from kiro_crew.dashboard.loop_watchdog import ( # noqa: F401 + PAGING_GRACE_MAX_SECS_MACOS, + read_loop_pageins_under_pressure, +) from kiro_crew.dashboard.origin import ( # noqa: F401 AUDIT_CLAIMED_KEY, PROBE_PATHS, diff --git a/src/kiro_crew/dashboard/server_runtime/heartbeat.py b/src/kiro_crew/dashboard/server_runtime/heartbeat.py index 400bede2c8a3..5881a95184db 100644 --- a/src/kiro_crew/dashboard/server_runtime/heartbeat.py +++ b/src/kiro_crew/dashboard/server_runtime/heartbeat.py @@ -14,6 +14,7 @@ if TYPE_CHECKING: from kiro_crew.dashboard.server import ( + PAGING_GRACE_MAX_SECS_MACOS, DashboardState, LoopStallWatchdog, attribute_dump, @@ -27,6 +28,7 @@ newest_dump_with_stacks, open_dump_file, platform_compat, + read_loop_pageins_under_pressure, record_healthy_boot, register_pre_exec_stop, resolve_loop_stall_exit_after, @@ -79,7 +81,19 @@ async def _open_loop_watchdog(_launch_environment: dict[str, str]) -> LoopStallW # Config failure must not erase the managed-service grace that protects # the process while its config filesystem is itself under pressure. _exit_after = float(resolve_loop_stall_exit_after(environ=_launch_environment)) - watchdog = LoopStallWatchdog(dump_file=_dump_file, exit_after=_exit_after) + # On macOS the Linux per-thread major-fault reader returns None, so the + # paging deferral never fires. Swap in the process-wide page-in reader + # (gated on memory pressure) with a shorter grace cap, since that signal + # cannot attribute paging to the loop thread specifically. + if platform_compat.IS_MACOS: + watchdog = LoopStallWatchdog( + dump_file=_dump_file, + exit_after=_exit_after, + loop_major_faults=read_loop_pageins_under_pressure, + paging_grace_max=PAGING_GRACE_MAX_SECS_MACOS, + ) + else: + watchdog = LoopStallWatchdog(dump_file=_dump_file, exit_after=_exit_after) async def _stop_before_exec() -> None: # An in-place restart can exec right after a recovered stall, before diff --git a/src/kiro_crew/platform_compat.py b/src/kiro_crew/platform_compat.py index 048a16b635b0..483f38638859 100644 --- a/src/kiro_crew/platform_compat.py +++ b/src/kiro_crew/platform_compat.py @@ -605,6 +605,7 @@ def __dir__() -> list[str]: _DARWIN_PTI_RESIDENT_SIZE_OFFSET, _DARWIN_PTI_TOTAL_SYSTEM_OFFSET, _DARWIN_PTI_TOTAL_USER_OFFSET, + _DARWIN_RI_PAGEINS_OFFSET, _DARWIN_RI_PHYS_FOOTPRINT_OFFSET, _DARWIN_RUSAGE_INFO_V2, _DARWIN_RUSAGE_INFO_V2_SIZE, @@ -639,6 +640,7 @@ def __dir__() -> list[str]: _darwin_peer_is_loopback, _darwin_process_cpu_nanos, _darwin_process_cwd, + _darwin_process_pageins, _darwin_process_phys_footprint_bytes, _darwin_process_start_identity, _darwin_process_start_microtime, @@ -676,6 +678,7 @@ def __dir__() -> list[str]: pids_holding_file, proc_child_map, proc_cpu_jiffies_for_pids, + proc_pageins_for_pid, proc_phys_footprint_bytes_for_pid, read_proc_stat, read_proc_status_int, diff --git a/src/kiro_crew/platform_compat_proctable.py b/src/kiro_crew/platform_compat_proctable.py index 63aed7d33f78..4f714a37246f 100644 --- a/src/kiro_crew/platform_compat_proctable.py +++ b/src/kiro_crew/platform_compat_proctable.py @@ -83,6 +83,13 @@ class ProcessStartIdentity(NamedTuple): _DARWIN_RUSAGE_INFO_V2 = 2 _DARWIN_RUSAGE_INFO_V2_SIZE = 16 + 18 * 8 _DARWIN_RI_PHYS_FOOTPRINT_OFFSET = 16 + 7 * 8 +# ``ri_pageins`` is the fifth uint64 in ``rusage_info_v2`` (user time, system +# time, the two wakeup counters, then pageins), so 16 bytes of ``ri_uuid`` plus +# four uint64 fields. It counts the disk pageins this process has taken since it +# started -- a monotonic process-wide counter, NOT per-thread, and it does NOT +# include compressor decompressions (macOS serves most memory pressure from the +# compressor, so this can stay flat while the process stalls paging). +_DARWIN_RI_PAGEINS_OFFSET = 16 + 4 * 8 _darwin_libproc: Any = None _darwin_libproc_loaded = False @@ -232,6 +239,44 @@ def proc_phys_footprint_bytes_for_pid(pid: int) -> int | None: return _darwin_process_phys_footprint_bytes(pid) +def _darwin_process_pageins(pid: int) -> int | None: + """macOS ``ri_pageins`` of *pid* via ``proc_pid_rusage``, or None when unreadable. + + The count of disk pageins the process has taken since it started: a + monotonic, process-wide counter. It is NOT per-thread, and it does NOT + count compressor decompressions, so under memory pressure served entirely + from the compressor it can stay flat while the process stalls paging. + Needs no entitlement for a same-uid process and never spawns a subprocess. + """ + lib = _darwin_libproc_handle() + rusage = getattr(lib, "proc_pid_rusage", None) if lib is not None else None + if rusage is None: + return None + try: + buf = ctypes.create_string_buffer(_DARWIN_RUSAGE_INFO_V2_SIZE) + if rusage(pid, _DARWIN_RUSAGE_INFO_V2, buf) != 0: + return None + off = _DARWIN_RI_PAGEINS_OFFSET + # Both x86_64 and arm64 macOS are little-endian. + return int.from_bytes(buf.raw[off : off + 8], "little") + except Exception: + return None + + +def proc_pageins_for_pid(pid: int) -> int | None: + """``ri_pageins`` of *pid* on macOS; None elsewhere or unreadable. + + The process-wide disk page-in count (see ``_darwin_process_pageins``). Other + platforms have no equivalent and answer None. A rise between two readings of + one silence is the macOS analogue of the Linux loop-stall watchdog's + per-thread major-fault delta -- coarser, because it is process-wide, which + is why the deferral it feeds uses a shorter grace cap. + """ + if not IS_MACOS: + return None + return _darwin_process_pageins(pid) + + def darwin_libproc_available() -> bool: """True when ``libproc`` loads, so the macOS memory and tree probes can answer.""" return _darwin_libproc_handle() is not None diff --git a/test/test_loop_watchdog.py b/test/test_loop_watchdog.py index 74299c0f08c7..6344e6914e92 100644 --- a/test/test_loop_watchdog.py +++ b/test/test_loop_watchdog.py @@ -14,6 +14,7 @@ import ctypes import logging +import os import signal import subprocess import sys @@ -502,6 +503,234 @@ def test_the_fault_reader_reads_this_threads_real_procfs_row() -> None: assert isinstance(count, int) and count >= 0 +# ── macOS page-in reader + its paging deferral ─────────────────────────────── +# +# None of these tests induce real memory pressure. The pressure level and the +# page-in count are injected callables, so the whole gate+delta behaviour is +# driven deterministically. This is the safety contract: simulating the paging +# scenario must never risk the test host under real swap. + +_MACOS_WARN = platform_compat.MEMORY_PRESSURE_WARN +_MACOS_NORMAL = platform_compat.MEMORY_PRESSURE_NORMAL + + +def _macos_reader(*, pressure: "Callable[[], int | None]", pageins: "Callable[[int], int | None]"): + """A macOS reader bound to injected pressure/pageins, matching the hook signature.""" + + def read(native_thread_id: int | None) -> int | None: + return loop_watchdog.read_loop_pageins_under_pressure( + native_thread_id, pressure_level=pressure, pageins=pageins, warn_level=_MACOS_WARN + ) + + return read + + +def test_macos_reader_returns_none_without_pressure() -> None: + read = _macos_reader(pressure=lambda: _MACOS_NORMAL, pageins=lambda _p: 500) + assert read(None) is None + + +def test_macos_reader_returns_pageins_under_pressure() -> None: + read = _macos_reader(pressure=lambda: _MACOS_WARN, pageins=lambda _p: 500) + assert read(None) == 500 + + +def test_macos_reader_returns_none_when_pressure_unreadable() -> None: + read = _macos_reader(pressure=lambda: None, pageins=lambda _p: 500) + assert read(None) is None + + +def test_macos_reader_returns_none_when_pageins_unreadable() -> None: + read = _macos_reader(pressure=lambda: _MACOS_WARN, pageins=lambda _p: None) + assert read(None) is None + + +class _MacosPagingFixture: + """A started watchdog wired with the macOS page-in reader and the shorter + macOS grace cap, stepped by hand against fake clocks and injected pressure + and page-in readings. No real memory pressure is ever created.""" + + def __init__( + self, *, exit_after: float = _MANAGED_BUDGET, paging_grace_max: float | None = None + ) -> None: + self.clock = _Clock() + self.arms: list[float] = [] + self.pressure: int | None = _MACOS_WARN + self.pageins: int = 100 + self.handler = _RecordingHandler() + log = logging.Logger("test.loop_watchdog.macos_paging") + log.addHandler(self.handler) + self.wd = LoopStallWatchdog( + stall_after=30.0, + exit_after=exit_after, + poll_interval=5.0, + now=self.clock, + suspend_clock=lambda: None, + dump=lambda: None, + arm_later=self.arms.append, + cancel_later=lambda: None, + loop_major_faults=_macos_reader( + pressure=lambda: self.pressure, pageins=lambda _p: self.pageins + ), + paging_grace_max=( + loop_watchdog.PAGING_GRACE_MAX_SECS_MACOS + if paging_grace_max is None + else paging_grace_max + ), + log=log, + ) + self.wd._run = lambda: None # type: ignore[method-assign] + self.wd.start() + self.wd.beat() + self.arms.clear() + + def poll_at(self, silence: float, *, add_pageins: int = 0, pressure: int | None = -1) -> None: + self.clock.t = self.wd._last_beat + silence + if pressure != -1: + self.pressure = pressure + self.pageins += add_pageins + self.wd.check() + + def paging_lines(self) -> list[str]: + return [m for m in self.handler.messages if "is paging" in m] + + +def test_macos_pressure_and_rising_pageins_defers_and_says_so_once() -> None: + fx = _MacosPagingFixture() + try: + fx.poll_at(5.0, add_pageins=7) # under two poll intervals: ignored + fx.poll_at(10.0) # baseline + fx.poll_at(15.0, add_pageins=40) + fx.poll_at(20.0, add_pageins=12) + assert fx.arms == [_MANAGED_BUDGET, _MANAGED_BUDGET] + assert len(fx.paging_lines()) == 1 + line = fx.paging_lines()[0] + assert "40 page-ins" in line + # The macOS signal is process-wide, so the message must not claim the + # loop thread is paging or that the loop is not wedged (Opus review). + assert "major faults" not in line + assert "its thread" not in line + assert "not the loop" not in line + finally: + fx.wd.stop() + + +def test_macos_rising_pageins_without_pressure_never_defers() -> None: + fx = _MacosPagingFixture() + fx.pressure = _MACOS_NORMAL + try: + fx.poll_at(10.0) + fx.poll_at(15.0, add_pageins=40) + fx.poll_at(20.0, add_pageins=40) + assert fx.arms == [] + assert fx.paging_lines() == [] + finally: + fx.wd.stop() + + +def test_macos_pressure_but_flat_pageins_exits_on_budget() -> None: + """The compressor case: pressure is high but disk page-ins stay flat, so the + count never rises, nothing defers, and the pre-armed budget stands.""" + fx = _MacosPagingFixture() + try: + for silence in (10.0, 15.0, 20.0, 25.0, 30.0): + fx.poll_at(silence) # add_pageins=0 -> flat + assert fx.arms == [] + assert fx.paging_lines() == [] + finally: + fx.wd.stop() + + +def test_macos_pressure_clearing_mid_episode_does_not_fake_a_rise() -> None: + """Proves the reader returns None (not 0) without pressure: a pressured poll + following an unpressured one must not read as a huge page-in jump.""" + fx = _MacosPagingFixture() + try: + fx.poll_at(10.0) # pressured baseline, pageins=100 + fx.poll_at(15.0, pressure=_MACOS_NORMAL, add_pageins=500) # reader -> None + fx.poll_at(20.0, pressure=_MACOS_WARN) # back under pressure, pageins=600 + # The jump to 600 must NOT be seen as a 500 rise from a phantom 0 baseline. + assert fx.arms == [] + assert fx.paging_lines() == [] + finally: + fx.wd.stop() + + +def test_macos_unreadable_signal_never_defers() -> None: + fx = _MacosPagingFixture() + fx.pressure = None + try: + for silence in (10.0, 15.0, 20.0, 25.0): + fx.poll_at(silence, add_pageins=40) + assert fx.arms == [] + finally: + fx.wd.stop() + + +def test_macos_foreground_budget_defers_nothing() -> None: + """A budget below stall_after is the desktop/foreground class: no deferral, + and the page-in reader is never even consulted.""" + reads: list[int | None] = [] + fx = _MacosPagingFixture(exit_after=_EXIT_BUDGET) + original = fx.wd._loop_major_faults + + def _counting(tid: int | None) -> int | None: + reads.append(tid) + return original(tid) + + fx.wd._loop_major_faults = _counting # type: ignore[method-assign] + try: + fx.poll_at(10.0) + fx.poll_at(15.0, add_pageins=40) + fx.poll_at(20.0, add_pageins=40) + assert fx.arms == [] + assert reads == [] + finally: + fx.wd.stop() + + +def test_macos_deferral_stops_at_the_shorter_grace_cap() -> None: + """The macOS cap is honored through the constructor kwarg, mirroring the + Linux module-constant cap test but without monkeypatching a global.""" + fx = _MacosPagingFixture(paging_grace_max=30.0) + try: + fx.poll_at(105.0) + fx.poll_at(110.0, add_pageins=1) # 10s of the 90+30=120s cap left + assert fx.arms == [pytest.approx(10.0)] + fx.poll_at(125.0, add_pageins=1) # past the cap: no re-arm + assert fx.arms == [pytest.approx(10.0)] + finally: + fx.wd.stop() + + +def test_macos_uses_the_120s_default_grace_cap() -> None: + fx = _MacosPagingFixture() + try: + assert fx.wd._paging_grace_max == loop_watchdog.PAGING_GRACE_MAX_SECS_MACOS == 120.0 + finally: + fx.wd.stop() + + +@pytest.mark.skipif(sys.platform != "darwin", reason="ri_pageins is a macOS counter") +def test_macos_pageins_reader_reads_this_processs_real_counter() -> None: + """Read-only ABI check: ri_pageins at offset 48 is a non-negative int for + our own pid, and distinct from phys_footprint (offset 72) on the same buffer. + Does NOT try to force a page-in (that would need real swap).""" + pid = os.getpid() + pageins = platform_compat.proc_pageins_for_pid(pid) + assert isinstance(pageins, int) and pageins >= 0 + footprint = platform_compat.proc_phys_footprint_bytes_for_pid(pid) + assert isinstance(footprint, int) and footprint > 0 + # A mis-sliced offset straddling the wrong field would make these collide; + # a live process's footprint in bytes dwarfs its page-in count. + assert pageins != footprint + + +def test_macos_pageins_reader_is_none_off_darwin(monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr(platform_compat, "IS_MACOS", False) + assert platform_compat.proc_pageins_for_pid(os.getpid()) is None + + class _FakeDumpFile: """DumpFile stand-in proving the watchdog NEVER writes into the sentinel.""" diff --git a/test/test_platform_compat_refactor_facade.py b/test/test_platform_compat_refactor_facade.py index a66612a8c256..a2c9166efa5f 100644 --- a/test/test_platform_compat_refactor_facade.py +++ b/test/test_platform_compat_refactor_facade.py @@ -140,9 +140,11 @@ _DARWIN_PBI_START_TVUSEC_OFFSET ProcessStartIdentity _DARWIN_PROC_PIDTASKINFO _DARWIN_PROC_TASKINFO_SIZE _DARWIN_PTI_TOTAL_USER_OFFSET _DARWIN_PTI_TOTAL_SYSTEM_OFFSET _DARWIN_RUSAGE_INFO_V2 _DARWIN_RUSAGE_INFO_V2_SIZE _DARWIN_RI_PHYS_FOOTPRINT_OFFSET + _DARWIN_RI_PAGEINS_OFFSET _darwin_libproc _darwin_libproc_loaded _darwin_libproc_handle _darwin_process_cwd _darwin_process_start_identity _darwin_process_phys_footprint_bytes - proc_phys_footprint_bytes_for_pid darwin_libproc_available darwin_footprint_tree_mb + _darwin_process_pageins + proc_phys_footprint_bytes_for_pid proc_pageins_for_pid darwin_libproc_available darwin_footprint_tree_mb _darwin_process_start_microtime _darwin_process_cpu_nanos _DARWIN_PBI_STATUS_OFFSET _DARWIN_SZOMB _DARWIN_PIDPATH_MAXSIZE _DARWIN_CTL_KERN _DARWIN_KERN_PROCARGS2 _DARWIN_PROCARGS_BUFSIZE _DARWIN_PROCARGS_ENV_BUFSIZE _DARWIN_KERN_PROC From 140fed50a4bfd500f9ab8f4e72312b856823bbe4 Mon Sep 17 00:00:00 2001 From: EC2 Default User Date: Fri, 6 Nov 2026 16:53:55 +0000 Subject: [PATCH 2/2] test(watchdog): mark the macOS fixture's fake-clock grace as deliberate The Flake Pattern Ratchet counts a literal on a grace-named kwarg as a short wall-clock wait (K19). Both new sites feed the hand-stepped fake clock, so carry flake-ok with that reason. --- test/test_loop_watchdog.py | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/test/test_loop_watchdog.py b/test/test_loop_watchdog.py index 6344e6914e92..d0f8ee601e78 100644 --- a/test/test_loop_watchdog.py +++ b/test/test_loop_watchdog.py @@ -572,7 +572,7 @@ def __init__( loop_major_faults=_macos_reader( pressure=lambda: self.pressure, pageins=lambda _p: self.pageins ), - paging_grace_max=( + paging_grace_max=( # flake-ok: fake-clock seconds, checked by hand-stepped polls, never a wall wait loop_watchdog.PAGING_GRACE_MAX_SECS_MACOS if paging_grace_max is None else paging_grace_max @@ -692,7 +692,9 @@ def _counting(tid: int | None) -> int | None: def test_macos_deferral_stops_at_the_shorter_grace_cap() -> None: """The macOS cap is honored through the constructor kwarg, mirroring the Linux module-constant cap test but without monkeypatching a global.""" - fx = _MacosPagingFixture(paging_grace_max=30.0) + fx = _MacosPagingFixture( + paging_grace_max=30.0 + ) # flake-ok: fake-clock seconds, never a wall wait try: fx.poll_at(105.0) fx.poll_at(110.0, add_pageins=1) # 10s of the 90+30=120s cap left