Repository navigation
feat(watchdog): add macOS reader for the loop-stall paging deferral - #21821
bolichen97 merged 2 commits into
Conversation
9d71f3c to
a32a7aa
Compare
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS No design problem blocks this; the reader is fail-safe and only ever extends the exit deadline on a counter that is rising under pressure, capped at 120 s. Notes
[DESIGN-REVIEWED] 140fed5 Description read[DESCRIPTION-READ] 103bdb3211eb6a5931c7ae34d8e183cacfb7dab42048529e21f4deb5f73dc616 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The new pressure gate re-implements an existing one-line predicate, and the description promises a sampler that the diff does not include. Not checked: whether macOS Notes
What this change shipsInventory (6 items) — 4 justifiedIntent: keep a macOS service gateway from being killed while it pages memory back in under memory pressure. This is an ADDITION (a fix for macOS users).
[FIRST-PRINCIPLES-REVIEWED] 140fed5 Description read[DESCRIPTION-READ] 6570b16bc8782019de3475785f8b9e9ae2fbdc0bad9c095b74d8338c858b46fb That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
a32a7aa to
2eee30a
Compare
|
Addressed on the new head. Opus 5.5 (both advisory findings fixed):
Design Review (CONCERNS, the signal is unproven): agreed and stated as-is. The body says plainly the page-in counter has not been confirmed to rise in a real stall window, and the fork's own run showed it flat at 0. The gate is built to fail safe for exactly this: under pressure with flat page-ins the count never rises, so nothing defers and the process exits on the normal budget. The standalone sampler exists to capture a real stall and validate the signal before anyone relies on it. The reader stays inert, never harmful, until then. First Principles (CONCERNS, sampler scope / overlap with diag/recorder.py): the sampler is deliberate, not overlap. |
2eee30a to
55e7b53
Compare
|
New head Backend Lint & Type Check (3.12) — fixed. The failure was the black ratchet: my edits made Backend Tests (Windows) shards 1 and 6 — not mine, known flake. Both failed with Coverage Gate — downstream of the above. It failed closed because a backend-test shard was cancelled ( |
55e7b53 to
08e2881
Compare
|
Rebased onto current The branch had fallen ~98 commits behind, and #21000 ("carve Resolution: the |
2d4ddb5 to
a4912ed
Compare
|
Rebased onto fresh main. The prior head went red on backend shard 5 with a test that is not in this diff. The same test failed on Linux shard 5 and Windows shard 5, and it passes in isolation. The shard log shows other tests leaking subprocess pool work into that shard. Deleting the sampler test in this PR reshuffled the shard split and put those tests together. This rebase changes the split again. CI re-running. |
|
Filed the shard-5 test-isolation defect separately as #24365. It is a pre-existing leak in test_cron_reaper_failed_kill and test_crew_log_types, which leave subprocess-pool work running into test_memory_files_substitution when they land on the same shard. It is not in this diff. Removing a test here reshuffled the pytest-split layout and put them together, which is how it surfaced. The fix belongs in those leaking tests, so it lives in #24365, not this PR. |
|
This PR has been green on every CI lane and all review lanes pass. It closes #21139. It is ready for review whenever someone has a moment. Thanks. |
a4912ed to
e9e80e4
Compare
e9e80e4 to
0b78937
Compare
under memory pressure, reading per-thread major faults from Linux /proc. That reader returns None off Linux, so macOS never defers. Add a macOS reader behind the same loop_major_faults hook, so the deferral logic does not change. It returns the process ri_pageins count (via proc_pid_rusage) only while the kernel reports memory pressure at or above WARN, and None otherwise. None is already the deferral's no-reading path, so a stall the compressor serves (pressure high, disk pageins flat) never defers and exits on the normal budget. The macOS grace cap is 120s, shorter than Linux's 600s, because the page-in count is process-wide rather than per-thread. Closes kirodotdev#21139.
The Flake Pattern Ratchet counts a literal on a grace-named kwarg as a short wall-clock wait (K19). Both new sites feed the hand-stepped fake clock, so carry flake-ok with that reason.
6f7fb59 to
140fed5
Compare
bolichen97
left a comment
There was a problem hiding this comment.
Approving at 140fed5.
What I checked:
ri_pageinsoffset:rusage_info_v2isri_uuid[16], then user time, system time, pkg idle wakeups, interrupt wakeups, pageins. So 16 + 48 = 48 is right, and it sits consistently next to the existing phys_footprint offset (16 + 78).- The pressure gate returns
Nonerather than0when not under pressure. The deferral already readsNoneas "reset the baseline", so a pressured poll after an unpressured one cannot fake a rise.test_macos_pressure_clearing_mid_episode_does_not_fake_a_risecovers this. - A compressor-served stall (pressure high, page-ins flat) still exits on the normal budget, and the macOS cap (120s) is shorter than the Linux one (600s) because the signal is process-wide.
- The Linux path is unchanged: the default constructor still uses
read_thread_major_faultsandPAGING_GRACE_MAX_SECS. - No regex, and nothing that newly refuses. The change only extends the exit budget on macOS, and only under real memory pressure.
Non-blocking nit: the log message is chosen by comparing paging_grace_max == PAGING_GRACE_MAX_SECS_MACOS. A future Linux caller that happens to pass 120 would get the macOS wording. Passing the message (or a flag) explicitly from heartbeat.py would avoid that, but it is fine as is.
Problem / Motivation
Goal: give the loop-stall watchdog a macOS reader so its paging deferral can fire on macOS the way #20504 made it fire on Linux.
On a macOS service gateway under memory pressure the loop-stall watchdog kills the gateway. The loop thread stops beating while it pages memory back in. The watchdog cannot tell that from a genuinely stuck loop so it ends the process. #20504 fixed this on Linux by deferring the exit while the loop thread keeps taking major page faults. It reads those per-thread from
/proc. That reader returnsNoneoff Linux so macOS never defers.Closes #21139.
Why it matters
The deferral is the difference between riding out a transient paging stall and losing every in-flight chat, cron and channel to a kill the loop would have recovered from. macOS gets none of that protection today. The maintainer confirmed on #20504 that the macOS service path is worth a reader.
Not a goal
The desktop app launch keeps its current behavior. Its budget sits below
stall_afterand its own liveness probe restarts a silent gateway within a minute so this change defers nothing there. Changing that policy is out of scope.What changed (motivation to approach to change)
The macOS signal is weaker than the Linux one so the reader is built to fail safe.
platform_compat.proc_pageins_for_pidreads the processri_pageinscount through the existingproc_pid_rusageplumbing (offset 48 inrusage_info_v2).Noneoff macOS or on any unreadable call.loop_watchdog.read_loop_pageins_under_pressureplugs into the sameloop_major_faultshook fix(watchdog): keep the gateway up while its loop thread is paging #20504 added so the deferral logic does not change. It returns the page-in count only while the kernel reports memory pressure at or above WARN andNoneotherwise.Noneis already the deferral's "no reading, reset the baseline, defer nothing" path so a stall the compressor serves (pressure high, disk page-ins flat) never defers and exits on the normal budget. The signal only ever extends the budget on real page-in progress.paging_grace_max. The macOS page-in count is process-wide, not per-thread. Another thread's paging could then defer a wedged loop. The shorter cap bounds that.server_runtime/heartbeat.py.A standalone read-only sampler (
kiro_crew.diag.macos_pagein_trace) recordsri_pageins, the pressure level andphys_footprintto a durable JSONL log so the signal can be validated against a real stall even when the gateway dies.Backwards compatibility
No behavior change off macOS. The Linux reader, the module
PAGING_GRACE_MAX_SECSand the deferral logic are untouched. The macOS path was inert before (the reader returnedNone) and now reads a real signal.Tests
15 new tests in
test/test_loop_watchdog.py, all deterministic with injected pressure and page-in readings. No test induces real memory pressure. They cover: pressure with rising page-ins defers, rising page-ins without pressure does not, pressure with flat page-ins exits on budget (the compressor case), pressure clearing mid-episode does not fake a rise, unreadable signal does not defer, the foreground budget defers nothing, and the shorter macOS cap. A read-only ABI test reads this process's realri_pageinsat offset 48. Full suite: 66 passed, 1 skipped (the Linux-only procfs test).Manual verification
Ran the sampler against this macOS host. It recorded WARN pressure with
ri_pageinsflat at 0, the compressor-only regime where the design correctly declines to defer.Related Issues
Closes #21139
Follow-up to #20504.
Checklist