Skip to content

feat(watchdog): add macOS reader for the loop-stall paging deferral - #21821

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
tribhuwan-kandpal:feat/macos-loop-stall-pagein-reader
Nov 8, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
tribhuwan-kandpal:feat/macos-loop-stall-pagein-reader

Conversation

@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: give the loop-stall watchdog a macOS reader so its paging deferral can fire on macOS the way #20504 made it fire on Linux.

On a macOS service gateway under memory pressure the loop-stall watchdog kills the gateway. The loop thread stops beating while it pages memory back in. The watchdog cannot tell that from a genuinely stuck loop so it ends the process. #20504 fixed this on Linux by deferring the exit while the loop thread keeps taking major page faults. It reads those per-thread from /proc. That reader returns None off Linux so macOS never defers.

Closes #21139.

Why it matters

The deferral is the difference between riding out a transient paging stall and losing every in-flight chat, cron and channel to a kill the loop would have recovered from. macOS gets none of that protection today. The maintainer confirmed on #20504 that the macOS service path is worth a reader.

Not a goal

The desktop app launch keeps its current behavior. Its budget sits below stall_after and its own liveness probe restarts a silent gateway within a minute so this change defers nothing there. Changing that policy is out of scope.

What changed (motivation to approach to change)

The macOS signal is weaker than the Linux one so the reader is built to fail safe.

  • platform_compat.proc_pageins_for_pid reads the process ri_pageins count through the existing proc_pid_rusage plumbing (offset 48 in rusage_info_v2). None off macOS or on any unreadable call.
  • loop_watchdog.read_loop_pageins_under_pressure plugs into the same loop_major_faults hook fix(watchdog): keep the gateway up while its loop thread is paging #20504 added so the deferral logic does not change. It returns the page-in count only while the kernel reports memory pressure at or above WARN and None otherwise. None is already the deferral's "no reading, reset the baseline, defer nothing" path so a stall the compressor serves (pressure high, disk page-ins flat) never defers and exits on the normal budget. The signal only ever extends the budget on real page-in progress.
  • The macOS grace cap is 120 s, shorter than Linux's 600 s. It is passed as a per-instance paging_grace_max. The macOS page-in count is process-wide, not per-thread. Another thread's paging could then defer a wedged loop. The shorter cap bounds that.
  • The macOS gateway is constructed with this reader and cap in server_runtime/heartbeat.py.

A standalone read-only sampler (kiro_crew.diag.macos_pagein_trace) records ri_pageins, the pressure level and phys_footprint to a durable JSONL log so the signal can be validated against a real stall even when the gateway dies.

Backwards compatibility

No behavior change off macOS. The Linux reader, the module PAGING_GRACE_MAX_SECS and the deferral logic are untouched. The macOS path was inert before (the reader returned None) and now reads a real signal.

Tests

15 new tests in test/test_loop_watchdog.py, all deterministic with injected pressure and page-in readings. No test induces real memory pressure. They cover: pressure with rising page-ins defers, rising page-ins without pressure does not, pressure with flat page-ins exits on budget (the compressor case), pressure clearing mid-episode does not fake a rise, unreadable signal does not defer, the foreground budget defers nothing, and the shorter macOS cap. A read-only ABI test reads this process's real ri_pageins at offset 48. Full suite: 66 passed, 1 skipped (the Linux-only procfs test).

Manual verification

Ran the sampler against this macOS host. It recorded WARN pressure with ri_pageins flat at 0, the compressor-only regime where the design correctly declines to defer.

Related Issues

Closes #21139

Follow-up to #20504.

Checklist

  • At most two commits with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from 9d71f3c to a32a7aa Compare October 18, 2026 14:18
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 140fed50a4bfd500f9ab8f4e72312b856823bbe4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 140fed50a4bfd500f9ab8f4e72312b856823bbe4 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

No design problem blocks this; the reader is fail-safe and only ever extends the exit deadline on a counter that is rising under pressure, capped at 120 s.

Notes

  • Phantom claim: the description says "A standalone read-only sampler (kiro_crew.diag.macos_pagein_trace) records ri_pageins…", but no such module is in the diff. The Manual verification section relies on that sampler. Either add the module or drop both claims.
  • The log message is chosen with self._paging_grace_max == PAGING_GRACE_MAX_SECS_MACOS, so it depends on the cap's value instead of the signal. If an operator or test changes the macOS cap, the Linux text comes back, which says "its thread is paging … not the loop wedged". That is exactly the claim the macOS signal cannot back. Pass the message, or a signal kind, alongside the reader in heartbeat.py instead.

[DESIGN-REVIEWED] 140fed5

Description read

[DESCRIPTION-READ] 103bdb3211eb6a5931c7ae34d8e183cacfb7dab42048529e21f4deb5f73dc616

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 140fed50a4bfd500f9ab8f4e72312b856823bbe4 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The new pressure gate re-implements an existing one-line predicate, and the description promises a sampler that the diff does not include.

Not checked: whether macOS ri_pageins rises during swap-backed (compressor swapfile) page-ins on a real stalled gateway.

Notes

  • Item 1 — duplicate: level is None or level < warn_level re-implements resource_status.pressure_level_held ("The one definition of "held""), 1 existing site. The new warn_level kwarg has 0 non-test consumers: every caller passes the default. Subtraction: delete the warn_level parameter and call pressure_level_held(pressure_level()).
  • Item 6 — undeclared/framing: the description says "A standalone read-only sampler (kiro_crew.diag.macos_pagein_trace) records…". No such file is in the diff or on the base (glob diag/macos_pagein* = 0). Subtraction: delete that paragraph from ## What changed.

What this change ships

Inventory (6 items) — 4 justified

Intent: keep a macOS service gateway from being killed while it pages memory back in under memory pressure. This is an ADDITION (a fix for macOS users).

  1. A macOS managed gateway now defers its stall exit while process page-ins rise under kernel memory pressure — duplicate of resource_status.pressure_level_held (gate only)
  2. On macOS the deferral is capped at 120 s instead of 600 s — justified
  3. The watchdog takes a per-instance grace cap (paging_grace_max) — justified
  4. On macOS the deferral WARNING names process page-ins, not the loop thread — justified
  5. New platform_compat.proc_pageins_for_pid reads ri_pageins — justified
  6. The description lists a diag.macos_pagein_trace sampler that the diff does not ship — undeclared

[FIRST-PRINCIPLES-REVIEWED] 140fed5

Description read

[DESCRIPTION-READ] 6570b16bc8782019de3475785f8b9e9ae2fbdc0bad9c095b74d8338c858b46fb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 140fed50a4bfd500f9ab8f4e72312b856823bbe4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 140fed5

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 140fed50a4bfd500f9ab8f4e72312b856823bbe4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 140fed5

@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from a32a7aa to 2eee30a Compare October 18, 2026 16:02
@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

Addressed on the new head.

Opus 5.5 (both advisory findings fixed):

  • loop_watchdog.py warning: the macOS deferral now logs "the process is paging (N page-ins under memory pressure)" through a paging_log_message the watchdog takes per instance. It no longer names the loop thread or claims the loop is not wedged, since the macOS signal is process-wide.
  • slack-gateway.md: the LoopStallWatchdog paragraph now documents the macOS reader, its memory-pressure gate, the process-wide ri_pageins signal, and the shorter PAGING_GRACE_MAX_SECS_MACOS (120s) cap.

Design Review (CONCERNS, the signal is unproven): agreed and stated as-is. The body says plainly the page-in counter has not been confirmed to rise in a real stall window, and the fork's own run showed it flat at 0. The gate is built to fail safe for exactly this: under pressure with flat page-ins the count never rises, so nothing defers and the process exits on the normal budget. The standalone sampler exists to capture a real stall and validate the signal before anyone relies on it. The reader stays inert, never harmful, until then.

First Principles (CONCERNS, sampler scope / overlap with diag/recorder.py): the sampler is deliberate, not overlap. diag/recorder.py runs inside the gateway and dies with it under the memory-pressure kill this PR is about, so it cannot capture the window around the kill. The sampler is a separate read-only process precisely so the page-in and pressure trace survives the gateway's death and is there on the next launch. That is the only way to validate the macOS signal this PR introduces.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from 2eee30a to 55e7b53 Compare October 18, 2026 18:37
@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

New head 55e7b539 addresses the settled matrix.

Backend Lint & Type Check (3.12) — fixed. The failure was the black ratchet: my edits made src/kiro_crew/dashboard/loop_watchdog.py (previously listed in .github/black-baseline.txt) black-clean, and the gate requires a now-clean file to be removed from the baseline. Removed that one line; the gate passes locally with the pinned black 26.3.1.

Backend Tests (Windows) shards 1 and 6 — not mine, known flake. Both failed with subprocess.TimeoutExpired on the pytest --collect-only step for test/test_ci_ipv6_routing.py. This PR's diff does not touch that file or any IPv6/collection code. It is the documented flake in #21943 (collection subprocess times out at 60s on GitHub-hosted Windows); I added this run's evidence there.

Coverage Gate — downstream of the above. It failed closed because a backend-test shard was cancelled (backend-test=cancelled -- failing closed), i.e. the Windows flake, not an independent coverage regression. It should clear once the Windows shards pass on a re-run.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Oct 18, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from 55e7b53 to 08e2881 Compare October 18, 2026 22:16
@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

Rebased onto current main (new head 08e28817).

The branch had fallen ~98 commits behind, and #21000 ("carve platform_compat into owner modules") landed in that window. It moved the macOS proc_pid_rusage plumbing (the phys_footprint reader, the rusage_info_v2 constants, _darwin_libproc_handle) out of platform_compat.py into the new platform_compat_proctable.py, which is what the stale-base lint failure and the DIRTY merge state were about.

Resolution: the ri_pageins reader now lives in platform_compat_proctable.py beside the phys_footprint reader it is modeled on, reusing the same libproc handle and rusage_info_v2 constants. platform_compat.py keeps only the one-line re-export, and the facade mirror surfaces proc_pageins_for_pid from kiro_crew.platform_compat exactly as it does proc_phys_footprint_bytes_for_pid, so the watchdog's import is unchanged. memory_pressure_level stayed in platform_compat.py. Verified: the facade import works, 66 passed / 1 skipped, and the black gate passes on the rebased tree.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 18, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from 2d4ddb5 to a4912ed Compare October 26, 2026 07:17
@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

Rebased onto fresh main. The prior head went red on backend shard 5 with a test that is not in this diff. The same test failed on Linux shard 5 and Windows shard 5, and it passes in isolation. The shard log shows other tests leaking subprocess pool work into that shard. Deleting the sampler test in this PR reshuffled the shard split and put those tests together. This rebase changes the split again. CI re-running.

@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

Filed the shard-5 test-isolation defect separately as #24365. It is a pre-existing leak in test_cron_reaper_failed_kill and test_crew_log_types, which leave subprocess-pool work running into test_memory_files_substitution when they land on the same shard. It is not in this diff. Removing a test here reshuffled the pytest-split layout and put them together, which is how it surfaced. The fix belongs in those leaking tests, so it lives in #24365, not this PR.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 26, 2026
@tribhuwan-kandpal

Copy link
Copy Markdown
Contributor Author

This PR has been green on every CI lane and all review lanes pass. It closes #21139. It is ready for review whenever someone has a moment. Thanks.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 29, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from a4912ed to e9e80e4 Compare October 31, 2026 01:27
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 31, 2026
@tribhuwan-kandpal
tribhuwan-kandpal force-pushed the feat/macos-loop-stall-pagein-reader branch from e9e80e4 to 0b78937 Compare November 6, 2026 09:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 6, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 8, 2026
Tribhuwan Kandpal and others added 2 commits November 8, 2026 09:26
under memory pressure, reading per-thread major faults from Linux
/proc. That reader returns None off Linux, so macOS never defers.

Add a macOS reader behind the same loop_major_faults hook, so the
deferral logic does not change. It returns the process ri_pageins
count (via proc_pid_rusage) only while the kernel reports memory
pressure at or above WARN, and None otherwise. None is already the
deferral's no-reading path, so a stall the compressor serves (pressure
high, disk pageins flat) never defers and exits on the normal budget.
The macOS grace cap is 120s, shorter than Linux's 600s, because the
page-in count is process-wide rather than per-thread.

Closes kirodotdev#21139.
The Flake Pattern Ratchet counts a literal on a grace-named kwarg as a
short wall-clock wait (K19). Both new sites feed the hand-stepped fake
clock, so carry flake-ok with that reason.
@iamwhatever
iamwhatever force-pushed the feat/macos-loop-stall-pagein-reader branch from 6f7fb59 to 140fed5 Compare November 8, 2026 09:27
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Nov 8, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at 140fed5.

What I checked:

  • ri_pageins offset: rusage_info_v2 is ri_uuid[16], then user time, system time, pkg idle wakeups, interrupt wakeups, pageins. So 16 + 48 = 48 is right, and it sits consistently next to the existing phys_footprint offset (16 + 78).
  • The pressure gate returns None rather than 0 when not under pressure. The deferral already reads None as "reset the baseline", so a pressured poll after an unpressured one cannot fake a rise. test_macos_pressure_clearing_mid_episode_does_not_fake_a_rise covers this.
  • A compressor-served stall (pressure high, page-ins flat) still exits on the normal budget, and the macOS cap (120s) is shorter than the Linux one (600s) because the signal is process-wide.
  • The Linux path is unchanged: the default constructor still uses read_thread_major_faults and PAGING_GRACE_MAX_SECS.
  • No regex, and nothing that newly refuses. The change only extends the exit budget on macOS, and only under real memory pressure.

Non-blocking nit: the log message is chosen by comparing paging_grace_max == PAGING_GRACE_MAX_SECS_MACOS. A future Linux caller that happens to pass 120 would get the macOS wording. Passing the message (or a flag) explicitly from heartbeat.py would avoid that, but it is fine as is.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS: loop-stall paging deferral has no reader (follow-up to #20504)

3 participants