Skip to content

fix(acp): judge the macOS runtime ceilings by phys_footprint, not ps RSS - #15760

Merged
bolichen97 merged 1 commit into
mainfrom
fix/macos-bg-runtime-phys-footprint
Oct 1, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/macos-bg-runtime-phys-footprint

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: on macOS, the background runtime's memory ceiling judges the memory the OS actually charges the runtime, so a runtime that has grown past it is recycled.

On macOS the 500 MiB ceiling (AcpRuntime._is_stale) never fires (#15757). From one 16 GB Mac:

  • the gateway pid read 124 MB through ps RSS against 1983 MB of phys_footprint (16x);
  • two idle background runtimes held 462 MB of footprint for hours;
  • there were nine jetsam kills in eight days.

Why it matters

The ceiling is the only bound on a background runtime that keeps getting reused, and on macOS it is effectively disabled. Memory the runtime holds compressed or swapped is invisible to it, and that is exactly the memory jetsam counts when it kills processes on a full Mac. The Sessions panel's macOS memory figure, which reads the same tree sum, under-reports the same way.

Not a goal

  • Reaping a zero-session background runtime after a fixed idle time, regardless of size. test_bg_runtime_idle_recycle.py::test_a_fresh_idle_runtime_is_left_alone pins the opposite ("idle is not a reason to recycle, staleness is"). Changing that is a maintainer decision, recorded on macOS: background runtime memory ceiling reads ps RSS, which omits compressed pages, so it never fires #15757.
  • Changing the 500 MiB / 6 h ceilings or their defaults.
  • The session watchdog's per-session RSS recycle, which is Linux-only (get_session_rss_mb returns 0 on macOS).
  • Linux and Windows measurement, which is unchanged.

What changed (motivation → approach → change)

Symptom: a macOS runtime holding gigabytes is never judged stale.

Root cause: on darwin, _get_rss_tree_mb in src/kiro_crew/acp/runtime_process_tree.py summed ps -Ao pid=,ppid=,rss=, and _get_rss_mb read ps -o rss=.

  • ps RSS counts only resident pages. macOS compresses and swaps idle memory aggressively, which is what an idle grown runtime mostly consists of.
  • jetsam and Activity Monitor's "Memory" column read phys_footprint, which includes those pages.

Change:

  • Footprint reader. src/kiro_crew/platform_compat.py gains proc_phys_footprint_bytes_for_pid(pid), which reads ri_phys_footprint via proc_pid_rusage(pid, RUSAGE_INFO_V2). It needs no subprocess and no entitlement for a same-uid pid, and it answers None off macOS or when unreadable. The struct layout (16-byte uuid, then 18 uint64s, footprint the 8th) is pinned by a test. proc_pid_rusage is configured separately on the shared libproc handle, so a libproc without it still serves every proc_pidinfo probe.
  • Tree sum. On macOS the tree sum measures each pid by its footprint, falling back to that pid's ps RSS only when the footprint cannot be read. The ps snapshot still provides the tree shape in one memoized walk, and the depth bound is unchanged. _get_rss_mb prefers the footprint too.
  • Specs. docs/system-specs/common/platform-compat.md gets a new helper row; docs/system-specs/modules/session.md documents the staleness measurement.

Backwards compatibility

Compatible: same function signatures and units (MiB). On macOS the figure is now the footprint, so the ceiling can fire where it could not before. That is the intended fix, and a runtime with sessions is still only parked, never killed under a co-tenant.

Tests

test/test_macos_phys_footprint.py (13 tests plus 3 darwin-only):

  • Reader:
    • reads the footprint field and not the resident one;
    • a failed call, and a missing libproc, both read as None;
    • a libproc without proc_pid_rusage still serves the shared handle;
    • the reader is macOS-only;
    • the constants restate the header's field order;
    • TestTheRealDarwinAbi (darwin-only) calls the REAL proc_pid_rusage on its own pid. It checks that the footprint is a positive int, that dirtying 256 MiB grows it by at least 192 MiB (which no neighbouring field does), and that a nonexistent pid reads as None. It skips on Linux, and runs on this PR's ci:macos run and in the nightly macOS suite.
  • Tree sum:
    • it sums footprints;
    • a pid with no readable footprint falls back to its RSS;
    • with no footprint at all it equals the old RSS sum;
    • the depth bound still holds;
    • the single-pid read prefers the footprint.
  • Ceiling: 1983 MiB of footprint behind 120 MiB of RSS makes _is_stale() return "rss", and a small footprint does not.

Existing fixtures that pin the ps path now pin the footprint off. Otherwise a real Mac would answer for a same-uid pid the fake table names. The fixtures changed are in test/test_macos_process_table.py, test/test_acp_refactor_process_tree.py and the _POSIX_PS stand-in in test/test_acp_refactor_facade.py.

Regression proof: with runtime_process_tree.py reverted and the tests kept, 5 fail. The related files (session memory, bg-runtime recycle, platform_compat, process tree, facade, spawn audit) pass: 1180 passed.

Manual verification

Not run on a real Mac: this host is Linux, so the libproc call is exercised through a fake library that fills a real ctypes buffer, which is the established pattern for the other _darwin_* probes. A reviewer on macOS can compare python3 -c "from kiro_crew import platform_compat as p, os; print(p.proc_phys_footprint_bytes_for_pid(os.getpid()))" with Activity Monitor's Memory column for that python process.

Gates:

Related Issues

Fixes #15757

Pattern harvest

Rule candidate: review-prompt
Pattern: a macOS memory ceiling that reads ps RSS; on macOS the figure the OS acts on is phys_footprint, and RSS can understate it by an order of magnitude.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team as a code owner October 1, 2026 07:23
@bolichen97
bolichen97 requested a review from CrysisDeu October 1, 2026 07:23
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — ✅ PASS

Design-level review of 1909f1cdedb0767c06241c5c9930af2cd42306d3 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the ci:macos lane actually ran TestTheRealDarwinAbi green on this head — it can skip under pool pressure, and that canary is the only real-kernel check of the hand-transcribed struct offsets.

Suggestions

  • The 500 MiB ceiling was calibrated against a figure that under-read ~16x; after merge, watch recycle cadence on a real Mac (two idle runtimes already measured 462 MB footprint) and re-tune the ceiling in a follow-up if parking churn appears.

[DESIGN-REVIEWED] 1909f1c

Description read

[DESCRIPTION-READ] e86df699f100137d8740f5db40cc8ac57a51bf9a43c5e3a5f0d42b6c5eef879a

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — ✅ PASS

Premise-level review of 1909f1cdedb0767c06241c5c9930af2cd42306d3 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Confirm the ci:macos lane ran TestTheRealDarwinAbi green on this head — the rusage_info_v2 offsets are machine-checked only there, never on Linux CI.

What this change ships

Inventory (6 items) — 6 justified

Intent: FIX — make the macOS background-runtime memory ceiling (#15757) fire on the memory the OS actually charges, by measuring phys_footprint instead of ps RSS.

  1. On macOS, a grown background runtime now trips the 500 MiB staleness ceiling and is recycled — justified
  2. The Sessions panel's macOS memory figure now reads footprint (same tree sum) — justified
  3. New platform_compat.proc_phys_footprint_bytes_for_pid helper — justified
  4. Per-pid fallback to ps RSS when a footprint is unreadable — justified
  5. Single-pid read (_get_rss_mb) also prefers the footprint on macOS — justified
  6. Spec rows in platform-compat.md and session.md updated in the same commit — justified

Counts run: proc_phys_footprint_bytes_for_pid has 2 non-test consumers (runtime_process_tree.py:319,599), one mechanism. Grepped ps -o rss in src/: the only remaining macOS RSS read outside this fix is cli_doctor._gateway_rss_bytes (cli_doctor.py:1264), a diagnostic explicitly labeled RSS — no ceiling judges it, so it is not an unfixed sibling of this defect. Defect provenance: linked issue #15757, measured 16x gap, 5 new tests fail on revert. No pin reversed: the "idle is not staleness" pin is untouched and named in ## Not a goal.

[FIRST-PRINCIPLES-REVIEWED] 1909f1c

Description read

[DESCRIPTION-READ] 5f66c1f8dd3787af7bfa03e053c0f2c1c6ad418a1675c206b699a93bbe19a37d

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Stale verdict notice (2026-10-01 21:06 UTC): a later GPT 5.6 run did not produce a completed verdict for 1909f1cdedb0767c06241c5c9930af2cd42306d3; the verdict below is from an earlier completed run. Inspect the GPT 5.6 Review job logs and re-run the workflow, or have a repository writer comment /ai-review override gpt 1909f1cdedb0767c06241c5c9930af2cd42306d3: <one-sentence reason>.

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 3ec7b6131c3fc9dfc890667c843752a9fb022210 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 3ec7b61

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 3ec7b6131c3fc9dfc890667c843752a9fb022210: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 1909f1cdedb0767c06241c5c9930af2cd42306d3 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1909f1c

Verdict parsed from the review's SHA-scoped output markers for commit 1909f1cdedb0767c06241c5c9930af2cd42306d3.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 1909f1cdedb0767c06241c5c9930af2cd42306d3: <one-sentence reason>

On macOS the background runtime's 500 MiB ceiling never fired. The tree
sum behind AcpRuntime._is_stale read `ps -Ao pid=,ppid=,rss=`, and ps
RSS counts only pages resident right now. macOS compresses and swaps
idle memory aggressively, and an idle runtime that has grown consists
mostly of exactly those pages. One Mac measured the gateway at 124 MB
of RSS against 1983 MB of footprint (16x), with two idle runtimes
holding 462 MB of footprint for hours.

Add platform_compat.proc_phys_footprint_bytes_for_pid. It reads
ri_phys_footprint through proc_pid_rusage(RUSAGE_INFO_V2): no
subprocess, no entitlement for a same-uid pid. It is configured
separately on the shared libproc handle, so a libproc without that
symbol still serves the proc_pidinfo probes. On macOS the tree sum, and
the single-pid read, measure each pid by its footprint, falling back to
that pid's ps RSS only when the footprint cannot be read. The ps
snapshot still provides the tree shape, in one walk.
@bolichen97
bolichen97 force-pushed the fix/macos-bg-runtime-phys-footprint branch from 3ec7b61 to 1909f1c Compare October 1, 2026 08:17
@bolichen97 bolichen97 added the ci:macos Run the on-demand macOS suite against this PR head label Oct 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: darwin-only live canary against the real proc_pid_rusage, run by this PR's ci:macos label and the nightly macOS suite

  • span=8cfd2268296f fixed: the offset "pin" asserted the constants against their own arithmetic

fixed in 1909f1c: TestTheRealDarwinAbi in test/test_macos_phys_footprint.py calls the real proc_phys_footprint_bytes_for_pid, with no monkeypatching, on its own pid:

  • the reading is a positive int;
  • dirtying 256 MiB (one write per 4 KiB page) grows it by at least 192 MiB. No neighbouring counter (wakeups, pageins) moves like that, and a mis-sliced offset reads garbage. Footprint >= RSS is deliberately not asserted, because the footprint excludes clean shared pages;
  • a nonexistent pid reads as None.
    It skips off darwin. This PR now carries ci:macos, so macos-on-demand.yml runs it on macos-15 against this head; the nightly platform-tests.yml runs it after merge. The old layout test is renamed test_the_constants_restate_the_headers_field_order and says plainly that it only pins the transcription. The platform_compat.py comment now points at the canary.

@bolichen97

Copy link
Copy Markdown
Contributor Author

self-added: yes

  • span=2036c6f30da4 fixed: replace or supplement the self-referential layout assertion with a live own-pid canary

fixed in 1909f1c alongside the WATCH above: the live canary is added, and the self-referential test is kept only as a statement of the transcribed field order (renamed and re-docstringed to say exactly that), not as a check of the ABI.

bolichen97 added a commit that referenced this pull request Oct 1, 2026
On macOS the background runtime's 500 MiB ceiling never fired. The tree
sum behind AcpRuntime._is_stale read `ps -Ao pid=,ppid=,rss=`, and ps
RSS counts only pages resident right now. macOS compresses and swaps
idle memory aggressively, and an idle runtime that has grown consists
mostly of exactly those pages. One Mac measured the gateway at 124 MB
of RSS against 1983 MB of footprint (16x), with two idle runtimes
holding 462 MB of footprint for hours.

Add platform_compat.proc_phys_footprint_bytes_for_pid. It reads
ri_phys_footprint through proc_pid_rusage(RUSAGE_INFO_V2): no
subprocess, no entitlement for a same-uid pid. It is configured
separately on the shared libproc handle, so a libproc without that
symbol still serves the proc_pidinfo probes. On macOS the tree sum, and
the single-pid read, measure each pid by its footprint, falling back to
that pid's ps RSS only when the footprint cannot be read. The ps
snapshot still provides the tree shape, in one walk.

Backport of #15760 to release/0.8.0.
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=ae2e97f3ead6 deferred: watch recycle cadence on a real Mac after merge and re-tune the 500 MiB ceiling if parking churn appears

Agreed, and it is a post-merge observation by nature: it needs field cadence that no pre-merge run can produce. Re-tuning _DEFAULT_MAX_RSS_MB is outside this PR's goal, which explicitly does not change the ceilings.
The risk is bounded: a stale verdict only PARKS an idle runtime (never one with a live or initializing session), and the next background call spawns a fresh one. A follow-up issue should be filed if parking shows up in the field; the evidence to look for is the "idle and stale by rss (periodic sweep)" log line.

@bolichen97

Copy link
Copy Markdown
Contributor Author

GPT 5.6 Review on 1909f1c has failed closed four times with "no [GPT-REVIEWED] marker", but the model did finish. Its falsification-pass step log (run 36835414599) ends with No findings. / [GPT-REVIEWED] 1909f1cdedb0767c06241c5c9930af2cd42306d3, and the gate then did not find that text in codex-review-output.md. The earlier attempts failed on API rate limit exceeded for installation. I've re-run it once more; if it repeats, it needs a maintainer look at the lane rather than another re-run.

@bolichen97
bolichen97 enabled auto-merge (squash) October 1, 2026 20:02
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 1909f1c: maintainer decision (bolichen): the GPT lane did not complete on this head (runner/quota), the earlier completed run found nothing blocking; Design/FP/Opus pass and the darwin ABI canary ran green on a real Mac via ci:macos.

@bolichen97
bolichen97 merged commit 8a9b736 into main Oct 1, 2026
107 of 132 checks passed
@bolichen97
bolichen97 deleted the fix/macos-bg-runtime-phys-footprint branch October 1, 2026 21:20
bolichen97 added a commit that referenced this pull request Oct 1, 2026
…RSS (#15766)

On macOS the background runtime's 500 MiB ceiling never fired. The tree
sum behind AcpRuntime._is_stale read `ps -Ao pid=,ppid=,rss=`, and ps
RSS counts only pages resident right now. macOS compresses and swaps
idle memory aggressively, and an idle runtime that has grown consists
mostly of exactly those pages. One Mac measured the gateway at 124 MB
of RSS against 1983 MB of footprint (16x), with two idle runtimes
holding 462 MB of footprint for hours.

Add platform_compat.proc_phys_footprint_bytes_for_pid. It reads
ri_phys_footprint through proc_pid_rusage(RUSAGE_INFO_V2): no
subprocess, no entitlement for a same-uid pid. It is configured
separately on the shared libproc handle, so a libproc without that
symbol still serves the proc_pidinfo probes. On macOS the tree sum, and
the single-pid read, measure each pid by its footprint, falling back to
that pid's ps RSS only when the footprint cannot be read. The ps
snapshot still provides the tree shape, in one walk.

Backport of #15760 to release/0.8.0.
@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the gpt AI finding as false positive, not applicable, or explicitly accepted for 1909f1cdedb0767c06241c5c9930af2cd42306d3.

maintainer decision (bolichen): the GPT lane did not complete on this head (runner/quota), the earlier completed run found nothing blocking; Design/FP/Opus pass and the darwin ABI canary ran green on a real Mac via ci:macos.

This decision applies only to this commit. A new push requires a new judgment.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded, but the following reviewer lane(s) could not be re-run automatically: GPT 5.6. Re-run the lane's latest workflow run for 1909f1cdedb0767c06241c5c9930af2cd42306d3 manually from the Actions tab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:macos Run the on-demand macOS suite against this PR head

Projects

None yet

Development

Successfully merging this pull request may close these issues.

macOS: background runtime memory ceiling reads ps RSS, which omits compressed pages, so it never fires

1 participant