Skip to content

feat(memory): stop the heaviest non-gateway tool tree at critical host memory - #39270

Open
iamwhatever wants to merge 1 commit into
mainfrom
feat/host-memory-guard-fast
Open

iamwhatever wants to merge 1 commit into
mainfrom
feat/host-memory-guard-fast

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: When host memory is critical, Crew frees memory by stopping its own heaviest non-gateway tool process tree (tests first), never the gateway, and tells the owning chat and the conductor that dispatched it.

On the owner's dev host (Linux, 61 GB, no swap, cgroup v1), heavy test runs pushed the whole host into page-cache thrash twice; the gateway went silent for ~35 minutes and the host rebooted. Today nothing acts on a busy session's running test tree at critical memory: the notifier only toasts, the RSS ceiling only recycles idle sessions, and the kernel OOM killer fires late or never on a swapless host.

Why it matters

A runaway pytest -n in one worker chat can take down the gateway and every other chat with it. Stopping that one tree early keeps the host, the gateway and the other workers alive, and the notice plus ledger report keep the stop from being a silent failure.

Not a goal

  • The conductor (slow) layer: disk posture, the posture-change wake, the per-worker view in resource_status, the conductor rule text. That is PR B of the RFC.
  • Stopping trees on macOS or Windows.
  • Node test runners (vitest, jest) as tier 1; they fall to tier 2 by size.
  • Any change to the idle RSS recycle, the spawn floor or the create 409.
  • A new config key, record file, daemon or sampler.

What changed (motivation → approach → change)

This is PR A (fast layer) of docs/request-for-change/rfc-host-memory-guard.md §4 and §6, with the §8 tests 1-11. The RFC landed on main in #39188; its status line is still draft, and moving it is the maintainer's acceptance, not this PR's.

Part Where What it does
Trigger notifications/resource_pressure.py The guard runs on critical, or on tight with memory PSI full avg10 >= 10. Samples every 5 s (the heartbeat's own beat) while tight or critical, 30 s on ample. agent.resource_critical_gb = 0 turns both arms off.
PSI reader platform_compat_metrics.py, listed in platform_compat.py's manifest linux_memory_psi_full_avg10(); None when the file is missing.
Victim choice and stop session_memory_guard.py (new) One single-flight pass; the process view and the walk run on the maintenance executor, never the loop: walks each session runtime's descendants and detached leftovers, never entering a protected pid (runtimes, pool spares, companions, in-flight launches, the gateway) or an MCP root, and walking through the harness (kiro-cli v3 runs tools under its KAS node) without counting it in a tree. Picks the largest topmost-pytest tree, else the largest tree rooted at a shell whose parent is the agent process (runtime, harness, or a harness's only child), so a shell an npx launcher runs for an MCP server is never a root; never under 512 MB, with a 60 s per-session cool-down. A runtime is walked only while its pid still carries the start identity its client recorded at spawn, and only its topmost leftovers seed the walk. Each pid's start identity is bound to the parent edge the walk saw (a pid reused in between is dropped). Linux stops it leaf first through terminate_pids_pinned (TERM, KILL after 5 s); a failed stop is retried on the next sample and told once.
Tell dashboard/memory_guard_events.py (new), dashboard/memory_guard_report.py (new), work_ledger_writes.py (re-exported from src/kiro_crew/work_ledger.py), seam on src/kiro_crew/session.py One Memory guard: ... log line; one notice in the owning chat (same delivery as the leftover-memory notice), naming the agent memory ceiling instead of host memory when that is the reading; for a bound worker one question report (no reason; artifacts, pr, discussion kept; an item whose report is done or carries a reason is left alone) recorded in the conductor's crew log like a worker's own report (same unit fallbacks), and undone if the record does not land.

Per OS:

Host Before After
Linux, no cgroup (v1) nothing acts until the kernel OOM killer stops the heaviest test tree at critical or PSI stall; tells chat and conductor
Linux, cgroup v2 kernel kills inside one agent scope at 65% RAM same, plus the host-level guard
macOS toasts only picks the tree and tells the chat it cannot stop it here; no stop, no ledger write
Windows toasts only one log line per gateway run; no argv reader for a process tree, so no tree is named

Docs: a session_memory_guard row in docs/system-specs/modules/runtime-ownership.md and in docs/architecture/resource-protection.md, and dashboard/memory_guard_report.py in the work-ledger row of docs/feature-map/README.md. The guard takes each session's runtime pid from SessionCleanup.session_runtime_pids in session_cleanup.py, the one reader the RSS ceiling already used, so no new pid reader is added.

A runtime two sessions share (the chat-runtime-sharing preview, off by default) cannot say which of them started a tree. Such a stop is logged with tenants=<n>, every sharing chat gets a notice that says so, and no item gets a report, so no other worker's status is replaced.

The chat notice is English text in a compaction row with notice: memory_guard, as leftover_memory is today; the RFC's i18n keys are not added because the compaction card renders the row's own text.

Backwards compatibility

Compatible: no config key, persisted format or API changes; the new report uses the existing report event and question status.

Timing change: the resource notifier samples every 5 s while the posture is tight or critical (30 s on ample, as before). Each sample is the same few /proc and /sys reads, off the loop; the notes it pushes keep their episode rules, so no extra toast fires.

Removes nothing: the change only adds a guard path; every existing guard keeps its behavior.

Tests

  • test/test_session_memory_guard.py: tier 1 beats a larger tier 2 and xdist is one tree; protected pids, harness, MCP roots and outside processes are never picked; a non-shell tier-2 root and a tree under 512 MB are never picked; a detached leftover tree is in reach; cool-down; real processes on Linux: TERM, KILL for a SIGTERM-ignoring survivor, a recycled pid and a protected pid are not signalled; a pass stops one tree off the loop with every log field, one notice and the ledger summary; macOS and Windows arms stop nothing and file nothing; request is single-flight.
  • test/test_resource_pressure_guard.py: the trigger table (critical, tight+PSI, tight under 10, missing PSI, PSI on ample, critical_gb = 0), 5 s/30 s sampling, the PSI parser.
  • test/test_memory_guard_ledger_report.py: one question report with pointers kept and the report stamp moved; closed item refused; the board's copy is recorded; an unbound session gets no write; an unrecorded write is undone.

Proof: not provable -- other: new feature, no prior behavior to revert to

Manual verification

Exit check on this host with the real notifier, guard, /proc walk, pinned stop, notice wiring and ledger store plus crew-log record; critical was simulated by forcing the posture from the 3rd sample (no memory was exhausted, no model turn ran). A session-leader runtime ran bash -c "python -m pytest -n 2" with two xdist-bootstrap workers holding 1.2 GB each:

Memory guard: stopped tier=test session=dashboard:chat-2-worker tenants=1 root_pid=28346 start=33629462 cmd=pytest tree_rss_mb=2502 procs=3 available_gb=43.2 psi_full_avg10=0.00 trigger=critical signal=TERM freed=yes
gateway_pid 27795 -> 27795, runtime alive, tree 2521 MB -> 14 MB, 0.9 s from the critical sample to stopped
chat notice: "Host memory was critically low (~43.2 GB free), so Crew stopped `pytest` (2.4 GB, 3 processes) that this chat started. ..."
item: status=question reason=None artifacts={'branch': 'feat/x'}; events create, bind, report/progress, report/question; report stamp moved

A read-only scan_host over a live dashboard session (runtime shim -> aim sandbox -> launcher -> kiro-cli -> KAS node -> zsh, with eight MCP servers beside the shell under KAS) found exactly the shell tool's tree and a python -m pytest -n 2 started in it (test 16029 pytest, tool 16028 python), and no MCP server.

Known limit: a shell that execs a single non-test command (zsh -c "npm run build") leaves no shell above it, so that tree is not tier 2; a pytest started that way is still tier 1.

Real-machine verification

Machine: owner dev host, Amazon Linux 2 (kernel 5.10), aarch64, 61 GB RAM, no swap, cgroup v1
Before (main f4754ac, the base this branch was cut from): no guard exists; the same tree keeps running at a critical reading.
After (head 9e224a3): the exit check above.
Cleanup: own computer; the harness killed its runtime and removed nothing else.

Related Issues

Part of #22681

Checklist

  • A Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 9e224a39d89857ea39815ba6bd9840818891f2a4 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of 9e224a39d89857ea39815ba6bd9840818891f2a4 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

product-shape change without accepted RFC: on every Linux gateway, Crew now kills a user's running tool process tree by default, and the RFC this rests on is draft on base.

Notes

  • Item 5 — rides along: "samples every 5 s while tight or critical" also changes when the existing notifier's toasts and episode notes run, not just the guard. Subtraction: run the 5 s cadence only when guard_trigger could fire, and keep 30 s for the notes path.

What this change ships

Inventory (8 items) — 6 justified

Intent: keep the host and the gateway alive when one chat's test run eats memory. This is an ADDITION.

  1. On Linux, at critical memory or a PSI stall, Crew stops one tool tree (tests first), and this is on by default — justified
  2. The owning chat gets a "Memory guard" notice saying what was stopped — justified
  3. A bound worker's item gets a question report in the conductor's ledger — justified
  4. Every stop writes one Memory guard: log line — justified
  5. The notifier samples every 5 s while memory is tight or critical (was 30 s) — rides along, timing change
  6. On macOS the chat is told about the tree but nothing is stopped; Windows logs once — justified
  7. Setting agent.resource_critical_gb = 0 now also turns off killing trees — undeclared
  8. New public linux_memory_psi_full_avg10() in platform_compat_metrics — justified

Blockers

product-shape change without accepted RFC: default-on automatic stop of user tool trees — product shape
Trigger: item 1. wire_memory_guard(self) is called unconditionally from session.py, and guard_trigger fires whenever agent.resource_critical_gb is above 0 (default 2 GB). So every Linux user gets automatic SIGTERM/SIGKILL of their running pytest or shell trees without opting in. The description says: "its status line is still draft, and moving it is the maintainer's acceptance". The base list reads docs/request-for-change/rfc-host-memory-guard.md:status: draft.
Outcome: nobody has recorded the decision that Crew may, by default, kill processes a user's session started.
Clears when: an RFC covering a default-on host memory guard that stops session tool trees is merged with a non-draft status on base (then push or rebase), or a maintainer posts /ai-review override first-principles 9e224a39d89857ea39815ba6bd9840818891f2a4 product-shape: <decision>.

[FIRST-PRINCIPLES-REVIEWED] 9e224a3

Description read

[DESCRIPTION-READ] 4e93f8777c86512262e6c5fd180045655bbcedd239c5e1f6bba9aede2a779feb

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9e224a39d89857ea39815ba6bd9840818891f2a4 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the guard is Linux-only, single-flight, never stops a runtime or the gateway, and was verified on the platform where it stops processes.

Notes

  • agent.resource_critical_gb = 0 is the only way to turn off the automatic stop, and it also turns off the critical posture's notes and floor. An operator who wants the toasts but not the stop has no setting for that. A dedicated off switch is a follow-up if that comes up.
  • The guard is wired through wire_session_recycle_callback, so it is only on where that function is called (server.py, api_entrypoint.py). A third gateway entry point that skipped that call would get no guard, and nothing would say so.

[DESIGN-REVIEWED] 9e224a3

Description read

[DESCRIPTION-READ] 3282c9f29fc947aea8e12e3b7c472b2b84688a0d5c996b0f2af9b4924b63a30b

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 3e3a1f03e24857f9186be1b5b4cfdce033984172 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 3e3a1f0

Verdict parsed from the review's SHA-scoped output markers for commit 3e3a1f03e24857f9186be1b5b4cfdce033984172.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 3e3a1f03e24857f9186be1b5b4cfdce033984172 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — 🔴 changes requested (blocking)

GPT 6.1 found at least one blocking issue that must be resolved before merging 9e224a39d89857ea39815ba6bd9840818891f2a4. Adjudication produced no [GPT-ADJUDICATED] marker for 9e224a3: the adjudication stage FAILED TO RUN (its raw output is in the Adjudication details block -- typically an API or setup error), so no proportionality review took place and GPT's blocking verdict stands unreviewed.

This comment is updated in place on each push.

BLOCKING -- src/kiro_crew/session_memory_guard.py:272 -- Bundled KAS shell-tool trees never qualify for tier 2

if up is None or up == runtime or up not in kids or not is_harness(up):
Trigger: Bundled KAS runs a shell-based build exceeding 512 MB under Crew’s Linux namespace sandbox. The launcher forks the Node engine with --experimental-wasm-modules; critical pressure reaches _pass → scan → hosts_tools, but is_harness does not recognize that engine and its parent is the runtime.
Outcome: With no pytest candidate, the guard selects nothing and leaves the heavy build running throughout critical pressure.
Anchor: defect
Fix: Recognize the bundled engine’s actual launch identity and admit its direct shell-tool children, without admitting arbitrary Node processes.
Sites: src/kiro_crew/session_memory_guard.py:225 — the harness classifier used by hosts_tools.

BLOCKING -- src/kiro_crew/dashboard/memory_guard_events.py:39 -- Unsurfaced channel conversations lose the stop notice

slot = self.get_slot(slot_key) if slot_key else None
Trigger: A native Slack incognito/temporary conversation runs a heavy pytest tree. Its session participates in _pass, but channel_slots excludes it from dashboard slots; after the stop, _on_memory_guard_stop → dashboard_slot_key returns an empty string.
Outcome: The append branch is skipped and no channel delivery runs, so the owning conversation receives no memory-guard explanation. An unbound session also gets no ledger report, and the handled tree’s notice is not retried.
Anchor: defect
Fix: Deliver the notice through the existing governed channel-delivery path when the owning session has no dashboard slot.
Sites: none

[BLOCK-MERGE] 9e224a3
[GPT-REVIEWED] 9e224a3
False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9e224a39d89857ea39815ba6bd9840818891f2a4 false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 2592ec6 to f873bb3 Compare November 18, 2026 21:52
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head f873bb3 (rebased on main cc14de1) answers the round on 2592ec6:

  • Opus 5.5 BLOCKING (session_memory_guard.py:379, view built on the loop): fixed. scan_host builds host_view() and runs scan in one maintenance-executor job; the loop only collects pids and awaits it. test_session_memory_guard.py now asserts the view is built off the main thread.
  • Backend Lint (isort on session.py): fixed; the guard state is imported inside SessionManager.__init__, so the top import block is unchanged.
  • Static Ratchet (pid reader): the guard reads each session's runtime pid once, as _rss_threshold_check does, to name the runtime whose tree it walks; _BASELINE records session_memory_guard.py: (1, 0).
  • SAST (test-autouse-shared-monkeypatch): fixed; the autouse fixture uses _floor_monkeypatch.
  • Source Gates (feature map): handlers/work_ledger_guard.py named in the work-ledger row.
  • Design notes: a session_memory_guard row added to runtime-ownership.md and resource-protection.md. The RFC status line is left at draft: moving it records a maintainer's acceptance, which this PR cannot do. The resource_critical_gb = 0 coupling is the RFC's chosen off switch (§4.1, no new key); the help-text note is left to a follow-up so this PR does not touch the config baseline.
  • First Principles note (undeclared sampling): declared under Backwards compatibility. The BLOCK itself clears only when a maintainer records the RFC as accepted on main or posts the product-shape override; I have not posted one.
  • Exit check re-run on f873bb3: same result (tree 2520 MB -> 14 MB 0.9 s after the critical sample, gateway pid unchanged, notice + question report).

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from f873bb3 to d17cb5f Compare November 18, 2026 22:12
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head d17cb5f answers the round on f873bb3:

  • Opus BLOCKING + GPT F1 (shared runtime credited to a guessed session; new pid reader in the ratchet): fixed. The guard no longer reads pids itself: it takes (key, runtime pid, session) from SessionCleanup.session_runtime_pids, the loop the RSS ceiling already ran (now shared by both), so the ratchet row is removed and the count is unchanged. A runtime with more than one session is stopped and logged with tenants=<n>; every sharing chat gets a notice saying Crew cannot tell which of them started the tree, and no item gets a report. Test: test_a_shared_runtime_tells_every_tenant_and_files_no_report.
  • Backend Ratchets (agent-sdk boundary, session_memory_guard.py:122): fixed; tree RSS reads session_pid_rss._process_mb instead of the ACP layer.
  • GPT F3 (prewarmed conductor unit loses the report): fixed; the guard report lands with the worker report's fallbacks (live conductor unit, then newest_work_unit, then the worker's unit when the board has no log). Test: test_a_prewarmed_conductor_unit_falls_back_to_the_boards_newest_log.
  • GPT F4 (inherited signal state): fixed; _spawn runs inside spawn_test_helpers.unblocked_signals(), the escalation child still sets its own SIGTERM ignore.
  • GPT F2 (channel-only chat gets no notice): deferred, as the adjudication downgraded it. The RFC (§4.4) puts the notice on the on_leftover_memory seam, which also reaches dashboard chats only; the log line and the bound worker's report still land for a channel session.
  • First Principles: waits on the RFC status PR; no override from me.
  • Exit check re-run on d17cb5f: same result (tenants=1, 2521 MB -> 14 MB 0.9 s after the critical sample, gateway pid unchanged, notice + question report).

@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from d17cb5f to 2d1cdaf Compare November 18, 2026 22:29
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 2d1cdaf answers the round on d17cb5f:

  • Opus BLOCKING (session_cleanup.py over the 2,000-line cap): fixed; session_runtime_pids is three lines and the file is 1,999 lines. check_file_line_cap.py: 0 failures.
  • Opus BLOCKING (shared runtime + failed stop or macOS credits one chat): fixed; shared_notice covers stopped, could-not-stop and cannot-stop-here, and never says "this chat started". Test: test_a_shared_runtime_on_macos_credits_no_single_chat.
  • Analyze (actions), Dependency License Gate, Publish readiness signal: GitHub API rate limit on d17cb5f (API rate limit exceeded for installation); they rerun on this head.
  • Exit check re-run on 2d1cdaf: same result.

@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 2d1cdaf to 12f9b13 Compare November 18, 2026 22:46
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 12f9b13 answers the round on 2d1cdaf:

  • Opus BLOCKING (session_memory_guard.py:233, a shell an npx launcher runs for an MCP server became a tier-2 root): fixed. A shell is a tier-2 root only when its parent is the agent process: the runtime, a harness, or a harness's only child (kiro-cli v3's KAS node). The npx shape (host -> node npx-cli.js -> sh -> server) is never a root, under KAS or beside the shell tool directly under kiro-cli. Tests: test_a_shell_an_npx_launcher_runs_beside_the_shell_tool_is_never_a_root, test_the_kiro_cli_v3_chain_reaches_the_shell_tool_below_the_harness.
  • Found while checking that fix on a live session: the walk used to stop at the kiro-cli harness, and on kiro-cli v3 every tool runs below it, so the guard reached nothing in a real session. The walk now goes through a harness and never counts one in a tree (a tree holding a harness is skipped). A read-only scan_host over a live dashboard session found the shell tool's tree and a python -m pytest -n 2 started in it, and none of the eight MCP servers beside it.
  • Exit check re-run on 12f9b13: same result.

@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 12f9b13 to 91a7c0e Compare November 18, 2026 23:05
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 91a7c0e answers the round on 12f9b13:

  • Opus BLOCKING (notices say "Host memory" when the agent slice ceiling binds): fixed. chat_notice, shared_notice and ledger_summary take the ResourceStatus and word the figure through memory_subject / memory_reading(), so a slice-bound reading reads "Agent memory was critically low (~1.4 GB left under the 40 GB agent memory ceiling, ~20 GB free on the host)". Test: test_an_agent_ceiling_reading_is_named_as_such.
  • Analyze (javascript-typescript): GitHub API rate limit (API rate limit exceeded for installation); reruns on this head.
  • Exit check re-run on 91a7c0e: same result.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 91a7c0e to 6a17c71 Compare November 18, 2026 23:39
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 6a17c71 answers the GPT round on 91a7c0e:

  • F1 (identity read after the parentage snapshot): fixed. Each pid's start identity is read, then its parent is re-read (platform_compat.parent_pid); a pid whose parent no longer matches the walk's edge is dropped with everything below it, and a leftover root must still carry the start identity find_session_leftovers listed. terminate_pids_pinned then pins those identities. Tests: test_a_pid_recycled_after_the_walk_is_never_a_target, test_a_leftover_root_whose_identity_changed_is_dropped.
  • F2 (a successful retry after a failed stop is silent): fixed. Only a successful stop (or a host that cannot stop) marks a tree handled; a failed stop is retried each sample and its notice is sent once. Test: test_a_failed_stop_is_told_once_and_a_later_success_still_reports.
  • Backend Tests (3.12, 8), E2E Gateway Boot, Analyze (actions): HTTP 429 / API rate limit on 91a7c0e; rerun on this head.
  • Exit check and a read-only live-session scan re-run on this code: same results.

@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 6a17c71 to 3e3a1f0 Compare November 19, 2026 00:04
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 3e3a1f0 (rebased on main 974172e) answers the round on 6a17c71:

  • Opus BLOCKING (guard report overwrites an open done or a reasoned report): fixed. apply_guard_report writes nothing when the item's report is done or carries a reason; the chat notice and log line still land. Test: test_a_done_or_reasoned_report_is_left_alone (done, blocked/needs_human, question/approval).
  • GPT F1 (runtime pid reused while the session stays registered): fixed. _pass carries each runtime client's spawn-time _start_time; scan_host walks a runtime only while get_process_start_id(pid) still equals it. Test: test_a_runtime_pid_now_naming_another_process_is_not_walked; a live read-only scan with a wrong identity returns [].
  • GPT F2 (detached MCP launcher's leftover children seeded on their own): fixed. Only topmost leftovers seed the walk, so the launcher's wall covers its shell and server. Test: test_a_detached_mcp_launcher_walls_its_leftover_children.
  • GPT F3 (stop-test children leak on assertion failure): fixed with a reap fixture that kills and waits every spawned child.
  • Source Gates (comment-history, "no longer"): reworded; check_comment_history.py against main passes.
  • Await Fast Gate / Coverage Gate: downstream of Source Gates.
  • Exit check re-run: same result.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Nov 19, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
@iamwhatever
iamwhatever force-pushed the feat/host-memory-guard-fast branch from 3e3a1f0 to 9e224a3 Compare November 19, 2026 01:01
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Head 9e224a3 answers the round on 3e3a1f0:

  • GPT F1 (real-interpreter stop cases run in parallel, D8): fixed; TestStop carries one xdist_group, so its children start one at a time.
  • Backend Tests (3.12, 1 / Windows 5, 8) — test_work_ledger_handler_facade: the guard report module matched the work_ledger_* handler family and imported an owner directly. It is now dashboard/memory_guard_report.py and reads the recording helpers through the handlers.work_ledger facade at call time.
  • Backend Tests (3.12, 5 / Windows 8) — test_platform_compat_refactor_facade: _LINUX_MEMORY_PSI_PATH and linux_memory_psi_full_avg10 added to the metrics owner's moved-name list.
  • Backend Tests (3.12, 7) — TestStop (terminate_pids_pinned returned 0): the runner cannot pin processes with pidfd; TestStop now skips where _process_group_supervisor.can_reap() is false, as test_real_shell_and_child_are_terminated_without_killing_the_runtime does. The stop is still proven on this Linux host (exit check).
  • Local: 1,074 related tests pass; feature-map, comment-history, line-cap, agent-sdk boundary, isort and semgrep gates pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: checking Automated validation is still running

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant