Repository navigation
feat(memory): stop the heaviest non-gateway tool tree at critical host memory - #39270
iamwhatever wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK product-shape change without accepted RFC: on every Linux gateway, Crew now kills a user's running tool process tree by default, and the RFC this rests on is Notes
What this change shipsInventory (8 items) — 6 justifiedIntent: keep the host and the gateway alive when one chat's test run eats memory. This is an ADDITION.
Blockersproduct-shape change without accepted RFC: default-on automatic stop of user tool trees — product shape [FIRST-PRINCIPLES-REVIEWED] 9e224a3 Description read[DESCRIPTION-READ] 4e93f8777c86512262e6c5fd180045655bbcedd239c5e1f6bba9aede2a779feb That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the guard is Linux-only, single-flight, never stops a runtime or the gateway, and was verified on the platform where it stops processes. Notes
[DESIGN-REVIEWED] 9e224a3 Description read[DESCRIPTION-READ] 3282c9f29fc947aea8e12e3b7c472b2b84688a0d5c996b0f2af9b4924b63a30b That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — 🔴 changes requested (blocking)GPT 6.1 found at least one blocking issue that must be resolved before merging This comment is updated in place on each push. BLOCKING -- src/kiro_crew/session_memory_guard.py:272 -- Bundled KAS shell-tool trees never qualify for tier 2
BLOCKING -- src/kiro_crew/dashboard/memory_guard_events.py:39 -- Unsurfaced channel conversations lose the stop notice
[BLOCK-MERGE] 9e224a3 |
2592ec6 to
f873bb3
Compare
|
Head f873bb3 (rebased on main cc14de1) answers the round on 2592ec6:
|
f873bb3 to
d17cb5f
Compare
|
Head d17cb5f answers the round on f873bb3:
|
d17cb5f to
2d1cdaf
Compare
|
Head 2d1cdaf answers the round on d17cb5f:
|
2d1cdaf to
12f9b13
Compare
|
Head 12f9b13 answers the round on 2d1cdaf:
|
12f9b13 to
91a7c0e
Compare
|
Head 91a7c0e answers the round on 12f9b13:
|
91a7c0e to
6a17c71
Compare
|
Head 6a17c71 answers the GPT round on 91a7c0e:
|
6a17c71 to
3e3a1f0
Compare
|
Head 3e3a1f0 (rebased on main 974172e) answers the round on 6a17c71:
|
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
3e3a1f0 to
9e224a3
Compare
|
Head 9e224a3 answers the round on 3e3a1f0:
|
Problem / Motivation
Goal: When host memory is critical, Crew frees memory by stopping its own heaviest non-gateway tool process tree (tests first), never the gateway, and tells the owning chat and the conductor that dispatched it.
On the owner's dev host (Linux, 61 GB, no swap, cgroup v1), heavy test runs pushed the whole host into page-cache thrash twice; the gateway went silent for ~35 minutes and the host rebooted. Today nothing acts on a busy session's running test tree at critical memory: the notifier only toasts, the RSS ceiling only recycles idle sessions, and the kernel OOM killer fires late or never on a swapless host.
Why it matters
A runaway
pytest -nin one worker chat can take down the gateway and every other chat with it. Stopping that one tree early keeps the host, the gateway and the other workers alive, and the notice plus ledger report keep the stop from being a silent failure.Not a goal
resource_status, the conductor rule text. That is PR B of the RFC.vitest,jest) as tier 1; they fall to tier 2 by size.create409.What changed (motivation → approach → change)
This is PR A (fast layer) of
docs/request-for-change/rfc-host-memory-guard.md§4 and §6, with the §8 tests 1-11. The RFC landed on main in #39188; its status line is stilldraft, and moving it is the maintainer's acceptance, not this PR's.notifications/resource_pressure.pycritical, or ontightwith memory PSIfull avg10>= 10. Samples every 5 s (the heartbeat's own beat) while tight or critical, 30 s on ample.agent.resource_critical_gb = 0turns both arms off.platform_compat_metrics.py, listed inplatform_compat.py's manifestlinux_memory_psi_full_avg10();Nonewhen the file is missing.session_memory_guard.py(new)pytesttree, else the largest tree rooted at a shell whose parent is the agent process (runtime, harness, or a harness's only child), so a shell annpxlauncher runs for an MCP server is never a root; never under 512 MB, with a 60 s per-session cool-down. A runtime is walked only while its pid still carries the start identity its client recorded at spawn, and only its topmost leftovers seed the walk. Each pid's start identity is bound to the parent edge the walk saw (a pid reused in between is dropped). Linux stops it leaf first throughterminate_pids_pinned(TERM, KILL after 5 s); a failed stop is retried on the next sample and told once.dashboard/memory_guard_events.py(new),dashboard/memory_guard_report.py(new),work_ledger_writes.py(re-exported fromsrc/kiro_crew/work_ledger.py), seam onsrc/kiro_crew/session.pyMemory guard: ...log line; one notice in the owning chat (same delivery as the leftover-memory notice), naming the agent memory ceiling instead of host memory when that is the reading; for a bound worker onequestionreport (no reason;artifacts,pr,discussionkept; an item whose report isdoneor carries a reason is left alone) recorded in the conductor's crew log like a worker's own report (same unit fallbacks), and undone if the record does not land.Per OS:
Docs: a
session_memory_guardrow indocs/system-specs/modules/runtime-ownership.mdand indocs/architecture/resource-protection.md, anddashboard/memory_guard_report.pyin the work-ledger row ofdocs/feature-map/README.md. The guard takes each session's runtime pid fromSessionCleanup.session_runtime_pidsinsession_cleanup.py, the one reader the RSS ceiling already used, so no new pid reader is added.A runtime two sessions share (the chat-runtime-sharing preview, off by default) cannot say which of them started a tree. Such a stop is logged with
tenants=<n>, every sharing chat gets a notice that says so, and no item gets a report, so no other worker's status is replaced.The chat notice is English text in a
compactionrow withnotice: memory_guard, asleftover_memoryis today; the RFC's i18n keys are not added because the compaction card renders the row's own text.Backwards compatibility
Compatible: no config key, persisted format or API changes; the new report uses the existing
reportevent andquestionstatus.Timing change: the resource notifier samples every 5 s while the posture is tight or critical (30 s on ample, as before). Each sample is the same few
/procand/sysreads, off the loop; the notes it pushes keep their episode rules, so no extra toast fires.Removes nothing: the change only adds a guard path; every existing guard keeps its behavior.
Tests
test/test_session_memory_guard.py: tier 1 beats a larger tier 2 and xdist is one tree; protected pids, harness, MCP roots and outside processes are never picked; a non-shell tier-2 root and a tree under 512 MB are never picked; a detached leftover tree is in reach; cool-down; real processes on Linux: TERM, KILL for a SIGTERM-ignoring survivor, a recycled pid and a protected pid are not signalled; a pass stops one tree off the loop with every log field, one notice and the ledger summary; macOS and Windows arms stop nothing and file nothing;requestis single-flight.test/test_resource_pressure_guard.py: the trigger table (critical, tight+PSI, tight under 10, missing PSI, PSI on ample,critical_gb = 0), 5 s/30 s sampling, the PSI parser.test/test_memory_guard_ledger_report.py: onequestionreport with pointers kept and the report stamp moved; closed item refused; the board's copy is recorded; an unbound session gets no write; an unrecorded write is undone.Proof: not provable -- other: new feature, no prior behavior to revert to
Manual verification
Exit check on this host with the real notifier, guard,
/procwalk, pinned stop, notice wiring and ledger store plus crew-log record; critical was simulated by forcing the posture from the 3rd sample (no memory was exhausted, no model turn ran). A session-leader runtime ranbash -c "python -m pytest -n 2"with two xdist-bootstrap workers holding 1.2 GB each:A read-only
scan_hostover a live dashboard session (runtime shim ->aim sandbox-> launcher ->kiro-cli-> KASnode->zsh, with eight MCP servers beside the shell under KAS) found exactly the shell tool's tree and apython -m pytest -n 2started in it (test 16029 pytest,tool 16028 python), and no MCP server.Known limit: a shell that execs a single non-test command (
zsh -c "npm run build") leaves no shell above it, so that tree is not tier 2; apyteststarted that way is still tier 1.Real-machine verification
Machine: owner dev host, Amazon Linux 2 (kernel 5.10), aarch64, 61 GB RAM, no swap, cgroup v1
Before (main f4754ac, the base this branch was cut from): no guard exists; the same tree keeps running at a critical reading.
After (head 9e224a3): the exit check above.
Cleanup: own computer; the harness killed its runtime and removed nothing else.
Related Issues
Part of #22681
Checklist