Repository navigation
docs(rfc): host memory guard stops the heaviest non-gateway tree, and the conductor keeps the host healthy - #39188
Conversation
516fedb to
a92b5a7
Compare
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: this is a draft RFC that changes no code, quotes the owner decision it relies on, and states what it changes in the #37164 ruling. [DESIGN-REVIEWED] 67ad6a8 Description read[DESCRIPTION-READ] 44a353ff6154f3005a3603174d223746d32e7daa01af2cdc6f299831398205e6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
… the conductor keeps the host healthy
a92b5a7 to
67ad6a8
Compare
|
Code change approver recommendation: A person should decideWhy: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 518 lines, more than the 400 it may approve on its own. Docs-only draft RFC plus one index row; no code changes. Cited symbols were checked against the repo: terminate_pids_pinned (pidfd pinning, Linux-only signalling, protected_pids/grace_secs), the test-runner and MCP classifiers, SAMPLE_INTERVAL_SECS=30, and the work-ledger and conductor_wake functions. The RFC's claim that a Tier: 2, held for a person: design-doc-gate: D. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Verified on 67ad6a8: the diff is docs only (the draft RFC plus one index row) and PR Readiness is green. The RFC's cited symbols and constants resolve at the stated audit commit 6bd6d5b. Main has no PSI reader, the RFC is not already on main, and it does not duplicate #22683 or #39189.
Done well: the RFC states up front that it widens what Crew may stop by default and reverses part of #37164, and the agent.resource_critical_gb = 0 off switch now also turns off the PSI trigger.
The design is signed off under the standing maintainer sign-off for changes by team members.
Optional:
- Section 0 asks the owner to restate the decision on this PR. Linking that comment from section 0 would put the decision on record in a place readers can check.
- The tier 2 "shell root" rule (section 4.2) still reaches a user-configured MCP server started through
bash -cor a shell wrapper. Excluding the session's own mcpServers fingerprints, or listing this as an accepted risk in section 7, would make Goal 2 hold. - Sections 4.2 and 4.4 do not say which session owns a stopped tree when one runtime serves several sessions. Attributing the stop to the session whose in-flight command roots the tree would keep notices and ledger events on the right chat and board.
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
…t memory At critical host memory (or a tight reading with memory PSI full avg10 >= 10), the gateway stops one Crew tool process tree before a swapless host thrashes: the largest test-runner tree under a session runtime, else the largest shell-rooted tool tree, never under 512 MB. It never touches the gateway, a session runtime, a warm-pool spare, a companion runtime, an in-flight launch, a managed harness, an MCP root, or anything outside the session trees. The stop is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per pass, with a 60 s per-session cool-down. Each stop writes one log line, one notice in the owning chat, and for a bound worker one question report on its item, which wakes its conductor through the existing report wake. macOS reports the tree it would stop and stops nothing; Windows logs once. The notifier samples every 5 s while tight or critical. Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
Problem / Motivation
Goal: Record, as a draft RFC, how Crew keeps a host alive under memory and disk pressure: the gateway stops its own heaviest non-gateway tree at critical memory and tells the chat and the conductor, and the conductor watches host resources and its workers' share.
Twice in two days (2026-11-14 and 2026-11-15) a swapless dev host with no cgroup v2 died during heavy test runs. The journal went silent for about 35 minutes, then the host rebooted. Nothing on main stops a running test tree at critical memory: the cleanup watchdog recycles only idle sessions, the notifier only shows toasts, and the conductor sees only its own session's memory.
Why it matters
When the host dies, the gateway, every chat and every worker die with it. On 2026-11-18 the repository owner decided that Crew must free memory by stopping internal non-gateway trees, heavy tests first, and never the gateway. Each stop must reach the owning chat and the conductor. The owner also wants the conductor to manage memory and disk without being tied to one kind of task. That decision changes the 2026-11-14 narrowing that #37164 implemented, so it goes RFC-first.
Not a goal
host_pressurerecord, and no stops on macOS or Windows in the first PR.What changed (motivation → approach → change)
Adds
docs/request-for-change/rfc-host-memory-guard.md(statusdraft) and its row in the RFC index.reportevent on the bound work item, which wakes the conductor.resource_statuson every patrol turn and wakes when the posture changes.resource_statusgains a per-worker memory and disk block, scoped to the conductor's own board. A short, task-agnostic prompt rule gives the goal and the hard bounds. Concrete actions appear only as examples.Backwards compatibility
Compatible: documentation only. No code, config or format changes.
Removes nothing: it adds one RFC file and one index row.
Tests
Proof: not provable -- docs or config only
Manual verification
Every
path:linecited in the RFC was checked by grep againstorigin/mainat6bd6d5b3ce. #22683 and #22681 were read live withgh.Real-machine verification
N/A -- the diff is documentation and runs nothing.
Related Issues
Part of #22681
Checklist