Skip to content

docs(rfc): host memory guard stops the heaviest non-gateway tree, and the conductor keeps the host healthy - #39188

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
docs/rfc-memory-guard-reclaim
Nov 18, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
docs/rfc-memory-guard-reclaim

Conversation

@iamwhatever

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Record, as a draft RFC, how Crew keeps a host alive under memory and disk pressure: the gateway stops its own heaviest non-gateway tree at critical memory and tells the chat and the conductor, and the conductor watches host resources and its workers' share.

Twice in two days (2026-11-14 and 2026-11-15) a swapless dev host with no cgroup v2 died during heavy test runs. The journal went silent for about 35 minutes, then the host rebooted. Nothing on main stops a running test tree at critical memory: the cleanup watchdog recycles only idle sessions, the notifier only shows toasts, and the conductor sees only its own session's memory.

Why it matters

When the host dies, the gateway, every chat and every worker die with it. On 2026-11-18 the repository owner decided that Crew must free memory by stopping internal non-gateway trees, heavy tests first, and never the gateway. Each stop must reach the owning chat and the conductor. The owner also wants the conductor to manage memory and disk without being tied to one kind of task. That decision changes the 2026-11-14 narrowing that #37164 implemented, so it goes RFC-first.

Not a goal

What changed (motivation → approach → change)

Adds docs/request-for-change/rfc-host-memory-guard.md (status draft) and its row in the RFC index.

  • Fast layer. The gateway acts on the notifier's existing critical reading. It samples every 5 s while memory is tight, and on Linux it also reads PSI. It stops the largest test-runner sub-tree under a Crew session host, one tree per pass. Each stop leaves a log line, a chat notice and one report event on the bound work item, which wakes the conductor.
  • Slow layer. The conductor reads resource_status on every patrol turn and wakes when the posture changes. resource_status gains a per-worker memory and disk block, scoped to the conductor's own board. A short, task-agnostic prompt rule gives the goal and the hard bounds. Concrete actions appear only as examples.
  • The RFC quotes the owner's decision, says exactly what it changes in the refactor(work-ledger): keep only the critical create refusal from host pressure #37164 ruling, and gives a per-OS table, two implementation PRs, tests and an out-of-scope list.

Backwards compatibility

Compatible: documentation only. No code, config or format changes.
Removes nothing: it adds one RFC file and one index row.

Tests

Proof: not provable -- docs or config only

Manual verification

Every path:line cited in the RFC was checked by grep against origin/main at 6bd6d5b3ce. #22683 and #22681 were read live with gh.

Real-machine verification

N/A -- the diff is documentation and runs nothing.

Related Issues

Part of #22681

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 18, 2026 18:43
@iamwhatever
iamwhatever force-pushed the docs/rfc-memory-guard-reclaim branch from 516fedb to a92b5a7 Compare November 18, 2026 18:44
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 67ad6a8

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: this is a draft RFC that changes no code, quotes the owner decision it relies on, and states what it changes in the #37164 ruling.

[DESIGN-REVIEWED] 67ad6a8

Description read

[DESCRIPTION-READ] 44a353ff6154f3005a3603174d223746d32e7daa01af2cdc6f299831398205e6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 67ad6a8

Verdict parsed from the review's SHA-scoped output markers for commit 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 67ad6a8575d8ca81a7665bd373ee4d7e30463f1a false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the docs/rfc-memory-guard-reclaim branch from a92b5a7 to 67ad6a8 Compare November 18, 2026 19:07
@iamwhatever

iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=0e825e882196 off switch did not cover the PSI trigger

fixed in 67ad6a8: §4.1 now says the PSI trigger only stands in for critical and is armed only while critical_gb > 0, so agent.resource_critical_gb = 0 turns the guard off on both triggers.
§8 test 7 now also asserts that a tight reading with PSI full avg10 >= 10 stops nothing when resource_critical_gb = 0.
Swept the same claim: the §7 security bullet ("off when agent.resource_critical_gb = 0") is now true as written.
self-added: no

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Nov 18, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates an 8% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 518 lines, more than the 400 it may approve on its own.

Docs-only draft RFC plus one index row; no code changes. Cited symbols were checked against the repo: terminate_pids_pinned (pidfd pinning, Linux-only signalling, protected_pids/grace_secs), the test-runner and MCP classifiers, SAMPLE_INTERVAL_SECS=30, and the work-ledger and conductor_wake functions. The RFC's claim that a question guard report wakes the conductor holds: repeats_status suppresses only blocked/done, and repeats_wait needs an identical question summary. The kill-authority widening is disclosed in §7, bounded, and tied to a quoted owner decision; as a draft design it is sound. Remaining items are follow-ups for the implementation PRs, not blockers.

Tier: 2, held for a person: design-doc-gate: D.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 67ad6a8575 · 518 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) November 18, 2026 20:57

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified on 67ad6a8: the diff is docs only (the draft RFC plus one index row) and PR Readiness is green. The RFC's cited symbols and constants resolve at the stated audit commit 6bd6d5b. Main has no PSI reader, the RFC is not already on main, and it does not duplicate #22683 or #39189.
Done well: the RFC states up front that it widens what Crew may stop by default and reverses part of #37164, and the agent.resource_critical_gb = 0 off switch now also turns off the PSI trigger.
The design is signed off under the standing maintainer sign-off for changes by team members.
Optional:

  • Section 0 asks the owner to restate the decision on this PR. Linking that comment from section 0 would put the decision on record in a place readers can check.
  • The tier 2 "shell root" rule (section 4.2) still reaches a user-configured MCP server started through bash -c or a shell wrapper. Excluding the session's own mcpServers fingerprints, or listing this as an accepted risk in section 7, would make Goal 2 hold.
  • Sections 4.2 and 4.4 do not say which session owns a stopped tree when one runtime serves several sessions. Attributing the stop to the session whose in-flight command roots the tree would keep notices and ledger events on the right chat and board.

@KiroCrewCaptain
KiroCrewCaptain merged commit 0470f71 into main Nov 18, 2026
98 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the docs/rfc-memory-guard-reclaim branch November 18, 2026 20:57
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 18, 2026
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 18, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 19, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
iamwhatever pushed a commit that referenced this pull request Nov 19, 2026
…t memory

At critical host memory (or a tight reading with memory PSI full avg10 >= 10),
the gateway stops one Crew tool process tree before a swapless host thrashes:
the largest test-runner tree under a session runtime, else the largest
shell-rooted tool tree, never under 512 MB. It never touches the gateway, a
session runtime, a warm-pool spare, a companion runtime, an in-flight launch,
a managed harness, an MCP root, or anything outside the session trees. The stop
is SIGTERM then SIGKILL after 5 s through the pinned terminator, one tree per
pass, with a 60 s per-session cool-down.

Each stop writes one log line, one notice in the owning chat, and for a bound
worker one question report on its item, which wakes its conductor through the
existing report wake. macOS reports the tree it would stop and stops nothing;
Windows logs once. The notifier samples every 5 s while tight or critical.

Fast layer (PR A) of docs/request-for-change/rfc-host-memory-guard.md (#39188).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants