Skip to content

feat(conductor): per-worker memory and disk view, posture-change wake, host-health rule - #39289

Open
iamwhatever wants to merge 1 commit into
mainfrom
feat/memory-guard-slow-layer
Open

iamwhatever wants to merge 1 commit into
mainfrom
feat/memory-guard-slow-layer

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A conductor sees its own workers' memory and disk, is woken once when the host's memory or disk posture changes, and follows a short task-agnostic rule to keep the host healthy.

This is PR B (the slow layer) of docs/request-for-change/rfc-host-memory-guard.md (§5, §6 row B, §8 tests 12-14). Today resource_status shows a conductor the host and its own session only, so it cannot tell which of its workers holds the memory or which volume is filling; it reads resources only before a dispatch wave; and nothing tells it when the host turns tight or critical mid-round. Two whole-host deaths on the owner's host (RFC §2.1) happened during heavy parallel test runs that no conductor was watching.

Why it matters

A conductor running several workers is the one actor that can slow the work down before a swapless host thrashes. Without its workers' share and a posture wake, it keeps dispatching into a host that is already short, for any kind of work (code, research, data, browser), not only tests.

Not a goal

  • The gateway guard that stops a test tree at critical memory, PSI sampling and the fast sampling cadence (RFC §4, PR A).
  • Any automatic deletion or stop by the gateway; the conductor acts through its own workers, inside the rule's bounds.
  • A host_pressure record, a new config key, or any change to the create 409, the spawn floor or the idle RSS recycle.
  • Per-worker du walks of build trees (RFC §5.3).
  • Flipping the RFC's status; the RFC is not on main yet (docs(rfc): host memory guard stops the heaviest non-gateway tree, and the conductor keeps the host healthy #39188).

What changed (motivation → approach → change)

The owner ruled (RFC §0) that the conductor reads resources every patrol turn, sees each worker's share, is woken on a posture change, and gets high-level guidance only.

  • Posture wake. The resource notifier now also reads the free space of the temp roots on each sample and publishes one combined posture: the worse of memory and disk. A disk root is tight by the existing TempHeadroom.low rule and critical under 3% space or 200 inodes. On a change it pulls every conductor with an active work-ledger patrol forward (wake_for_posture_change in src/kiro_crew/conductor_wake.py, through the existing _fire). The patrol's probe (src/kiro_crew/probes/work_ledger.py) reads the live posture and wakes once per level change per conductor (one seen token per level), so a repeat wakes nobody and ample wakes nobody until a worse level was seen. Pulls are charged to their own hourly budget, which bounds a flapping posture. The posture level also joins the person-wait fingerprint, so a patrol held because every item waits on a person still gets the posture wake. A disk reading names the axis that tripped (space or inodes).
  • Workers view. For a conductor caller, /api/spawn/session-memory (src/kiro_crew/dashboard/handlers/session_memory_scope.py) adds a workers list: one row per open item on the caller's own board whose binding file names that item, keyed by item id, with memory, process count, largest test-runner tree, busy/idle and the working volume's free space. No session key, title, path or pid is served; the route's spec (docs/system-specs/modules/subagent.md) names this one exception. resource_status renders it as "Your workers (by item)" (src/kiro_crew/mcp_tools/spawn_parts/session_memory.py).
  • Rule. resource_wave becomes host_health, the RFC §5.4 text (goal, facts, hard bounds, labelled examples), identical in spec_prompts.py and the crewmate piece; src/kiro_crew/builtin_skills/goal-conductor/SKILL.md points at it ("as your prompt's host-health rule says") instead of restating it, and src/kiro_crew/builtin_skills/team-lead/SKILL.md points at both paragraphs. This deviates from RFC §8 test 14 (identical text in the skill too) on purpose: the rule keeps one source and the repo's anti-copy check stays universal. One clause is added to the RFC text, "and before each dispatch wave", so the first dispatch turn still reads capacity. The skill keeps its queue paragraph.

Backwards compatibility

Compatible: the route only adds an optional workers field, the tool only adds a block when it is present, and the notifier's toasts are unchanged.

Reader: src/kiro_crew/agent_parts/spec_prompts.py:_LEDGER_CONDUCTOR_RULES["resource_wave"] -- chat (conductor prompt) -- renamed to host_health; every reader is the {host_health} placeholder in crewmate_prompts.json and the root charter's section list, both changed here; test_mate_conductor_parity.py holds them.

Tests

  • test/test_host_posture_wake.py (new): disk posture levels and floors; combined posture; one pull per change, none on a repeat, a disk-only change pulls, flapping pulls per change, unknown is no signal; the sample reads disk in the worker thread; the push fires only watched boards; the probe wakes each conductor once per change and not on a repeat or on the ample baseline (§8 test 13); worker rows hold only the caller's own bound items, serve no key, pid or path, withhold counts on a shared runtime, skip a closed item and a worker re-bound elsewhere (§8 test 12); the tool's rendering; the inode axis is named; a posture change moves the person-wait fingerprint and a repeat does not; the test tree is measured from a real /proc tree on Linux, counts a run left in the runtime's session (session_members=True), and recognizes python /venv/bin/pytest.
  • test/test_mate_conductor_parity.py: the host_health rule is the same text in the prompt and the crewmate piece, the goal-conductor skill carries its pointer and not a copy (§8 test 14, as amended above), the rule keeps its hard bounds and names no per-task recipe; the anti-copy check is unchanged.
  • test/conftest_resets.py: resets the published posture per test, so a notifier test cannot add a posture wake to a later probe test.

Proof: not provable -- other: new behaviour, no prior defect to revert to; each test above fails on main because the symbols it calls do not exist there.

Manual verification

On this host (alinux, Linux 5.10, aarch64), from this branch, with a throwaway data home: a real work ledger with one conductor and two bound workers plus a stranger bound to another conductor; two real process trees, one with a pytest-named child holding 200 MB; a real notifier fed ample -> tight -> tight, and a real WorkLedgerProbe tick per conductor after each sample.

rows served: [{"item_id": "it_4d0a7ed3", "rss_mb": 8.0, "procs": 2, "test_mb": 0.0, "turn": "idle", "volume_free_pct": 16.6}, {"item_id": "it_86b29384", "rss_mb": 230.0, "procs": 2, "test_mb": 212.2, "turn": "busy", "volume_free_pct": 16.6}]
  Your workers (by item):
    it_4d0a7ed3: 8 MB, 2 processes, no test run, idle, work volume 17% free
    it_86b29384: 230 MB, 2 processes, tests 212 MB, busy, work volume 17% free
sample 0: posture=('ample', 'memory 9.0 GB free') pulls=0
   tick chat-demo-conductor: []
   tick chat-demo-other: []
sample 1: posture=('tight', 'memory 3.1 GB free') pulls=1
   tick chat-demo-conductor: ['[work-ledger wake] host posture is tight (memory 3.1 GB free)']
   tick chat-demo-other: ['[work-ledger wake] host posture is tight (memory 3.1 GB free)']
sample 2: posture=('tight', 'memory 3.0 GB free') pulls=1
   tick chat-demo-conductor: []
   tick chat-demo-other: []

Not run: two live worker chats under a conductor in a gateway running this branch (the Dev Fleet pod tools are off on this host).

Per OS:

Host Before After
Linux conductor sees host + own session; no posture wake adds worker rows with test_mb from /proc; posture wake on memory or temp-root change
macOS same same rows and wake; test_mb is null (no /proc walk), rendered "tests unmeasured"
Windows same same as macOS; disk free space through shutil.disk_usage, no inode axis

Real-machine verification

Machine: dev host, Amazon Linux (alinux) kernel 5.10, aarch64, 61 GB RAM, no swap
Before (main 640338a): resource_status has no "Your workers" block and no posture wake exists (the symbols are absent).
After (head, this branch): the demo above, PYTHONPATH=src python /tmp/memguard-b-demo.py.
Cleanup: own computer; demo processes killed, throwaway data home only.

Related Issues

Part of #22681. Implements PR B of RFC #39188 (docs/request-for-change/rfc-host-memory-guard.md §5, §6, §8 tests 12-14). PR A (fast layer) is separate and also edits notifications/resource_pressure.py; this PR touches only disk posture and the posture-change wake there.

Checklist

  • A Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (src/kiro_crew/docs/work-ledger.md)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested review from a team and KiroCrewCaptain as code owners November 18, 2026 21:47
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 18, 2026
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision fc5e213a8da37866c0e898c91af8342250dc2fce touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)

Design-level review of fc5e213a8da37866c0e898c91af8342250dc2fce — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

This PR brings back the conductor posture wake and the level-change release of person-wait holds. A ruling on main withdrew both, and the RFC that would supersede that ruling is still a draft that is not on main.

Blockers

A recorded owner decision is reversed before the record that supersedes it lands
Trigger: the description says "Flipping the RFC's status; the RFC is not on main yet (#39188)" and cites "The owner ruled (RFC §0)". But docs/request-for-change/rfc-work-ledger-resource-pressure.md on main still reads "Withdrawn 2026-11-14 by the repository owner: … the conductor wake (§3.3) … and the §12 amendment", with superseding-by: [].
Outcome: main would carry a "wake_for_posture_change" and a "|posture:" person-wait fingerprint that its own on-main record calls withdrawn. The draft RFC says it must go "RFC-first … because it changes a recorded decision" and asks the owner to "restate this decision on the pull request". A reviewer or agent reading main follows the withdrawal and removes the code again. That would repeat the #36481 → #37164 churn.
Fix: land #39188 with a maintainer's acceptance first. Set superseded-by on the 2026-11-14 record, then rebase this PR.
Clears when: #39188 is merged with its Status line accepted, rfc-work-ledger-resource-pressure.md names it in superseded-by, and the owner's restated decision is linked from this PR.

Notes

  • The prompt no longer says how many workers to dispatch at each posture: "dispatch one" on tight, "none" on critical. Only "dispatch less or none" stays, as an example. That loosens the dispatch rule, and the description should say so.
  • test/fixtures/crewmate_prompts/ledger.golden.txt still has the old resource_wave text.

[DESIGN-REVIEWED] fc5e213

Description read

[DESCRIPTION-READ] 505b94ba17408f3ab19af2991f27361eaac2c7adb2814c5e8704df8cc37d3427

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🔴 BLOCK

Premise-level review of fc5e213a8da37866c0e898c91af8342250dc2fce — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: BLOCK

Product-shape change without accepted RFC: the conductor's default prompt rule now has it act on its workers (close chats, ask them to delete build output), and every patrol now wakes on host posture. The licensing RFC is draft on base.

What this change ships

Inventory (8 items) — 6 justified

Intent: let a conductor see which of its workers is using memory and disk, and slow the work down before the host runs out. ADDITION.

  1. Conductors' default instructions change from "resource_status sizes the wave" to a host-health rule that tells them to act on workers — product shape, no recorded decision
  2. Every conductor with an active work-ledger patrol is now woken when host memory or disk posture changes — product shape, no recorded decision
  3. resource_status lists the conductor's own workers by item (memory, processes, test tree, busy, volume free) — justified
  4. The resource notifier now also reads temp-root disk space and publishes the worse of memory and disk — justified
  5. A posture change releases a patrol held because every item waits on a person — justified
  6. Posture wakes spend their own hourly budget — justified
  7. The goal-conductor and team-lead skills point at the rule instead of restating it — justified
  8. The test-tree size also counts python /venv/bin/pytest launches — justified

Blockers

product-shape change without accepted RFC: the conductor's default host-health behaviour and posture wake — oversized
Trigger: items 1 and 2. The rule resource_wave is replaced by host_health ("act on what your workers are doing… close an idle finished worker chat… ask a worker to remove its own build output") in spec_prompts.py and crewmate_prompts.json, and observe_posture → wake_for_posture_change wakes every watched board. The description cites docs/request-for-change/rfc-host-memory-guard.md, which the base RFC list gives as status: draft, and the description itself says "the RFC is not on main yet (#39188)".
Outcome: every conductor on every install gets new default actions on its workers and a new wake source, and nobody recorded that decision.
Clears when: an RFC covering the conductor host-health rule and posture wake merges with a non-draft status on base, then the author pushes or rebases; or a maintainer posts /ai-review override first-principles fc5e213a8da37866c0e898c91af8342250dc2fce product-shape: <decision>.

[FIRST-PRINCIPLES-REVIEWED] fc5e213

Description read

[DESCRIPTION-READ] af60ef989a292ce65329fb9da6882b9aa680d919bbb763a719f09966605cebc7

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=3b7ea742f7dd

  • needs-a-decision: RFC status

The posture wake and the host_health rule are exactly RFC docs/request-for-change/rfc-host-memory-guard.md §5.2 and §5.4 (merged in #39188, owner decision quoted in its §0).
The only gap is its Status line, still draft on main. A separate docs PR moves it to accepted for a maintainer to approve.
Once that merges, this PR rebases on main and First Principles re-runs. No override is requested here.

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fc5e213a8da37866c0e898c91af8342250dc2fce — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fc5e213

Verdict parsed from the review's SHA-scoped output markers for commit fc5e213a8da37866c0e898c91af8342250dc2fce.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fc5e213a8da37866c0e898c91af8342250dc2fce false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fc5e213a8da37866c0e898c91af8342250dc2fce and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fc5e213

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fc5e213a8da37866c0e898c91af8342250dc2fce false-positive: <reason, 20 to 500 characters>

@iamwhatever
iamwhatever force-pushed the feat/memory-guard-slow-layer branch from c5a240a to 0c6ebe4 Compare November 18, 2026 22:10
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=17d4d44fdd96

  • fixed: subagent.md scope contract

docs/system-specs/modules/subagent.md § resource_status now states the one exception: a conductor's own workers rows, their fields, the read_binding own-board scope, remote items skipped, shared-runtime withholding, the /proc walk. Fixed in 0c6ebe4.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=0f80ad76b09c

  • fixed: disk reading names the tripped axis

disk_posture now uses TempHeadroom.text() ("LOW on inodes ... (N inodes)"), so an inode-only trip is visible in the wake. Test: test_disk_posture_names_the_inode_axis_when_inodes_trip_it. Fixed in 0c6ebe4.
Note on the 2 s budget: the disk read is a few statvfs calls on the temp roots, the same reads resource_status already makes; left as is.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=c52d1f25e53d

  • fixed: person-wait hold no longer swallows posture wakes

person_wait_fingerprint appends |posture:<level>, so a level change releases the hold for one tick and a repeat does not. Test: test_a_posture_change_releases_a_person_wait_hold; test_autonudge_person_wait_hold.py still passes. Fixed in 0c6ebe4.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=00ab54c7f3d2

  • needs-a-decision: RFC status

The host-health rule and posture wake are RFC docs/request-for-change/rfc-host-memory-guard.md §5.2 and §5.4 (merged in #39188; owner decision in its §0).
Its Status line is still draft on main; a separate docs PR moves it to accepted for a maintainer to approve. This PR rebases once that merges. No override is requested here.

@iamwhatever
iamwhatever force-pushed the feat/memory-guard-slow-layer branch from 0c6ebe4 to 9093866 Compare November 18, 2026 22:59
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=9ab830e425a1

  • fixed: test reaps its whole process group

The tree test now spawns with start_new_session=True and reaps with os.killpg(..., SIGKILL) then a bounded wait(timeout=30); no sleep 30 outlives it (checked with pgrep after a local run). Fixed in 9093866.
Same push: the per-worker rows' store read is listed as a seam in test_work_ledger.py with its identity argument, _worker_line is in the spawn facade manifest, and the short poll uses kiro_crew.testing.wait.wait_until.

@iamwhatever
iamwhatever force-pushed the feat/memory-guard-slow-layer branch from 9093866 to 8606d70 Compare November 18, 2026 23:41
@iamwhatever
iamwhatever force-pushed the feat/memory-guard-slow-layer branch from 8606d70 to fc5e213 Compare November 19, 2026 00:04
@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=928162d0d64a

  • fixed: tree walk counts a run left in the runtime's session

_largest_test_tree_mb now uses platform_compat.proc_child_map(session_members=True), the RSS ceiling's measuring map; measurement only, nothing is signalled. Test: test_the_tree_walk_counts_a_run_left_in_the_runtimes_session. Fixed in fc5e213.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=902e05c941b9

  • fixed: interpreter-prefixed pytest is a test run

New measurement-only _measures_as_test_runner: the sweep's classifier (kill path unchanged) plus argv[1] basename pytest; xdist workers fold into that root's tree. Test: test_an_interpreter_prefixed_pytest_counts_as_a_test_run. Fixed in fc5e213.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=bb485b3cd47c

  • fixed: anti-copy ratchet restored, skill points at the rule

The exemption is gone; goal-conductor SKILL.md now says "Keep the host healthy, as your prompt's host-health rule says" and the parity test pins that pointer (prompt and crewmate piece stay identical).
This deviates from RFC rfc-host-memory-guard.md §8 test 14 (identical text in the skill) in this one way, by the conductor's decision: the rule keeps one source, and a ratchet may only tighten. Fixed in fc5e213.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=62df1360695d

  • needs-a-decision: record that supersedes the 2026-11-14 ruling

RFC #39188 is merged on main (2026-11-18T20:57:55Z) with the owner's decision quoted in its §0; its Status line is still draft.
A separate docs PR moves it to accepted for a maintainer to approve; the superseded-by link on rfc-work-ledger-resource-pressure.md belongs with that record, not with this code PR. This PR rebases once it lands.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

span=00ab54c7f3d2

  • needs-a-decision: RFC status

The host-health rule and posture wake are RFC rfc-host-memory-guard.md §5.2 and §5.4 (merged in #39188). Its Status line is still draft; a separate docs PR moves it to accepted. This PR rebases once that merges. No override is requested here.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: action required A blocking check or review needs attention

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant