Repository navigation
feat(conductor): per-worker memory and disk view, posture-change wake, host-health rule - #39289
iamwhatever wants to merge 1 commit into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASS (a BLOCK with no Trigger and no Outcome does not block)Design-level review of Design-Verdict: PASS This PR brings back the conductor posture wake and the level-change release of person-wait holds. A ruling on main withdrew both, and the RFC that would supersede that ruling is still a draft that is not on main. BlockersA recorded owner decision is reversed before the record that supersedes it lands Notes
[DESIGN-REVIEWED] fc5e213 Description read[DESCRIPTION-READ] 505b94ba17408f3ab19af2991f27361eaac2c7adb2814c5e8704df8cc37d3427 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🔴 BLOCKPremise-level review of First-Principles-Verdict: BLOCK Product-shape change without accepted RFC: the conductor's default prompt rule now has it act on its workers (close chats, ask them to delete build output), and every patrol now wakes on host posture. The licensing RFC is What this change shipsInventory (8 items) — 6 justifiedIntent: let a conductor see which of its workers is using memory and disk, and slow the work down before the host runs out. ADDITION.
Blockersproduct-shape change without accepted RFC: the conductor's default host-health behaviour and posture wake — oversized [FIRST-PRINCIPLES-REVIEWED] fc5e213 Description read[DESCRIPTION-READ] af60ef989a292ce65329fb9da6882b9aa680d919bbb763a719f09966605cebc7 That sha256 names the PR description this verdict read. Recompute it with |
|
span=3b7ea742f7dd
|
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
c5a240a to
0c6ebe4
Compare
|
span=17d4d44fdd96
|
|
span=0f80ad76b09c
|
|
span=c52d1f25e53d
|
|
span=00ab54c7f3d2
|
0c6ebe4 to
9093866
Compare
|
span=9ab830e425a1
|
9093866 to
8606d70
Compare
…, host-health rule
8606d70 to
fc5e213
Compare
|
span=928162d0d64a
|
|
span=902e05c941b9
|
|
span=bb485b3cd47c
|
|
span=62df1360695d
|
|
span=00ab54c7f3d2
|
Problem / Motivation
Goal: A conductor sees its own workers' memory and disk, is woken once when the host's memory or disk posture changes, and follows a short task-agnostic rule to keep the host healthy.
This is PR B (the slow layer) of
docs/request-for-change/rfc-host-memory-guard.md(§5, §6 row B, §8 tests 12-14). Todayresource_statusshows a conductor the host and its own session only, so it cannot tell which of its workers holds the memory or which volume is filling; it reads resources only before a dispatch wave; and nothing tells it when the host turnstightorcriticalmid-round. Two whole-host deaths on the owner's host (RFC §2.1) happened during heavy parallel test runs that no conductor was watching.Why it matters
A conductor running several workers is the one actor that can slow the work down before a swapless host thrashes. Without its workers' share and a posture wake, it keeps dispatching into a host that is already short, for any kind of work (code, research, data, browser), not only tests.
Not a goal
host_pressurerecord, a new config key, or any change to thecreate409, the spawn floor or the idle RSS recycle.duwalks of build trees (RFC §5.3).What changed (motivation → approach → change)
The owner ruled (RFC §0) that the conductor reads resources every patrol turn, sees each worker's share, is woken on a posture change, and gets high-level guidance only.
tightby the existingTempHeadroom.lowrule andcriticalunder 3% space or 200 inodes. On a change it pulls every conductor with an active work-ledger patrol forward (wake_for_posture_changeinsrc/kiro_crew/conductor_wake.py, through the existing_fire). The patrol's probe (src/kiro_crew/probes/work_ledger.py) reads the live posture and wakes once per level change per conductor (one seen token per level), so a repeat wakes nobody andamplewakes nobody until a worse level was seen. Pulls are charged to their own hourly budget, which bounds a flapping posture. The posture level also joins the person-wait fingerprint, so a patrol held because every item waits on a person still gets the posture wake. A disk reading names the axis that tripped (space or inodes)./api/spawn/session-memory(src/kiro_crew/dashboard/handlers/session_memory_scope.py) adds aworkerslist: one row per open item on the caller's own board whose binding file names that item, keyed by item id, with memory, process count, largest test-runner tree, busy/idle and the working volume's free space. No session key, title, path or pid is served; the route's spec (docs/system-specs/modules/subagent.md) names this one exception.resource_statusrenders it as "Your workers (by item)" (src/kiro_crew/mcp_tools/spawn_parts/session_memory.py).resource_wavebecomeshost_health, the RFC §5.4 text (goal, facts, hard bounds, labelled examples), identical inspec_prompts.pyand the crewmate piece;src/kiro_crew/builtin_skills/goal-conductor/SKILL.mdpoints at it ("as your prompt's host-health rule says") instead of restating it, andsrc/kiro_crew/builtin_skills/team-lead/SKILL.mdpoints at both paragraphs. This deviates from RFC §8 test 14 (identical text in the skill too) on purpose: the rule keeps one source and the repo's anti-copy check stays universal. One clause is added to the RFC text, "and before each dispatch wave", so the first dispatch turn still reads capacity. The skill keeps its queue paragraph.Backwards compatibility
Compatible: the route only adds an optional
workersfield, the tool only adds a block when it is present, and the notifier's toasts are unchanged.Reader: src/kiro_crew/agent_parts/spec_prompts.py:_LEDGER_CONDUCTOR_RULES["resource_wave"] -- chat (conductor prompt) -- renamed to
host_health; every reader is the{host_health}placeholder increwmate_prompts.jsonand the root charter's section list, both changed here;test_mate_conductor_parity.pyholds them.Tests
test/test_host_posture_wake.py(new): disk posture levels and floors; combined posture; one pull per change, none on a repeat, a disk-only change pulls, flapping pulls per change,unknownis no signal; the sample reads disk in the worker thread; the push fires only watched boards; the probe wakes each conductor once per change and not on a repeat or on the ample baseline (§8 test 13); worker rows hold only the caller's own bound items, serve no key, pid or path, withhold counts on a shared runtime, skip a closed item and a worker re-bound elsewhere (§8 test 12); the tool's rendering; the inode axis is named; a posture change moves the person-wait fingerprint and a repeat does not; the test tree is measured from a real/proctree on Linux, counts a run left in the runtime's session (session_members=True), and recognizespython /venv/bin/pytest.test/test_mate_conductor_parity.py: thehost_healthrule is the same text in the prompt and the crewmate piece, the goal-conductor skill carries its pointer and not a copy (§8 test 14, as amended above), the rule keeps its hard bounds and names no per-task recipe; the anti-copy check is unchanged.test/conftest_resets.py: resets the published posture per test, so a notifier test cannot add a posture wake to a later probe test.Proof: not provable -- other: new behaviour, no prior defect to revert to; each test above fails on main because the symbols it calls do not exist there.
Manual verification
On this host (alinux, Linux 5.10, aarch64), from this branch, with a throwaway data home: a real work ledger with one conductor and two bound workers plus a stranger bound to another conductor; two real process trees, one with a
pytest-named child holding 200 MB; a real notifier fedample->tight->tight, and a realWorkLedgerProbetick per conductor after each sample.Not run: two live worker chats under a conductor in a gateway running this branch (the Dev Fleet pod tools are off on this host).
Per OS:
test_mbfrom/proc; posture wake on memory or temp-root changetest_mbisnull(no/procwalk), rendered "tests unmeasured"shutil.disk_usage, no inode axisReal-machine verification
Machine: dev host, Amazon Linux (alinux) kernel 5.10, aarch64, 61 GB RAM, no swap
Before (main 640338a):
resource_statushas no "Your workers" block and no posture wake exists (the symbols are absent).After (head, this branch): the demo above,
PYTHONPATH=src python /tmp/memguard-b-demo.py.Cleanup: own computer; demo processes killed, throwaway data home only.
Related Issues
Part of #22681. Implements PR B of RFC #39188 (
docs/request-for-change/rfc-host-memory-guard.md§5, §6, §8 tests 12-14). PR A (fast layer) is separate and also editsnotifications/resource_pressure.py; this PR touches only disk posture and the posture-change wake there.Checklist
src/kiro_crew/docs/work-ledger.md)