Repository navigation
fix(subagent): queue every capacity verdict and read memory off the loop - #16355
Conversation
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS A non-durable row popped by the pump now lives only in Watch
[DESIGN-REVIEWED] 7dff121 Description read[DESCRIPTION-READ] 035bac66f131ed28f25d58ec66c547c650a45a633b9c367a21f5e523dd678878 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS When a memory reading times out, the queued chip says "not enough free memory; free up memory to continue", which is false: nobody measured the memory. Watch
Suggestions
[UX-REVIEWED] 7dff121 Description read[DESCRIPTION-READ] 888376248e81a05851277171ced3cfa14cf01f37c540ecc4eb1921a9a2f9a5cf That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new off-loop floor read rebuilds a cached off-thread reader that already exists in Not justified as shipped
What this change shipsInventory (8 items) — 7 justifiedIntent: let a subagent spawn that doesn't fit in memory wait in the queue and start later, and stop reading host memory on the gateway's event loop. FIX.
[FIRST-PRINCIPLES-REVIEWED] 7dff121 Description read[DESCRIPTION-READ] a4a6b2ef85e78830d723d840c65436ef920e4af0758c460abbf3877affb52e89 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
96cba66 to
34dbeaf
Compare
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
34dbeaf to
79dbe23
Compare
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
|
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
|
|
Maintainer decisions on the judgement calls in this round: |
79dbe23 to
358b7a5
Compare
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
Also in this push: rebased onto origin/main 6ce8c4b. Main's #16191 |
|
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
|
Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.
|
|
Maintainer decisions:
|
358b7a5 to
7dcdd14
Compare
|
bfb67cf to
6a6ce2b
Compare
|
|
Pushed 6a6ce2b. Commit 2 is amended, so the PR still has 2 commits by the same author, and the base is unchanged (ae67568; it merges cleanly). This push carries the three fixes accepted on bfb67cf and the two should-fix items a post-push review raised on the same code path.
Deterministic barrier tests, all failing on bfb67cf:
The spec paragraph in Local results: black, subprocess-encoding, isort, flake8, mypy (2102 files), docs-lint and harness-parity are clean. 3472 tests pass across the 94 subagent, taskq, overload, spawn, admission, cancel, fairness and pressure test files, plus 809 in test_session.py and the floor, gate and census files. |
|
|
|
The fixture pinned the host reading at 8 GB but answered True for any floor, so a test whose wave of unsettled starts had outgrown an 8 GB host still read as admitted. It now answers (8.0 >= min_gb, 8.0), the real reader's answer on such a host, and a 9 GB floor is refused. overload_fakes' ManagerHarness pins the same host the same way. The 2000-submission drain test ran eight unsettled dedicated starts at once, which the 2 GB floor plus the start price does not fit on the pinned host; it now runs four, which do. testing-conventions records the fixture's contract.
A spawn that did not fit the 2 GiB floor was refused when it had no durable row (incognito or temporary memory, or the task queue off), and every spawn was also held by the posture tier, which at defaults sits on the same 2 GiB line, so a host admission had filled to the floor read critical and deferred every start. The floor was read on the event loop, walking cgroup files. Now a capacity verdict never refuses: a start with no durable row that does not fit waits in the in-memory window, skipped by the pump until the admit wait passes, with the same low_memory label. Its wake is armed one clock tick past that stamp and re-arms if it fires early, so the wait cannot strand. The posture tier is no longer consulted for spawns (cron every/at firings still defer on it), so posture_critical is gone as a spawn wait reason. Event-loop entries (spawn_async and the pump, both with or without a store, so a timer re-check of an in-memory wait is a coroutine too) stop at a MemoryReadPoint, read the host on a worker and re-enter; the fit is decided on the loop against the bar recomputed then. The worker read has no on-loop fallback: one that misses its bound, or a pool that cannot start a thread, comes back unanswered and the start waits as low_memory, re-checked after the admit wait, never failing open; the dedicated top-up's poll uses the same reader and keeps waiting on an unanswered read. A disabled floor (spawn_min_memory_gb 0) takes no reading at all, so an unanswered read cannot defer a start it does not gate. The re-entry re-runs the policy gates on every path, the row spawn_async already committed included, so a governance change made during the read still refuses; a committed row refused there, or because admission closed during the read, is failed in the store (the glue test that pinned the old skip now pins the re-vet). A durable defer the store could not write is refused as the store's failure (task_store_unavailable), never with capacity wording. Governance, cwd and memory-identity refusals are unchanged. A census test pins every outcome on the spawn path, every wait label and every refusal text the gate can return at defaults. A row with no durable record that the coroutine pump has popped is held in _undurable_in_dispatch from its pop until the gate takes it (through the pass's resume grants and the row's own off-loop reads), as its only copy. cancel, Stop all, the parent-end snapshot and a stage boundary cancel all find it there, and the pump re-checks it after each await, so a stopped row never starts. A raise in that gap, in a read or in a grant ahead of the dispatch, puts the row back in the window, retried after the admit wait, instead of dropping it. The host read is single-flight per bar, so a hung reader holds one executor thread rather than one more per re-check. With the macOS kernel memory-pressure hold (#15876) both hold: a start that fits the floor can still wait as memory_pressure in the window, and one that does not fit waits as low_memory, which outranks the hold and carries no pressure clock, so the hold's bound never ends it. With spawns no longer reading the posture, no posture verdict precedes the hold. The pump's pick does not ask the hold about a stamped floor wait, and a clock an earlier hold started is dropped when the start goes back to the floor, so time below the floor never counts toward the hold's bound. A durable row deferred on the floor keeps only next_run_at in the store, so the manager remembers it and the window refill stamps it as a floor wait as well. A spawn with no row awaits its W3 parent registration off the loop, as the durable path does. The census counts memory_pressure among the wait labels and the hold's audit rows and "never started" terminal on the spawn path. Breaking: the posture_critical wait reason, the refused_low_memory, refused_memory_critical and deferred_memory_critical SEL outcomes, and subagent.cached_admission_check are removed. Writer sweep on origin/main @ 6ce8c4b: the only readers are the ones listed below (overload_fakes' posture fake, which a queue-depth test used to defer a spawn, is now a floor reading). Reader: src/kiro_crew/subagent_wait_reasons.py:DEFERRED_QUEUED_REASONS -- subagent -- dashboard/handlers/messaging.py and messaging/commands.py read the set; low_memory, adaptive_cap_zero and memory_pressure remain (test_the_wait_reasons_are_the_complete_set) Reader: website/src/pages/chat/subagentQueuedReason.ts:parseSubagentQueuedReason -- chat -- an older gateway's posture_critical label parses to undefined and the chip keeps its default text (ActivityViewer.test.tsx) Reader: src/kiro_crew/taskq/adapters/runner.py:cached_admission_check -- cron -- the runner imports its own from resource_status, so the cron posture defer is unchanged (test_taskq_admission_integration.py) Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_POSTURE_CRITICAL -- release -- release/0.8.0 keeps its own copy until the G6-080 backport
6a6ce2b to
7dff121
Compare
|
|
|
|
This PR merged at 7dff121 before the last post-push review items were pushed. They are now in follow-up #16885: one in-flight host read per process, whatever the bar (it had been keyed by min_gb); a test that a stop landing during the host read is honoured; a test of the floor-off guard after the policy read; and the unfinished done-callback comment. Each new test fails against the mutation it guards. |
Follow-up to #16355. The off-loop host-memory read was single-flight per bar (keyed by min_gb), but the reading and its cause do not depend on the bar, and the bar differs per agent bucket and moves as warming rows settle. A hung reader therefore still took one more executor thread for every distinct bar. One read is now in flight per process, and each caller compares the shared figure against its own bar. The spec and docstring say so, and the done-callback comment is finished. Two stop guards in the pump's dispatch of a non-durable row were each covered only by the other one. The stop test now also blocks inside the gate's host read (only the check after the read sees that stop) and in the policy read with the floor disabled (the gate takes no read and starts the row in the same call, so the pump's check after its policy read is the only guard). Each variant fails when its guard is removed, and a new test fails when two bars each start their own read.
Mainline-plan U2+U3 (G6/WP2b + WP1 + F1 census).
Problem / Motivation
Goal: a subagent spawn that does not fit in memory always waits in the queue and starts once memory frees, instead of being refused, and the event-loop entries (
spawn_async, the coroutine pump, the dedicated top-up's poll) no longer read the floor on the gateway's event loop. One residual is accepted: withagent.task_queue_enabled=falsethe in-memory queue's synchronous drain still reads on the thread that calls it, assubagent.mdrecords.agent.task_queue_enabled=false) that did not fit the 2 GiB floor was refused with "spawn refused: only N GB memory available". The parent had to retry by hand.resource_critical_gb, also 2 GiB by default). A host that admission had filled to the floor therefore read as critical, and every later start was deferred asposture_critical, even ones that fit.Why it matters
A fan-out on a busy host loses work it could have run a few seconds later, and the parent model has to tell a capacity refusal apart from a policy refusal. The two 2 GiB checks also hold starts back twice for the same shortage. And a slow cgroup read on the loop stalls every chat, channel and dashboard request served by that gateway.
Not a goal
adaptive_cap_zeroor the count cap. That is U4 (G9/WP3).low_memorywait. That is feat: bound how long a subagent spawn waits for memory #16347. (fix(subagent): veto macOS starts under kernel memory pressure #15876, now on main, bounds only its ownmemory_pressurewait.) Until feat: bound how long a subagent spawn waits for memory #16347 lands, alow_memorywait has no limit, just as a durable deferral had none before this PR. It shows inspawn_statusand on the chip, it can be stopped like any queued spawn, and a gateway restart drops a non-durable one. The maintainer accepted this: fix(subagent): queue every capacity verdict and read memory off the loop #16355 (comment).What changed (motivation → approach → change)
The rule is the one rfc-overload-resilience §3 already states: pressure defers, it never refuses. This PR applies it to every capacity verdict and keeps policy refusals as they are.
healthy_host_memoryandoverload_fakes.ManagerHarnessnow pin an 8 GB host that compares the floor it is asked about,(8.0 >= min_gb, 8.0), instead of admitting any floor. One drain test is resized to fit that host.subagent_manager/admission/gate.py). A start with no durable row that does not fit waits in the in-memory window with the samelow_memorylabel. It is stamped withMEMORY_WAIT_UNTIL_KEY, and the pump skips it until the admit wait passes (admission/fairness.py).arm_memory_waitarms the re-pump one clock tick past the stamp and re-arms if it fires early, so the wait cannot strand.posture_criticalis gone as a spawn wait reason (subagent_wait_reasons.py), and so is its chip text (subagentQueuedReason.ts, plus its strings in the 13website/src/i18nlocale catalogs).src/kiro_crew/subagent.pydrops itscached_admission_checkre-export. Cron is unchanged; theresource_status.pydocstrings now name only cron and task-runner steps as gated.spawn_asyncand the coroutine pump, both with or without a store, so the timer re-check of an in-memory wait never reads on the loop) stop at aMemoryReadPoint(admission/types.py), read the host on a worker and re-enter. The fit is decided against the bar recomputed then, and the re-entry is not a second submission. There is no on-loop fallback: a read that misses_HOST_READ_OFF_LOOP_SECSor gets no thread comes back unanswered (MEMORY_CAUSE_READ_UNANSWERED), and the start waits aslow_memory. It never fails open. A spawn with no row registers its W3 parent wait off the loop too.agent.task_queue_enabled=false) that the coroutine pump has popped is held in_undurable_in_dispatchfrom its pop until the gate takes it: through the same pass's resume grants and the row's own off-loop reads (parent policy, agent check, the floor).cancel(through_unqueue), Stop all, the parent-end snapshot and a stage-boundary cancel all find it there, and the pump re-checks the entry after each await, so a Stop pressed mid-dispatch answers True and the row never starts. A raise in that gap (a pool that cannot start a thread, in a read or in a grant) puts the row back at the front of the window, retried after the admit wait (_requeue_undispatched), instead of dropping its only copy. A durable row needs none of this: the store keeps it and its claim re-checks the stop.task_store_unavailable), never with capacity wording.spawn_asyncalready committed included. A refused row that already exists is failed in the store, and so is a committed row when admission closes during the read. The dedicated top-up keeps waiting on an unanswered read instead of re-reading on the loop.test_spawn_admission_verdict_census.py) pins every outcome on the spawn path, every wait label and every refusal text thatspawn_implcan return at defaults.memory_pressure. Alow_memorywait outranks the hold and carries no pressure clock (the pick does not ask the hold about it, and a clock an earlier hold started is dropped), so the hold's bound never ends it. The census includesmemory_pressureand the hold's outcomes.admission_gatehelp in config/sections.py and config-baseline.json. Also the feature map and thewriting-testsskill (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md).Backwards compatibility
Breaking: spawns no longer read the posture tier, so an operator who raised
agent.resource_critical_gbaboveagent.spawn_min_memory_gb(or set the floor to0) no longer has spawns held in that band; only the floor and the adaptive cap gate them, and cron'severy/atfirings still defer on posture (a cron-expression job never did). This is an accepted change (rfc-overload-resilience Q9). Also removed: theposture_criticalspawn wait reason, therefused_low_memory,refused_memory_criticalanddeferred_memory_criticalSEL outcomes, andkiro_crew.subagent.cached_admission_checkare removed. A spawn that used to be refused for memory now returns a queued id. Writer sweep on origin/main @ 6ce8c4b: the only readers are the ones listed below (a queue-depth test that deferred a spawn through the posture fake now uses a floor reading).Reader: src/kiro_crew/subagent_wait_reasons.py:DEFERRED_QUEUED_REASONS -- subagent -- dashboard/handlers/messaging.py and messaging/commands.py read the set; low_memory, adaptive_cap_zero and memory_pressure remain (test_the_wait_reasons_are_the_complete_set)
Reader: website/src/pages/chat/subagentQueuedReason.ts:parseSubagentQueuedReason -- chat -- an older gateway's posture_critical label parses to undefined and the chip keeps its default text (ActivityViewer.test.tsx)
Reader: src/kiro_crew/taskq/adapters/runner.py:cached_admission_check -- cron -- the runner imports its own from resource_status, so the cron posture defer is unchanged (test_taskq_admission_integration.py)
Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_POSTURE_CRITICAL -- release -- release/0.8.0 keeps its own copy until the G6-080 backport
Tests
test_spawn_admission_verdict_census.py(new): every outcome, wait reason and refusal text is accounted for. Every capacity condition queues (memory_pressureincluded), only policy refuses, the gate never reads the posture, and the read's re-entry is not a second submission.test_spawn_floor_defaults.py: the gate reads memory, and registers a no-row spawn's parent wait, off the loop. A memory wait woken early still starts.test_spawn_floor_defaults.py(in-dispatch hold): a Stop by each of the four stop paths while a blocked policy read holds the dispatch, a Stop while a blocked resume grant of the same pass holds the popped row, and a raise in either place (the row goes back to the window and starts later), each with the task queue off and for an incognito row beside a store. The barriers are threading/asyncio events, never sleeps. The stuck-read top-up test also asserts the read is submitted once.test_fairness_lanes.py: low memory queues every subagent and defers only a durable row; a drain before the stamp neither starts the wait nor reads memory, and the pick never asks the pressure hold about it.test_subagent_spawn_host_pin.py: a floor above the pinned host is refused, and one it clears is admitted.test_subagent.py,test_admission_gate.py,test_overload_integration_glue.pyandintegration/test_governance.py: the former refusals now expect a queuedlow_memorywait. The legacy-entry case expects the store wording and code.ActivityViewer.test.tsxandsubagentActivity.test.tscheck that aposture_criticallabel falls back to the default text.Manual verification
N/A — the verdicts are pure gate logic, covered by the census and by the integration governance test run in-process with
KIROCREW_INTEGRATION=1.Screenshots / video
Why no screenshot: a current gateway never sends
posture_critical, so no chip renders differently. Only an older gateway's label now falls back to the existing default text, whichActivityViewer.test.tsxasserts.Related Issues
Part of #16480
Builds on PR #13109: triage orders it before G6, and it is still open. U2, the other dependency, is commit 1 of this PR.
Dependency notes:
memory_pressurewait reason and landed first; this PR adds it to the census (test_the_wait_reasons_are_the_complete_set).ADAPTIVE_CAP_ZERO. This PR keeps it.Pattern harvest
Rule candidate: review-prompt
Pattern: a capacity condition (memory, posture, cap) answered with a refusal instead of a wait; the census test is the guard.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)