Skip to content

fix(subagent): queue every capacity verdict and read memory off the loop - #16355

Merged
chenmingwei23 merged 2 commits into
mainfrom
fix/subagent-capacity-verdicts-queue
Oct 5, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
fix/subagent-capacity-verdicts-queue

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Mainline-plan U2+U3 (G6/WP2b + WP1 + F1 census).

Problem / Motivation

Goal: a subagent spawn that does not fit in memory always waits in the queue and starts once memory frees, instead of being refused, and the event-loop entries (spawn_async, the coroutine pump, the dedicated top-up's poll) no longer read the floor on the gateway's event loop. One residual is accepted: with agent.task_queue_enabled=false the in-memory queue's synchronous drain still reads on the thread that calls it, as subagent.md records.

  • A spawn with no durable row (incognito or temporary memory mode, or agent.task_queue_enabled=false) that did not fit the 2 GiB floor was refused with "spawn refused: only N GB memory available". The parent had to retry by hand.
  • Every spawn was also held by the posture tier (resource_critical_gb, also 2 GiB by default). A host that admission had filled to the floor therefore read as critical, and every later start was deferred as posture_critical, even ones that fit.
  • The floor was read on the event loop, which walks cgroup files on Linux.

Why it matters

A fan-out on a busy host loses work it could have run a few seconds later, and the parent model has to tell a capacity refusal apart from a policy refusal. The two 2 GiB checks also hold starts back twice for the same shortage. And a slow cgroup read on the loop stalls every chat, channel and dashboard request served by that gateway.

Not a goal

What changed (motivation → approach → change)

The rule is the one rfc-overload-resilience §3 already states: pressure defers, it never refuses. This PR applies it to every capacity verdict and keeps policy refusals as they are.

  • Commit 1 (U2/WP1). healthy_host_memory and overload_fakes.ManagerHarness now pin an 8 GB host that compares the floor it is asked about, (8.0 >= min_gb, 8.0), instead of admitting any floor. One drain test is resized to fit that host.
  • Commit 2 (U3/G6).
    • A capacity verdict never refuses (subagent_manager/admission/gate.py). A start with no durable row that does not fit waits in the in-memory window with the same low_memory label. It is stamped with MEMORY_WAIT_UNTIL_KEY, and the pump skips it until the admit wait passes (admission/fairness.py). arm_memory_wait arms the re-pump one clock tick past the stamp and re-arms if it fires early, so the wait cannot strand.
    • Spawns no longer read the posture tier. posture_critical is gone as a spawn wait reason (subagent_wait_reasons.py), and so is its chip text (subagentQueuedReason.ts, plus its strings in the 13 website/src/i18n locale catalogs). src/kiro_crew/subagent.py drops its cached_admission_check re-export. Cron is unchanged; the resource_status.py docstrings now name only cron and task-runner steps as gated.
    • The event-loop entries (spawn_async and the coroutine pump, both with or without a store, so the timer re-check of an in-memory wait never reads on the loop) stop at a MemoryReadPoint (admission/types.py), read the host on a worker and re-enter. The fit is decided against the bar recomputed then, and the re-entry is not a second submission. There is no on-loop fallback: a read that misses _HOST_READ_OFF_LOOP_SECS or gets no thread comes back unanswered (MEMORY_CAUSE_READ_UNANSWERED), and the start waits as low_memory. It never fails open. A spawn with no row registers its W3 parent wait off the loop too.
    • The in-dispatch hold. A row with no durable record (incognito or temporary memory, or agent.task_queue_enabled=false) that the coroutine pump has popped is held in _undurable_in_dispatch from its pop until the gate takes it: through the same pass's resume grants and the row's own off-loop reads (parent policy, agent check, the floor). cancel (through _unqueue), Stop all, the parent-end snapshot and a stage-boundary cancel all find it there, and the pump re-checks the entry after each await, so a Stop pressed mid-dispatch answers True and the row never starts. A raise in that gap (a pool that cannot start a thread, in a read or in a grant) puts the row back at the front of the window, retried after the admit wait (_requeue_undispatched), instead of dropping its only copy. A durable row needs none of this: the store keeps it and its claim re-checks the stop.
    • The host read is single-flight per bar: a caller that arrives while a read for the same bar is in flight awaits that one (shielded), so a hung cgroup read holds one executor thread instead of one more per re-check.
    • A durable defer that the store could not write is refused as the store's failure (task_store_unavailable), never with capacity wording.
    • The re-entry re-runs the policy gates (governance, cwd, the parent spec's allowlist) on every path, the row spawn_async already committed included. A refused row that already exists is failed in the store, and so is a committed row when admission closes during the read. The dedicated top-up keeps waiting on an unanswered read instead of re-reading on the loop.
    • F1 census (test_spawn_admission_verdict_census.py) pins every outcome on the spawn path, every wait label and every refusal text that spawn_impl can return at defaults.
    • With fix(subagent): veto macOS starts under kernel memory pressure #15876's pressure hold: a start that fits the floor can still wait as memory_pressure. A low_memory wait outranks the hold and carries no pressure clock (the pick does not ask the hold about it, and a clock an earlier hold started is dropped), so the hold's bound never ends it. The census includes memory_pressure and the hold's outcomes.
    • Specs: subagent.md, taskq.md, testing-conventions.md. User docs: subagents.md, configuration.md, dynamic-subagent-sizing.md, plus the admission_gate help in config/sections.py and config-baseline.json. Also the feature map and the writing-tests skill (src/kiro_crew/builtin_skills/kirocrew-dev/writing-tests/SKILL.md).

Backwards compatibility

Breaking: spawns no longer read the posture tier, so an operator who raised agent.resource_critical_gb above agent.spawn_min_memory_gb (or set the floor to 0) no longer has spawns held in that band; only the floor and the adaptive cap gate them, and cron's every/at firings still defer on posture (a cron-expression job never did). This is an accepted change (rfc-overload-resilience Q9). Also removed: the posture_critical spawn wait reason, the refused_low_memory, refused_memory_critical and deferred_memory_critical SEL outcomes, and kiro_crew.subagent.cached_admission_check are removed. A spawn that used to be refused for memory now returns a queued id. Writer sweep on origin/main @ 6ce8c4b: the only readers are the ones listed below (a queue-depth test that deferred a spawn through the posture fake now uses a floor reading).

Reader: src/kiro_crew/subagent_wait_reasons.py:DEFERRED_QUEUED_REASONS -- subagent -- dashboard/handlers/messaging.py and messaging/commands.py read the set; low_memory, adaptive_cap_zero and memory_pressure remain (test_the_wait_reasons_are_the_complete_set)
Reader: website/src/pages/chat/subagentQueuedReason.ts:parseSubagentQueuedReason -- chat -- an older gateway's posture_critical label parses to undefined and the chip keeps its default text (ActivityViewer.test.tsx)
Reader: src/kiro_crew/taskq/adapters/runner.py:cached_admission_check -- cron -- the runner imports its own from resource_status, so the cron posture defer is unchanged (test_taskq_admission_integration.py)
Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_POSTURE_CRITICAL -- release -- release/0.8.0 keeps its own copy until the G6-080 backport

Tests

  • test_spawn_admission_verdict_census.py (new): every outcome, wait reason and refusal text is accounted for. Every capacity condition queues (memory_pressure included), only policy refuses, the gate never reads the posture, and the read's re-entry is not a second submission.
  • test_spawn_floor_defaults.py: the gate reads memory, and registers a no-row spawn's parent wait, off the loop. A memory wait woken early still starts.
  • test_spawn_floor_defaults.py (in-dispatch hold): a Stop by each of the four stop paths while a blocked policy read holds the dispatch, a Stop while a blocked resume grant of the same pass holds the popped row, and a raise in either place (the row goes back to the window and starts later), each with the task queue off and for an incognito row beside a store. The barriers are threading/asyncio events, never sleeps. The stuck-read top-up test also asserts the read is submitted once.
  • test_fairness_lanes.py: low memory queues every subagent and defers only a durable row; a drain before the stamp neither starts the wait nor reads memory, and the pick never asks the pressure hold about it.
  • test_subagent_spawn_host_pin.py: a floor above the pinned host is refused, and one it clears is admitted.
  • test_subagent.py, test_admission_gate.py, test_overload_integration_glue.py and integration/test_governance.py: the former refusals now expect a queued low_memory wait. The legacy-entry case expects the store wording and code.
  • Frontend: ActivityViewer.test.tsx and subagentActivity.test.ts check that a posture_critical label falls back to the default text.

Manual verification

N/A — the verdicts are pure gate logic, covered by the census and by the integration governance test run in-process with KIROCREW_INTEGRATION=1.

Screenshots / video

Why no screenshot: a current gateway never sends posture_critical, so no chip renders differently. Only an older gateway's label now falls back to the existing default text, which ActivityViewer.test.tsx asserts.

Related Issues

Part of #16480

Builds on PR #13109: triage orders it before G6, and it is still open. U2, the other dependency, is commit 1 of this PR.

Dependency notes:

Pattern harvest

Rule candidate: review-prompt
Pattern: a capacity condition (memory, posture, cap) answered with a refusal instead of a wait; the census test is the guard.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team October 3, 2026 01:47
@bolichen97
bolichen97 requested a review from a team as a code owner October 3, 2026 01:47
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 7dff121105c2feb387e587bd8929dcff508dfdad — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A non-durable row popped by the pump now lives only in _undurable_in_dispatch, so any cancel path added later that skips that set will let a stopped spawn start.

Watch

  • The in-dispatch hold works by listing its readers by hand. "cancel (through _unqueue), Stop all, the parent-end snapshot and a stage-boundary cancel all find it there." The cause is that the window no longer holds the row's only copy. Each stop path has to be taught about the second location. A future stop path, or a new await in the pump that skips the post-await re-check, finds nothing to cancel, so a stopped incognito or task-queue-disabled spawn starts anyway. No single test makes "every stop path consults the hold" fail when a path is added.
    Clears when: one helper owns lookup and removal of a non-durable queued row across the window and the hold, every stop path calls it, and a test fails when a stop path reads the window directly.
  • The description is cut off inside ## Backwards compatibility, so I could only check its reader list up to that point. In the tree I found no unlisted reader that breaks. test/test_subagent_sizing.py still emits posture_critical as an opaque string, which looks harmless.
    Clears when: the full Backwards compatibility section is readable within the description limit.

[DESIGN-REVIEWED] 7dff121

Description read

[DESCRIPTION-READ] 035bac66f131ed28f25d58ec66c547c650a45a633b9c367a21f5e523dd678878

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — 🟡 CONCERNS

UX-level review of 7dff121105c2feb387e587bd8929dcff508dfdad — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

When a memory reading times out, the queued chip says "not enough free memory; free up memory to continue", which is false: nobody measured the memory.

Watch

  • The chip blames memory when the reading never answered. Cause: in gate.py, a start whose off-loop reading returns MEMORY_CAUSE_READ_UNANSWERED now waits as low_memory with no available_gb (**({} if memory_cause == MEMORY_CAUSE_READ_UNANSWERED else {"available_gb": avail_gb})). Mechanism: queuedWaitText falls back to low_memory_no_figures, which reads "waiting to start — not enough free memory; free up memory to continue". What the user experiences: on a host with plenty of free memory, they close apps to fix a shortage that isn't there, and the subagent still waits. The backend already knows the cause (its log says "memory headroom unknown: the host memory reading did not answer in time"), but the chip states a shortage instead. Severity: low frequency (only a stalled cgroup read) × misleading wording × repeats on every re-check while the reading stalls. Smallest fix: pass cause through on the wait event and give it its own string, e.g. "waiting to start — couldn't read free memory; retrying".

Suggestions

  • pages.chat.subagentQueued.low_memory_no_figures: limit this text to waits where memory was actually measured, and give the unanswered-reading case and the existing unreadable-cgroup case a separate "memory reading unavailable" string, so neither one tells the user to free memory.

[UX-REVIEWED] 7dff121

Description read

[DESCRIPTION-READ] 888376248e81a05851277171ced3cfa14cf01f37c540ecc4eb1921a9a2f9a5cf

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 7dff121105c2feb387e587bd8929dcff508dfdad and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7dff121

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 7dff121105c2feb387e587bd8929dcff508dfdad: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 7dff121105c2feb387e587bd8929dcff508dfdad — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The new off-loop floor read rebuilds a cached off-thread reader that already exists in resource_status.py, and that reuse was never weighed against the split-and-re-enter machinery.

Not justified as shipped

  • Item 3 — duplicate of src/kiro_crew/resource_status.py:cached_admission_check: that helper already gives event-loop callers a single-flight, off-thread memory reading ("an admission decided on the gateway event loop reads the last off-thread verdict instead of probing inline"). The PR builds a second one, and it needs about 9 new pieces of surface to do it: MemoryReadPoint, _stop_before_memory_read, _memory_reading, _spawn_after_memory_read, _HOST_READS_IN_FLIGHT, _undurable_in_dispatch, _is_undurable, _requeue_undispatched and MEMORY_CAUSE_READ_UNANSWERED. Most of the in-dispatch hold exists only because the read became a re-entry with awaits around it.
    Subtraction: replace the re-entry split with a cached floor reading refreshed on the existing refresh thread. The bar would still be recomputed on the loop against that reading, so items 5 and 6 go away.
    Clears when: the description shows that a TTL-cached reading plus the start reserve would admit a start the fresh read refuses, or the split is replaced.

What this change ships

Inventory (8 items) — 7 justified

Intent: let a subagent spawn that doesn't fit in memory wait in the queue and start later, and stop reading host memory on the gateway's event loop. FIX.

  1. A spawn with no durable row that doesn't fit the memory floor now queues as low_memory instead of being refused — justified
  2. Spawns are no longer held at posture_critical, and its chip text is gone — justified
  3. The floor is read on a worker thread, and the gate is re-entered with the reading — duplicate of resource_status.cached_admission_check
  4. A read that doesn't answer in time now waits, with no on-loop fallback (the dedicated top-up included) — justified
  5. A hung cgroup read takes one executor thread, not one more per re-check — justified
  6. Stopping a non-durable row mid-dispatch now works, and a dispatch that raises puts the row back in the queue — justified
  7. A durable defer the store couldn't write is refused as a task-store failure — justified
  8. Test hosts pin 8 GB, plus a census test of every spawn outcome — justified

[FIRST-PRINCIPLES-REVIEWED] 7dff121

Description read

[DESCRIPTION-READ] a4a6b2ef85e78830d723d840c65436ef920e4af0758c460abbf3877affb52e89

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 7dff121105c2feb387e587bd8929dcff508dfdad — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7dff121

Verdict parsed from the review's SHA-scoped output markers for commit 7dff121105c2feb387e587bd8929dcff508dfdad.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 7dff121105c2feb387e587bd8929dcff508dfdad: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from 96cba66 to 34dbeaf Compare October 3, 2026 02:28
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • fixed in 34dbeaf: the gate's off-loop read has no on-loop fallback (span=599ab81e58eb)

Upheld and fixed. _host_memory_reading_off_loop no longer re-reads on the loop when the worker misses _HOST_READ_OFF_LOOP_SECS or to_thread raises RuntimeError. It returns (-1.0, MEMORY_CAUSE_READ_UNANSWERED), and the gate treats that cause as a start that does not fit yet: it waits as low_memory ("memory headroom unknown: the host memory reading did not answer in time") and is re-checked after the admit wait.
This also avoids the cost the adjudication pointed out in the suggested fix. An unanswered read is NOT the reader's "unmeasurable" -1, so the floor does not fail open under the load that caused the timeout. The wait label leaves out available_gb when nothing was read, and the Linux "guard could not run, proceeding unchecked" audit now runs only for a start that actually proceeds.
The dedicated top-up's poller (_host_available_gb_off_loop) is back to byte-identical with origin/main. Its fallback predates this PR and is outside its diff.
Test: test_spawn_floor_defaults.py::test_an_unanswered_gate_read_waits_and_never_reads_on_the_loop[persistent|incognito x stuck|refuses-a-thread] asserts no check_memory_available call ever runs on the loop thread, that the start queues as low_memory on an ample host, and that it starts on the re-check once the pool answers. All 4 cases fail on 96cba66 (the fallback reads inline and starts it). Spec: subagent.md "The floor is read off the event loop".

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • declined: no shipped client can receive posture_critical from a gateway (span=f6bee0cb752e)

The SPA that runs this parser is always the one the gateway serves. It is built into src/kiro_crew/static/dist/ and ships in the same wheel, and the Electron shell loads the dashboard from the gateway's own URL (website/electron/gateway-supervisor.js, dashboardEntryUrl(targetBackendUrl, ...)). So a client running this code never talks to an older gateway that still emits posture_critical. The only real skew goes the other way, a stale tab running the OLD SPA against this gateway, and that tab never sees the label because this gateway no longer emits it.
Keeping a mapping would keep a removed wait kind alive in the client, with a test pinning it, for a pairing that cannot happen. The unknown-kind fallback is the same backward-compatibility rule the file header already documents for any label the client does not know. The test pins that rule (ActivityViewer.test.tsx, subagentActivity.test.ts).

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from 34dbeaf to 79dbe23 Compare October 3, 2026 04:15
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • F1 fixed in 79dbe23: the memory read's re-entry re-runs the policy gates (gate.py:386)

_gate no longer carries not _memory_reentry; the batch count and the taskq_accept write still skip on the re-entry, so it is still one submission. A spawn admitted by governance before the off-loop read and forbidden during it is now refused. A drained row (_from_queue) refused there goes through _refuse_row, which fails the row. spawn_async passes the _parent_spawn_policy it read off the loop to the re-entry on the no-store and non-persistent paths, so the allowlist vet does not scan the parent spec on the loop. The pump already passed it. A row spawn_async committed (_store_accepted) keeps the documented skip, because a refusal there would strand a committed row.
Tests: test_spawn_floor_defaults.py::test_governance_tightened_during_the_read_still_refuses[incognito|no-store] and ::test_governance_tightened_during_a_drain_read_fails_the_row. Each parks the read, tightens _vet_spawn_governance during it, then releases it. test_spawn_admission_verdict_census.py::test_the_memory_read_re_entry_is_not_a_second_submission now pins the re-vet and that the parent spec is never read on the loop. All of them fail on 34dbeaf. Spec: subagent.md, "The floor is read off the event loop".

  • F2 fixed in 79dbe23: a committed row refused because admission closed is failed (subagent.py:5223)

When gateway admission is closed and the entry carries _store_accepted (and no _claimed), the gate now fails the row (taskq_fail) in the same step that returns the refusal. The caller and the store therefore give the same answer, and the row cannot run once admission reopens. I chose to fail the row rather than return it as queued, because the caller of a closed gateway is told it was refused, and that is what admission-closed means on every other entry.
Test: test_spawn_floor_defaults.py::test_admission_closing_during_the_read_fails_the_committed_row. It closes admission while the read is parked, asserts the refusal, asserts the row is failed, then reopens admission, drains, and asserts nothing started. It fails on 34dbeaf with "the refused row is still runnable".

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed in 79dbe23: agent.spawn_min_memory_gb help no longer promises a refusal (config/sections.py:1190)

The help now reads "A spawn that does not fit waits in the queue (in memory when it has no durable row) and is re-checked after admit_wait_secs." config-baseline.json was regenerated with scripts/generate_config_baseline.py, and test_config_baseline.py passes.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • Watch 1, the posture drop for divergent configs: accepted and named in the Breaking list

The PR description's Breaking paragraph now states it plainly. An operator who raised agent.resource_critical_gb above agent.spawn_min_memory_gb, or set the floor to 0, no longer has spawns held in that band. Only the floor and the adaptive cap gate spawns, and cron still defers on posture. This is the change rfc-overload-resilience Q9 records ("until posture stops gating spawns"). A max(floor, critical) bar would bring back the double hold this PR exists to remove, so this disposition takes the alternative clearing condition: the change is named as accepted.

  • Watch 2, the no-store wake still reads on the loop: accepted as a residual, and the goal line is corrected

With agent.task_queue_enabled=false, the in-memory queue's synchronous _drain_queue still reads the floor on the thread that calls it. subagent.md already records this ("the inline pump used without a store, still read on the calling thread"). The PR goal line now names the event-loop entries this PR covers (spawn_async, the coroutine pump, and the dedicated top-up's poll) and states the no-store residual as accepted, so it no longer overclaims. Routing the no-store wake through the coroutine pump changes how the in-memory queue drains, and that belongs in a follow-up. 79dbe23 also removes the top-up's on-loop fallback (_host_available_gb_off_loop is gone), so the PR's own paths have no on-loop read left.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • declined: an unanswered read keeps the figure-less low-memory sentence

The chip's low_memory_no_figures sentence is the one main already shows for the other "headroom unknown" cause (MEMORY_CAUSE_CGROUP_USAGE_UNREADABLE). This PR adds no new chip state.
The read only goes unanswered when a worker thread misses _HOST_READ_OFF_LOOP_SECS (2 s) or cannot be started. That only happens on a host under heavy pressure, so "free up memory to continue" is still the useful action.
The precise wording is not lost. It rides on queued_reason_detail ("memory headroom unknown: the host memory reading did not answer in time"), which /api/spawn and the parent model receive. As of 79dbe23 the gate also leaves out available_gb, in both the wait label and the SEL row, when nothing was read (pinned by test_only_a_read_that_answered_unmeasurable_proceeds_unchecked[unanswered]).
A separate chip sentence would need a new cause field on the wire and a new string in all 13 locale catalogs, for a transient state. That is outside this PR's capacity-verdict scope.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Maintainer decisions on the judgement calls in this round:
(1) When admission closes during the memory read on an already-committed row, the row is failed through the store, so the store carries the answer the caller saw. Accepted.
(2) Design watch 1: spawns no longer hold in a raised resource_critical_gb band. Accepted as the intended change: posture is not consulted for spawns (only the memory floor is), and cron still reads posture. It is named in Breaking.
(3) Design watch 2: the no-store synchronous drain (task_queue_enabled=false) still reads memory on the calling thread. Accepted as a stated residual; moving that drain onto the coroutine pump is a follow-up.

@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from 79dbe23 to 358b7a5 Compare October 3, 2026 06:15
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • F1 follow-up, fixed in 358b7a5: the committed-row path is re-vetted too (gate.py _gate)

The 79dbe23 fix for F1 still skipped the policy gates on the default path (persistent memory plus a durable store, _store_accepted), so a spawn governance forbade during the off-loop read still started there. _gate is now _claimed is None and (not _store_accepted or _memory_reentry): the memory read's second half re-runs governance, cwd and the parent-spec allowlist on every path. A refusal of a committed, unclaimed row fails it through _refuse_row -> taskq_fail, as the admission-closed branch already did, so the refused work cannot run later. _spawn_async_accepted hands the re-entry the _parent_spawn_policy read off the loop, so the vet does not scan the agents directory on the loop. The one entry that still skips the gates is the committed row's first pass right after prepare_spawn; it never starts anything itself (it stops at the read or leaves the row queued, and both re-vet).
Tests (each fails on 79dbe23): test_spawn_floor_defaults.py::test_governance_tightened_during_the_read_still_refuses[persistent] (refused, row failed, nothing starts after a drain), test_spawn_admission_verdict_census.py::test_the_memory_read_re_entry_is_not_a_second_submission[persistent] (vetted twice, parent spec never read on the loop), and test_overload_integration_glue.py::test_store_accepted_row_is_re_vetted_after_its_awaits, which replaces main's test_store_accepted_reentry_skips_the_mutable_gates because that test pinned the skip. Spec: subagent.md, "The floor is read off the event loop" and the spawn_async bullet, which no longer says the gates run "once per submission".

Also in this push: rebased onto origin/main 6ce8c4b. Main's #16191 test_queue_depth_reconcile.py deferred spawns through the posture fake (subagent.cached_admission_check, which this PR removes), so overload_fakes.memory_critical is now memory_below_floor, a floor reading. That file's ten-running stop test now reads a 64 GB host, because ten unsettled starts do not fit the honest 8 GB pin. The docstring of test_only_a_read_that_answered_unmeasurable_proceeds_unchecked now says only what the test asserts.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • accepted, fix in the next push: a disabled floor (agent.spawn_min_memory_gb = 0) must not wait on an unanswered read (gate.py:862)

Confirmed: with the floor at 0, min_mem is 0 (the start reserve is added only under elif min_mem > 0), yet the gate still stops at a MemoryReadPoint, and an unanswered read makes mem_ok False, so the spawn waits as low_memory although the check is documented as disabled. Before this PR check_memory_available(min_gb=0) always admitted. The fix: when min_mem <= 0, settle mem_ok, avail_gb, memory_cause = True, -1.0, "" before the _memory_reading / _stop_before_memory_read branches, so no read is taken and nothing can defer. The test sets the floor to 0, makes the worker read unanswered (the pool raises), and asserts the start runs. This round's single push had already landed when the finding arrived, so the fix ships in the next push.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

A spawn the floor holds back now waits on every path. A durable spawn already waited without a bound before this PR (the deferred row is re-parked on each admit wait). #16347 ("bound how long a subagent spawn waits for memory") is the issue that ends the wait and tells the parent why. It covers a host that can never fit, since a floor above total memory is one way a host "stays short". Bringing back a capacity refusal for one sub-case would put back the refusal-versus-wait split this PR removes, so it is not added here.

  • Watch 2, non-durable waits lost on restart: accepted, a subagent.md line lands in the next push

Correct: an incognito, temporary or no-store start waiting as low_memory sits only in the in-memory window, and a gateway restart drops it, just as it drops any in-memory queued entry today. The line will say what the parent sees: the id it was handed is gone after the restart, and no completion arrives for it. That is the existing contract for every non-durable queued start, now extended to memory waits. This round's single push had already landed when the review arrived.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

Superseded disposition record: its marker was retired because it broke the one-finding-per-record rule. The rationale stays below for history. The current records are posted separately.

  • Item 2, spawns stop reading the posture tier: a recorded maintainer decision

The maintainer accepted this on the PR: #16355 (comment), decision (2): "posture is not consulted for spawns (only the memory floor is), and cron still reads posture". It is named in the Breaking list. An override on this head is requested from the orchestrator, citing that decision.

  • Item 4, the off-loop memory read: no measured stall, left for a maintainer decision

True: the PR links no issue and no measurement of a stall from the on-loop cgroup read. The split follows the rule this admission path already keeps for its store I/O (taskq.md: the window decision, the claim and the defer run on the writer thread, and _spawn_async_accepted does no store I/O on the loop), and it puts a bound on the one remaining filesystem walk in the gate. Whether that is enough reason without a measurement is a maintainer call. It is not rebutted here.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Maintainer decisions:

  • First-principles item 4 (off-loop memory read): kept. Reading host memory off the event loop is a requirement of the redesign plan (no synchronous probe on the loop), not a reaction to a measured stall. It is the shape the existing wait already uses (test_the_wait_reads_memory_off_the_event_loop).
  • Design watch 1 (a floor above total RAM waits without a bound): accepted as tracked by feat: bound how long a subagent spawn waits for memory #16347, which bounds every memory wait with agent.subagent_queue_max_wait_secs.

@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from 358b7a5 to 7dcdd14 Compare October 3, 2026 07:30
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed in 7dcdd14: a disabled floor takes no reading, so an unanswered read cannot defer the start (gate.py:862)

When agent.spawn_min_memory_gb is <= 0, the gate now sets mem_ok, avail_gb, memory_cause = True, -1.0, "" before the _memory_reading / _stop_before_memory_read branches. No MemoryReadPoint is returned and no read is taken. The Linux "memory guard could not run" warning and its memory_check_unavailable SEL row are skipped for a disabled floor too, because no guard was asked to run. Without that, the -1 placeholder would fire that warning on every spawn. The dedicated top-up already returned early on floor <= 0. test_spawn_floor_defaults.py::test_a_disabled_floor_takes_no_reading_so_an_unanswered_pool_cannot_hold_it[persistent|incognito] uses floor 0, a short host (0.5 GiB) and a pool that raises RuntimeError("can't start new thread"). It asserts that the start runs, that no deferred_low_memory or memory_check_unavailable is written, and that no read was attempted. On 358b7a5 it fails with queued_reason == 'low_memory'. subagent.md § Memory guard now says 0 disables the host reading as well as the floor and the reserve.

@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from bfb67cf to 6a6ce2b Compare October 4, 2026 15:48
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • fixed in 6a6ce2b: src/kiro_crew/subagent_manager/admission/pump.py:152, a non-durable row popped by the coroutine pump is lost if an offload raises before the gate takes it

    Accepted in fix(subagent): queue every capacity verdict and read memory off the loop #16355 (comment); the fix is now pushed in commit 2 (the PR still has 2 commits). A row with no durable record (incognito or temporary memory, or the task queue off) is held in _undurable_in_dispatch from the moment the pass pops it until the gate takes it. That covers the same pass's resume grants (resume_grant_async) as well as the row's own parent_spawn_policy / _check_agent_off_loop / floor reads. If anything in that gap raises, the row goes back to the front of the window, stamped with MEMORY_WAIT_UNTIL_KEY and re-armed after the admit wait (_requeue_undispatched). This happens in _dispatch_async_impl's except for a read, and in the pass's finally for a grant that raised ahead of the dispatch, so the only copy is never dropped.

    Tests, each fails on bfb67cf:

    • test_spawn_floor_defaults.py::test_a_non_durable_row_whose_dispatch_raises_goes_back_to_the_window[incognito|no-store]: the policy read raises RuntimeError("can't start new thread") once, and the row still starts later.
    • ::test_a_non_durable_row_is_held_from_its_pop_through_the_pass_grants[*-it-raises]: a resume grant of the same pass raises while the popped row waits, and the row still starts later. On the old pump the row is lost ("a raising grant lost the row").

@bolichen97

Copy link
Copy Markdown
Contributor Author

Pushed 6a6ce2b. Commit 2 is amended, so the PR still has 2 commits by the same author, and the base is unchanged (ae67568; it merges cleanly). This push carries the three fixes accepted on bfb67cf and the two should-fix items a post-push review raised on the same code path.

  • GPT (blocking), a popped non-durable row lost when an offload raises: fixed. The disposition is the comment above. A raise in a read or in a grant of the same pass re-queues the row at the head of the window, retried after the admit wait.
  • Opus/fable, a Stop missed while the coroutine pump dispatches a popped row: fixed (fix(subagent): queue every capacity verdict and read memory off the loop #16355 (comment)). The fix is the in-dispatch hold. _undurable_in_dispatch holds the row from its pop, so cancel (through _unqueue), Stop all, the parent-end snapshot and a stage-boundary cancel all find it. cancel() answers True, and the pump re-checks the entry after every await and does not start a row a stop took. The hold starts at the pop, not when the row's own dispatch begins, so a Stop that lands while the same pass awaits a resume grant is caught too. The disposition named _stopped_while_claimed; the hold replaces that, because the row has no _agents record to check there.
  • Design, a hung host read takes one more executor thread per re-check: fixed (fix(subagent): queue every capacity verdict and read memory off the loop #16355 (comment)). _host_memory_reading_off_loop is now single-flight per bar (shielded), and the stuck-read top-up test asserts the read is submitted once.

Deterministic barrier tests, all failing on bfb67cf:

  • test_a_stop_reaches_a_non_durable_row_the_pump_is_dispatching: 4 stop paths × 2 modes, with a threading-event barrier inside the policy read.
  • test_a_non_durable_row_is_held_from_its_pop_through_the_pass_grants: a Stop and a raise × 2 modes, with an asyncio-event barrier inside the same pass's resume grant.
  • test_a_non_durable_row_whose_dispatch_raises_goes_back_to_the_window: 2 modes.

The spec paragraph in docs/system-specs/modules/subagent.md and the PR description are updated to match.

Local results: black, subprocess-encoding, isort, flake8, mypy (2102 files), docs-lint and harness-parity are clean. 3472 tests pass across the 94 subagent, taskq, overload, spawn, admission, cancel, fairness and pressure test files, plus 809 in test_session.py and the floor, gate and census files.

@bolichen97

Copy link
Copy Markdown
Contributor Author

@bolichen97

Copy link
Copy Markdown
Contributor Author

@bolichen97

Copy link
Copy Markdown
Contributor Author

The fixture pinned the host reading at 8 GB but answered True for any
floor, so a test whose wave of unsettled starts had outgrown an 8 GB
host still read as admitted. It now answers (8.0 >= min_gb, 8.0), the
real reader's answer on such a host, and a 9 GB floor is refused.
overload_fakes' ManagerHarness pins the same host the same way.

The 2000-submission drain test ran eight unsettled dedicated starts at
once, which the 2 GB floor plus the start price does not fit on the
pinned host; it now runs four, which do. testing-conventions records
the fixture's contract.
A spawn that did not fit the 2 GiB floor was refused when it had no
durable row (incognito or temporary memory, or the task queue off), and
every spawn was also held by the posture tier, which at defaults sits
on the same 2 GiB line, so a host admission had filled to the floor read
critical and deferred every start. The floor was read on the event
loop, walking cgroup files.

Now a capacity verdict never refuses: a start with no durable row that
does not fit waits in the in-memory window, skipped by the pump until
the admit wait passes, with the same low_memory label. Its wake is
armed one clock tick past that stamp and re-arms if it fires early, so
the wait cannot strand. The posture tier is no longer consulted for
spawns (cron every/at firings still defer on it), so posture_critical
is gone as a spawn wait reason. Event-loop entries (spawn_async and
the pump, both
with or without a store, so a timer re-check of an in-memory wait is a
coroutine too) stop at a MemoryReadPoint, read the host on a worker and
re-enter; the fit is decided on the loop against the
bar recomputed then. The worker read has no on-loop fallback: one that
misses its bound, or a pool that cannot start a thread, comes back
unanswered and the start waits as low_memory, re-checked after the
admit wait, never failing open; the dedicated top-up's poll uses the
same reader and keeps waiting on an unanswered read. A disabled floor
(spawn_min_memory_gb 0) takes no reading at all, so an unanswered read
cannot defer a start it does not gate. The re-entry
re-runs the policy gates on every path, the row spawn_async already
committed included, so a governance change made during the read still
refuses; a committed row refused there, or because admission closed
during the read, is failed in the store (the glue test that pinned the
old skip now pins the re-vet). A durable defer the store could not
write is refused as the store's failure (task_store_unavailable), never
with capacity wording. Governance, cwd and memory-identity refusals are
unchanged. A census test pins every outcome on the spawn path, every
wait label and every refusal text the gate can return at defaults.

A row with no durable record that the coroutine pump has popped is held
in _undurable_in_dispatch from its pop until the gate takes it (through
the pass's resume grants and the row's own off-loop reads), as its only
copy. cancel, Stop all, the parent-end snapshot and a stage boundary
cancel all find it there, and the pump re-checks it after each await,
so a stopped row never starts. A raise in that gap, in a read or in a
grant ahead of the dispatch, puts the row back in the window, retried
after the admit wait, instead of dropping it. The host read is single-flight per bar, so a hung reader
holds one executor thread rather than one more per re-check.

With the macOS kernel memory-pressure hold (#15876) both hold: a start
that fits the floor can still wait as memory_pressure in the window,
and one that does not fit waits as low_memory, which outranks the hold
and carries no pressure clock, so the hold's bound never ends it. With
spawns no longer reading the posture, no posture verdict precedes the
hold. The pump's pick does not ask the hold about a stamped floor wait,
and a clock an earlier hold started is dropped when the start goes back
to the floor, so time below the floor never counts toward the hold's
bound. A durable row deferred on the floor keeps only next_run_at in the
store, so the manager remembers it and the window refill stamps it as a
floor wait as well. A spawn with no row awaits its W3 parent
registration off the loop, as the durable path does. The census counts memory_pressure among
the wait labels and the hold's audit rows and "never started" terminal
on the spawn path.

Breaking: the posture_critical wait reason, the refused_low_memory,
refused_memory_critical and deferred_memory_critical SEL outcomes, and
subagent.cached_admission_check are removed. Writer sweep on
origin/main @ 6ce8c4b: the only readers are the ones listed below
(overload_fakes' posture fake, which a queue-depth test used to defer
a spawn, is now a floor reading).
Reader: src/kiro_crew/subagent_wait_reasons.py:DEFERRED_QUEUED_REASONS -- subagent -- dashboard/handlers/messaging.py and messaging/commands.py read the set; low_memory, adaptive_cap_zero and memory_pressure remain (test_the_wait_reasons_are_the_complete_set)
Reader: website/src/pages/chat/subagentQueuedReason.ts:parseSubagentQueuedReason -- chat -- an older gateway's posture_critical label parses to undefined and the chip keeps its default text (ActivityViewer.test.tsx)
Reader: src/kiro_crew/taskq/adapters/runner.py:cached_admission_check -- cron -- the runner imports its own from resource_status, so the cron posture defer is unchanged (test_taskq_admission_integration.py)
Reader: src/kiro_crew/subagent_wait_reasons.py:QUEUED_REASON_POSTURE_CRITICAL -- release -- release/0.8.0 keeps its own copy until the G6-080 backport
@bolichen97
bolichen97 force-pushed the fix/subagent-capacity-verdicts-queue branch from 6a6ce2b to 7dff121 Compare October 4, 2026 22:41
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 4, 2026
@chenmingwei23
chenmingwei23 merged commit 293fe88 into main Oct 5, 2026
98 checks passed
@chenmingwei23
chenmingwei23 deleted the fix/subagent-capacity-verdicts-queue branch October 5, 2026 01:44
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Watch: every stop path has to know about _undurable_in_dispatch, and no test fails when a new path skips it. Accepted as a watch item. The coverage half is now in fix(subagent): share one in-flight host read across every bar #16885.

    Today the hold has four readers: cancel (through _unqueue), Stop all, the parent-end snapshot and the stage-boundary cancel. test_a_stop_reaches_a_non_durable_row_the_pump_is_dispatching drives all four. In fix(subagent): share one in-flight host read across every bar #16885 it also blocks inside the gate's host read and in the policy read with the floor off, so each of the pump's two post-await guards has a test that fails when that guard is removed (mutation-checked, 8/8 each). A single owner helper for "find and remove a non-durable queued row" would be a refactor of the cancellation module. That is outside this fix, and nothing in the tree reads the window directly to cancel today.

  • Watch: the description was truncated inside Backwards compatibility. Noted. The reader list the lane could read found no unlisted reader that breaks.

@bolichen97

Copy link
Copy Markdown
Contributor Author

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Not justified as shipped, item 3: the off-loop read duplicates resource_status.cached_admission_check. Declined. The cached posture verdict cannot gate the floor.

    cached_admission_check returns the last off-thread posture verdict (AdmissionDecision), refreshed on its own cadence. The floor needs a fresh figure, compared on the loop against a bar recomputed at that moment (this start's price plus every row still warming, _startup_memory_reserve_gb). A TTL-cached figure would admit a wave of starts against one stale reading taken before any of them launched, and preventing that is what the warming reserve exists for. That over-admission is the failure fix(subagent): queue every capacity verdict and read memory off the loop #16355 set out to remove. The split's re-entry also re-runs governance after the read (test_governance_tightened_during_the_read_still_refuses). The lane rated the other 7 inventory items justified, and the verdict is advisory.

@bolichen97

Copy link
Copy Markdown
Contributor Author

This PR merged at 7dff121 before the last post-push review items were pushed. They are now in follow-up #16885: one in-flight host read per process, whatever the bar (it had been keyed by min_gb); a test that a stop landing during the host read is honoured; a test of the floor-off guard after the policy read; and the unfinished done-callback comment. Each new test fails against the mutation it guards.

iamwhatever pushed a commit that referenced this pull request Oct 5, 2026
Follow-up to #16355. The off-loop host-memory read was single-flight
per bar (keyed by min_gb), but the reading and its cause do not depend
on the bar, and the bar differs per agent bucket and moves as warming
rows settle. A hung reader therefore still took one more executor
thread for every distinct bar. One read is now in flight per process,
and each caller compares the shared figure against its own bar. The
spec and docstring say so, and the done-callback comment is finished.

Two stop guards in the pump's dispatch of a non-durable row were each
covered only by the other one. The stop test now also blocks inside
the gate's host read (only the check after the read sees that stop)
and in the policy read with the floor disabled (the gate takes no read
and starts the row in the same call, so the pump's check after its
policy read is the only guard). Each variant fails when its guard is
removed, and a new test fails when two bars each start their own read.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants