Skip to content

fix(subagent): restart a held start's pressure wait when another root starts - #18912

Closed
neilieva wants to merge 1 commit into
kirodotdev:mainfrom
neilieva:fix/subagent-pressure-wait-progress
Closed

neilieva wants to merge 1 commit into
kirodotdev:mainfrom
neilieva:fix/subagent-pressure-wait-progress

Conversation

@neilieva

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A large spawn_run wave held by macOS memory pressure runs every member as long as its queue keeps starting, instead of ending the back of the queue as "never started".

I sent one spawn_run batch of 84 short tasks while macOS reported memory pressure WARN. The queue kept draining the whole time, with a new start every 30 to 90 seconds, but 28 of the 84 tasks ended with "never started: waiting for memory" after waiting about 1,800 to 2,000 seconds. The details are in #18891.

Why it matters

Anyone on a Mac under memory pressure who fans out more work than drains in 30 minutes silently loses the tail of the wave. The run looks finished, and the parent has to notice the missing results and resubmit them.

Not a goal

  • Changing the bound itself, its default, or the per-episode bound (_pressure_episode_spent).
  • Changing the memory-floor wait, which already leaves out time spent waiting for a slot.
  • Changing how a wave reports members that never started.

What changed (motivation → approach → change)

The hold gave each queued root start a clock from the first time it held that start. The pump checks every queued root on each pass, so the whole queue's clocks start within seconds of each other. Rows near the back ran out while they were only waiting for their turn behind starts that were still going.

The hold's own wait in subagent_manager/admission/gate.py is now measured from the later of the row's first hold and the last root start the gateway registered. The manager in subagent.py keeps that time as _root_started_at, and the gate sets it where a start is registered. While the queue keeps starting, the rows behind it keep waiting. If nothing starts for a full bound, every held row still ends as before. Progress is counted gateway-wide because the queue is shared, and nested starts don't count since the hold never applies to them. The episode bound is untouched, so one pressure episode that outlasts the bound without a break still ends every held start.

The spec (docs/system-specs/modules/subagent.md) and the user guide (src/kiro_crew/docs/subagents.md) say how the wait is measured.

Backwards compatibility

Compatible: the change can only make a held start expire later, never sooner, and the bound, its config key and the expiry result are unchanged.

Removes nothing: the change only moves when a held start can expire.

Tests

  • test_a_root_start_restarts_a_held_rows_wait: a row held past the bound stays held when a root start happened within the bound, and expires once the last root start is a full bound old.
  • test_a_registered_start_stamps_root_progress: registering a root start records progress, and registering a nested start does not.

Both fail on main.

Manual verification

N/A. The unit tests cover the clock directly, and reproducing the original wave needs a host held at memory pressure WARN for over 30 minutes.

Related Issues

Closes #18891

Follow-up to #16480, which added the bounded wait.

Pattern harvest

Not generalizable: the defect is specific to how one wait clock was anchored, and the codebase has no other per-row clock that the whole queue starts together.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@neilieva
neilieva requested review from a team and KiroCrewCaptain as code owners October 10, 2026 19:19
@neilieva
neilieva requested a review from buluoray October 10, 2026 19:19
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision d67f294f64e251afbd2e1bddd47e8fa5f39d7495 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of d67f294f64e251afbd2e1bddd47e8fa5f39d7495 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

Nobody has checked the fix against the reported wave: if pressure stayed WARN the whole time, the untouched episode bound still ends the same tail at about 30 minutes.

Watch

  • The fix may not reach the reported failure. Cause: the change only moves the per-row clock, now - max(since, mgr._root_started_at) >= bound. The description also says "The episode bound is untouched." Mechanism: _memory_pressure_hold_impl marks _pressure_episode_spent after the hold has applied "without a break" for the bound. During a draining wave, a dedicated runtime of ours is always running, so the episode resets only when the pressure reading itself eases or isn't observed. Consequence: if subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480) #18891's run held WARN without a break, every held row still ends at the 30-minute mark with expired_by: "episode". The test only sets _pressure_holds and _root_started_at directly. "Manual verification: N/A" leaves it unknown which bound ended the 28 rows in subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480) #18891.
    Clears when: the #18891 SEL records (or a log excerpt) show the 28 expiries were expired_by: "wait" with the episode resetting during the wave, or a test drives the hold with a level that flickers and shows the tail survives.
  • The wait can now be extended without limit by another chat's starts. The new _root_started_at clock is gateway-wide, so another chat that keeps starting subagents keeps this chat's held rows waiting with no cap. The docs admit this ("a busy neighbouring chat's root starts also keep this chat's held rows waiting"). That gives up the old guarantee that a held row ends within the bound, and the description doesn't name that loss.
    Clears when: the description or spec says why lane fair-share makes sure the rows in a starved chat still start or end, or the clock is keyed by parent session.

Suggestions

  • A narrower fix may be to restart a row's clock when the hold's episode breaks, since that break is what let the queue start rows. That would aim at the actual cause instead of using "some root started anywhere" as a stand-in for progress.

[DESIGN-REVIEWED] d67f294

Description read

[DESCRIPTION-READ] cc8c01281e29b69ea1d68d172f3598d89eb5c8528bb9459251b78f69c49337e0

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of d67f294f64e251afbd2e1bddd47e8fa5f39d7495 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: claim re-entries (_dispatch_now) skip the hold but still stamp _root_started_at, so they reset held rows' wait. After this change the episode bound is the only cap.

What this change ships

Inventory (5 items) — 5 justified

Intent: when macOS memory pressure holds a large spawn_run wave whose queue is still draining, let every member run instead of ending the back of the queue as "never started" (#18891). This is a FIX.

  1. A held top-level start's own wait now restarts each time the gateway starts another top-level subagent — justified
  2. A neighbouring chat's top-level starts also keep this chat's held rows waiting, because the clock covers the whole gateway — justified
  3. Nested starts do not restart the wait — justified
  4. The manager keeps one new in-memory timestamp, _root_started_at, with one writer and one reader — justified
  5. The subagent spec and the user guide now say how the held wait is measured — justified

[FIRST-PRINCIPLES-REVIEWED] d67f294

Description read

[DESCRIPTION-READ] 83f262fd665f05fc95449be14aac8aabe2442ef786de94662783a0fd0c0991fc

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed d67f294f64e251afbd2e1bddd47e8fa5f39d7495 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] d67f294

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed d67f294f64e251afbd2e1bddd47e8fa5f39d7495 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] d67f294

@neilieva

Copy link
Copy Markdown
Contributor Author

rebutted: every task that ended in that run logged the per-row cause. The gateway log has 27 lines of the form Subagent <id>: never started -- macOS reported memory pressure (WARN) for its whole wait (<n>s), from 01:06:34 to 01:11:55, with waits of 1844 s to 2025 s.
It has no ... has lasted <n>s, past the <bound>s bound, so it was ended without waiting lines, so no task was ended by _pressure_episode_spent.
The episode kept restarting because the hold applies only while a dedicated runtime of the gateway is running. Each time the last one finished before the next start, the read came back unheld and _pressure_episode_since reset.
The per-row clock never reset across those breaks, which is the clock this PR changes.
Correction to the issue and description: the count is 27, not 28. The 28th match was a log line echoing a search command, not an expiry.

@neilieva
neilieva force-pushed the fix/subagent-pressure-wait-progress branch from 859d2f8 to 764066a Compare October 10, 2026 19:50
@neilieva

Copy link
Copy Markdown
Contributor Author
  • Blocker mechanism: a draining held queue breaks the episode at every start

rebutted: a held root start can only begin on a pass where _memory_pressure_hold() returns None. The pump's _root_held and the gate's verdict both release a root only then.
That same read takes the level is None branch at gate.py:1702 and sets _pressure_episode_since = None and _pressure_episode_spent = False.
So while held rows keep starting, the episode restarts at each start and cannot reach the bound. A continuous episode needs a full bound with no held root starting, and that is the case where this PR still ends the rows.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Blocker cause: the untouched episode bound only fires on a stalled queue

rebutted: the description's sentence is accurate and does not defeat the goal. The episode bound spends only after a full bound with the hold applying continuously, which (see span bffe6590f31f) means no held root started in that time.
A wave whose queue keeps starting never reaches it, so leaving it unchanged keeps the existing guarantee for a stuck queue.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Blocker consequence: the reported run ended its tail on the per-row wait, not the episode

rebutted: when an episode spends, the gateway logs macOS memory pressure has held subagent starts for <n>s; root starts it would hold are ended, never started, until it eases once (gate.py:1708).
The #18891 run's gateway log has no such line. It has 27 never started -- macOS reported memory pressure (WARN) for its whole wait (<n>s) lines, from 01:06:34 to 01:11:55, with waits of 1844 s to 2025 s.
The episode never spent during that wave, so the per-row clock this PR changes is what ended those tasks.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Blocker fix: the PR shows why the episode broke

rebutted: this item clears on its second branch, showing why the episode would have broken. Spans bffe6590f31f and 1a0531c5003d give the code path and the run's log evidence.
Restarting the episode clock on root starts would duplicate what the hold already does on every release, so the PR does not add it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@neilieva
neilieva force-pushed the fix/subagent-pressure-wait-progress branch from 764066a to ab5f4a6 Compare October 10, 2026 20:22
@neilieva

Copy link
Copy Markdown
Contributor Author
  • Watch: a busy neighbouring chat can extend a held row's wait

fixed in ab5f4a6: the spec (docs/system-specs/modules/subagent.md, the hold's per-start wait bullet) now says the clock is gateway-wide, so a busy neighbouring chat's root starts also keep this chat's held rows waiting, and that the episode bound is separate and unchanged.
Progress stays gateway-wide because the admission queue and the hold are both gateway-wide.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Item 1: the pick-time call has to seed the clock

rebutted: for a queued root, the pick predicate (pump.py, _root_held with commit_expiry=False) is the only hold check it gets while it waits. pick_window_index skips held rows, and an expired row is the one picked so the gate's re-check ends it.
In the #18891 wave the rows were queued before any dedicated runtime was running, so the gate's submission-time check saw no hold and seeded nothing. Every clock came from the pick.
If the pick stopped seeding, a held queued row would never get a clock, would never classify as expired and would wait without bound. That removes the guarantee #16480 added.
Moving where the clock starts therefore needs some other per-row event, and the last root start is that event: it is the earliest moment the row could have been the next one admitted.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Watch: the gateway-wide stamp is the right scope for a shared queue

rebutted: the admission window and the hold are gateway-wide, so a row waits behind other chats' rows as well as its own. A stamp scoped to the row's parent or batch would let a row expire while the shared queue is still starting other chats' rows ahead of it, which is #18891's defect in a cross-chat form.
The extension lasts only while root starts keep happening. Under the weighted round-robin each lane with eligible rows gets turns, so a throttled lane's own rows start in that stream too. A row that is never picked while others start would be a fairness defect, which the per-row memory bound does not cover today either.
A fully stalled gateway (no root start for one bound) still ends every held row, as before.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Watch: end-to-end evidence comes from the reported run

rebutted: the changed logic is the clock comparison, and the tests cover it directly, including a fail-on-main case. The end-to-end question was how the episode behaves on a real host, and the #18891 run's gateway log answers it: no episode-spent warning, and 27 per-row expiries at 1844 s to 2025 s.
Holding a real macOS host at WARN for over 30 minutes is not reproducible on demand, and a pump-level fake-clock test would re-assert the same comparison through more scaffolding.

@neilieva
neilieva force-pushed the fix/subagent-pressure-wait-progress branch from ab5f4a6 to 516fe92 Compare October 10, 2026 22:28
… starts

The macOS memory-pressure hold ended a queued root start once
agent.subagent_queue_max_wait_secs had passed since the start was first
held. Every queued row is first held at about the same moment, so a large
wave whose queue was still draining ended its tail as "never started".

The manager records when it last registered a root start, and the hold's
own wait is measured from that time or the row's first hold, whichever is
later. A queue that keeps starting no longer expires the rows behind it,
and a queue that stops starting still ends them after one full bound.

Closes kirodotdev#18891
@neilieva
neilieva force-pushed the fix/subagent-pressure-wait-progress branch from 516fe92 to d67f294 Compare October 11, 2026 00:46
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 11, 2026
@neilieva

Copy link
Copy Markdown
Contributor Author
  • Watch: the reported failure was the per-row wait, not the episode

rebutted: when an episode spends, gate.py logs macOS memory pressure has held subagent starts for <n>s; root starts it would hold are ended, never started, until it eases. The #18891 run's gateway log has no such line.
It has 27 never started -- macOS reported memory pressure (WARN) for its whole wait (<n>s) lines (01:06:34 to 01:11:55, waits of 1844 s to 2025 s), so every expiry was expired_by: "wait".
The episode reset during the wave because a held root starts only on a read where the hold does not apply, and that read takes the level is None branch that clears _pressure_episode_since.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Watch: lane fair-share bounds the extension

rebutted: the pump picks lanes by smooth weighted round-robin among lanes with eligible entries (fairness.py, pump.py's lane-aware pick), FIFO inside a lane. A chat with held rows is one of those lanes, so its rows take their turns in the same stream of starts that keeps the clock fresh.
A row's wait is extended only while root starts keep happening, and those starts include its own lane's. Once no root starts for a full bound, every held row ends as before.
A per-parent clock would expire rows that wait behind other chats' rows in the shared window while it is still starting, which is the #18891 defect across chats.

@neilieva

Copy link
Copy Markdown
Contributor Author
  • Suggestion: restarting on an episode break would drop the bound for a flickering hold

rebutted: the existing design keeps a row's clock across pauses in the hold on purpose (gate.py, "A row's clock outlives a pause in the hold"), so a reading that flickers between WARN and normal still ends held rows after the bound.
Restarting on every episode break would reset all held clocks on each flicker, including flickers where nothing started, so a held row could wait without end.
Restarting on a registered root start ties the reset to the queue actually moving, which is the condition the bound is meant to tolerate.

@neilieva

Copy link
Copy Markdown
Contributor Author

Closing in favor of #18910, which makes the same change to the pressure-hold wait and also covers approval-released starts and the wave summary. The log evidence from the #18891 run in the comments above is still here if it helps the review there.

@neilieva neilieva closed this Oct 11, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480)

1 participant