Skip to content

fix(subagent): bound the pressure-hold wait by queue progress - #18910

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/18891-queue-wait-progress
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/18891-queue-wait-progress

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A held subagent wave that keeps starting members no longer ends its tail as "never started", and members that do end that way are named as not run in the wave digest the parent reads.

On macOS under memory pressure WARN, a spawn_run wave of 84 short tasks drained at a held cap of 4. A new task started every 30 to 90 s the whole time, yet 28 tasks ended "never started" after 1844 to 2025 s, which is agent.subagent_queue_max_wait_secs (1800 s) counted from submission. The wave-close digest then read as a finished wave, and the parent only found the gap when output files were missing.

Why it matters

Any held wave that takes longer than the bound to drain silently loses its tail: work is dropped while the queue is healthy, and the parent believes it ran.

Not a goal

  • Changing the episode bound (a hold that applies without a break for the bound still ends new starts at once).
  • Changing the memory-floor deferral bound (it counts only time a row is parked on the floor, which a waiting tail row is not).
  • Retrying never-started members automatically. The parent decides whether to resubmit.
  • Changing the run card or the structured batch_finished / wave meta tallies.

What changed (motivation → approach → change)

Root cause: _memory_pressure_holds_impl (subagent_manager/admission/gate.py) stamps a row's clock on its first hold and expires it once the bound has passed since that stamp. The pump's pick classifies every root row in the window, so all rows are stamped at about submission.

Change:

  • The bound counts from the later of the row's first hold and its LANE's last root start. The lane is the parent session the row waits behind (taskq.lanes.lane_key_for), the lane the pump's weighted pick serves FIFO. A root start in that lane is a row ahead of this one leaving the queue, whichever spawn_run wave it came from, so a second wave or an unbatched spawn behind a draining wave waits its turn too. Another lane's starts never extend a row. Automation keys (cron, hook, heartbeat, background) all map to SYSTEM_LANE, one FIFO queue, so they share one progress lane; accepted and documented. A missing stamp is its own case (no 0.0 sentinel; a young runner's monotonic clock is below the bound).
  • State on SubagentManager: _pressure_hold_lanes (row → lane), _pressure_lane_held (lane → held rows) and _pressure_lane_started_at (lane → last root start). _pressure_lane_started stamps only a lane that holds a row; _forget_pressure_clock (every exit via _forget_pending_start, a floor wait taking over, the prune) drops the row, and the lane's stamp with its last row. Stamped where a root registers in the gate and where the pump releases an approved root (nested starts skip the hold, so are not progress).
  • The stale-clock prune drops a row's clock by liveness only: never a row still in the window, nor one whose lane started a root within the prune window, so a live row's expiry audit (waited_secs, the WARNING) keeps its whole wait.
  • Wave digest (slack/gateway.py): a failed member whose error opens with QUEUED_WAIT_EXPIRED_TEXT is counted. After the tally sentence (the card and notice regexes anchor on ... of N agents.) the line adds "K of the ❌ never started, so their work was not done: resubmit them once memory eases if it is still needed." Rows the spent episode ended at once are counted too, hence "once memory eases": a resubmit during that episode is ended at once. A refusal is not counted.
  • Docs and spec state the per-lane bound, the prune rule and the digest wording.

Why per lane, not per wave: the previous head keyed on batch_id, so a row behind a draining wave in its own FIFO lane got no credit and expired while the queue moved (the issue, one wave over). Keying on the lane also drops the reliance on a caller-supplied id. Why not one manager-wide stamp: a row whose chat keeps losing its lane share would wait with no limit while other chats start. Why not clock from the head of the queue: the window is a weighted round-robin, so there is no single head.

Deliberate trade-off: a row has no absolute ceiling while its lane keeps progressing (worst case about the rows ahead of it times the bound). What the bound protects against is a lane that has stopped; that still ends what it holds after one bound.

Design note (R3, R5.1): the maps are written only on the event loop by the hold check and the two start sites; no await is added. They hold at most one entry per held row, keyed by the parent key that row already carries, and a lane entry leaves with its last row.

Backwards compatibility

Compatible: no key, field, or event changes; the digest gains one sentence after the existing tally sentence, and every parser anchors on the unchanged prefix (website/src/pages/chat/subagentCompletion.ts, src/kiro_crew/dashboard/system_notices.py). Removes nothing: the expiry still ends a stalled lane and a spent episode.

Tests

  • test_a_root_start_in_its_lane_restarts_a_held_rows_bound; test_a_row_behind_a_draining_wave_in_its_own_lane_is_not_expired[second-wave|unbatched] (the independent review's repro, end to end).
  • test_a_held_wave_draining_slower_than_the_bound_never_expires: also starts with an episode older than the bound; the lane stamp leaves with its last held row.
  • test_a_lane_keeps_no_stamp_once_it_holds_no_row, test_the_stale_prune_drops_only_rows_that_left, test_a_live_held_rows_clock_survives_a_pause_in_a_progressing_lane (review repro).
  • test_automation_keys_share_one_progress_lane, test_a_young_monotonic_clock_still_expires_a_row_with_no_lane_stamp, test_the_gate_judges_a_held_start_by_its_own_lane, test_a_nested_start_is_not_progress_of_a_held_lane, test_releasing_an_approved_root_member_is_its_lanes_progress[root|child].
  • test_wave_close_digest_names_members_that_never_started[False|True]: one ran, one failed after running, one never started, one refused; "1 of the ❌ never started" and "resubmit them once memory eases"; the card regex still parses the tally.

Mutation checks, each red: key the stamp per wave (9 tests and both repro cases); prune by age alone (2 tests and the repro); plus the earlier ones (drop the max(...), 0.0 sentinel, stamp nested starts).

Run locally on the final commit, rebased on main cbf2b36 (CI on main green there; clears the stale-base ui-index, GuideLayer and Windows spawn-publication reds):

  • test_admission_gate.py, test_subagent_scale.py and the admission/queue neighbours (fairness lanes, verdict census, queue max wait, memory wake, startup pressure, taskq admission, spawn gate, per-task overrides, subagent coverage): 877 passed.
  • All test_*ratchet*.py, test_agent_refactor_facade.py, test_error_code_contract.py, test_docs_*.py: 652 passed.
  • black, isort, flake8, mypy on the touched modules, docs_lint.py, comment-history from the merge-base, brand and feature-map: all pass.
  • The previous head also ran every importing test file (50909 passed; 34 host-environment failures, identical on its merge-base) and test_boot_smoke.py (15 passed).

Manual verification

Not run on macOS: the hold engages only there and the author has no macOS host. Tests fake the level and clocks, with no sleeps. A maintainer run of the #18891 held wave would close this.

Related Issues

Closes #18891. Follow-up to #16480 (bounded wait).

Pattern harvest

Rule candidate: review-prompt
Pattern: a per-item timeout in a shared queue measured from enqueue, or keyed on a grouping name, rather than from the last progress of the queue the item actually waits in, so a slowly draining queue expires its tail while healthy.

Checklist

  • One commit, tests, self-review, docs
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 10, 2026 19:14
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 67c7d8e9ec7ca2ccbde25567024b6ee351c5d120 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — 🟡 CONCERNS

Design-level review of 67c7d8e9ec7ca2ccbde25567024b6ee351c5d120 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The new per-lane bound has never run on macOS, the only platform where the hold engages; every test fakes the pressure level and the clocks.

Watch

  • The description says "Not run on macOS: the hold engages only there and the author has no macOS host." The new rule lets a held row wait as long as its chat's lane keeps starting root subagents. It depends on the real pump sending each root start through one of the two places that write the lane's timestamp (gate registration and the approval release). If a real macOS run reaches some other start path, the tail of a wave still ends "never started", which is the subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480) #18891 problem this PR is meant to fix. A revert undoes it, so this is not a block, but neither the change nor the deliberate "no absolute ceiling while the lane progresses" trade-off has been seen in a real run.
    Clears when: a maintainer runs the #18891 held wave (macos-on-demand.yml or a real WARN-level host), and the PR shows the tail rows staying held while their lane is still starting new rows, plus the digest's "K of the ❌ never started" line for any row that did expire.

Suggestions

  • The digest finds never-started members by checking whether the error text starts with QUEUED_WAIT_EXPIRED_TEXT. Reading the structured never_started_memory_pressure outcome the row already has would stop the count from depending on that wording.

[DESIGN-REVIEWED] 67c7d8e

Description read

[DESCRIPTION-READ] 5f6687fd0f33be30ed1b28e8321b20399da2bd350cea7e2f6920bffb629e8441

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 67c7d8e9ec7ca2ccbde25567024b6ee351c5d120 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Not verified on macOS, the only platform where the hold applies. A maintainer should rerun the #18891 84-task held wave and confirm none of the tail ends "never started".

What this change ships

Inventory (7 items) — 7 justified

Intent: stop a held subagent wave that is still draining from dropping its tail as "never started", and tell the parent which members never ran. This is a FIX.

  1. A held start's 30-min bound now restarts each time a top-level start in the same chat's queue begins — justified
  2. Cron, hook, heartbeat and background spawns share one progress queue, so a start from any of them extends all of them — justified
  3. The stale-clock cleanup no longer drops the clock of a row still queued, or of a row whose queue is still starting — justified
  4. Starts released after approval count as progress for their queue; nested starts do not — justified
  5. The wave digest gains "K of the ❌ never started… resubmit them once memory eases" after the tally — justified
  6. The user docs and config reference now say the bound counts time with no start from the same chat — justified
  7. The pin "the bound holds for the whole wave" is rewritten; the description names the trade-off and subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480) #18891 shows the defect — justified

[FIRST-PRINCIPLES-REVIEWED] 67c7d8e

Description read

[DESCRIPTION-READ] 16422a1a40bb01b95c40fa25a3f6f742b0487592b58391853e1114b88f6974ee

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 67c7d8e9ec7ca2ccbde25567024b6ee351c5d120 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 67c7d8e

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 67c7d8e9ec7ca2ccbde25567024b6ee351c5d120 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 67c7d8e

@jeeshofone
jeeshofone force-pushed the fix/18891-queue-wait-progress branch from 1bb1805 to 25a2cb6 Compare October 10, 2026 19:55
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design disposition, head 25a2cb6.

Watch 1 (refusals counted as resubmittable): fixed. The digest now counts only members whose error opens with QUEUED_WAIT_EXPIRED_TEXT, which both memory-wait expiry texts do (slack/gateway.py). A refused spawn is not counted. test_wave_close_digest_names_members_that_never_started now includes a refused member and asserts the count stays 1; a mutation that counts every failure goes red. The _never_ran extraction is reverted, so _injection_notice_outcome is unchanged.

Watch 2 (bounded wait becomes bounded stall): fixed by the second option you named. The clock now resets only on a start from the row's own wave: _pressure_wave_started_at maps batch_id to that wave's last root start, and the hold reads get(batch_id). Another chat's starts no longer extend a row, so a row that keeps losing its lane share cannot wait without limit. A held start's total wait stays finite: at most one bound past its own wave's last start (a wave has batch_total members), or one bound from its first hold when it is outside a wave. The description and spec now say this. test_a_start_of_its_own_wave_restarts_a_held_rows_bound asserts that another wave's start does not save the row; a mutation that reads any wave's stamp goes red. The map is popped in finalize_batch and pruned with the stale pressure clocks.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

GPT disposition, head 25a2cb6.

FINDING pump.py:1392 (an approved nested entry leaving the scan through the child break stamped progress): fixed. The release stamp is now if info.batch_id and not self.entry_is_child(params), and it writes the member's own wave, not a global. test_releasing_an_approved_root_member_is_its_waves_progress[child] asserts a child release stamps nothing; removing the guard goes red.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Opus disposition, head 25a2cb6.

FINDING configuration.md (it said "no other subagent start", but the code moves only on root starts): fixed. The bound is now per wave, and both docs say exactly that: "with no other top-level start of the same spawn_run batch beginning in the meantime" (configuration.md) and "with no other top-level start of its spawn_run batch beginning in that time" (subagents.md). test_the_user_docs_state_the_hold_bounds_the_code_uses pins the subagents.md wording.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles disposition, head 25a2cb6.

Punchline (episode bound in the 84-task wave): the episode bound is out of scope here by design and listed under Not a goal. It ends new starts only while a hold has applied without a break for the whole bound. In the reported wave the hold was not continuous: the log shows starts admitted at 01:04 and later, and a start is only admitted when the hold does not apply. Each such break resets the episode in _memory_pressure_hold_impl (level is None resets _pressure_episode_since). The end quoted in the report says "for its whole wait" (the per-row bound this PR changes), not the episode's "ended without waiting". Inventory items 4 and 5 changed this round: the digest now counts only memory-wait expiries, and the shared _never_ran helper is reverted because the digest no longer uses it.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@jeeshofone
jeeshofone force-pushed the fix/18891-queue-wait-progress branch from 25a2cb6 to cdcf824 Compare October 10, 2026 20:44
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 10, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design disposition, head cdcf824.

Watch (the episode bound can still cause the incident): this is answered by a test on this head, and the code needs no change. A held root is only picked on a pump pass where _memory_pressure_hold() returns None. _root_held treats a row as held whenever the level is set and _memory_pressure_holds says "held". When the hold returns None, _memory_pressure_hold_impl resets _pressure_episode_since and _pressure_episode_spent. So every root start in a wave comes right after a read that ends the episode. While a wave keeps starting members, the episode cannot outlive the bound. test_a_held_wave_draining_slower_than_the_bound_never_expires now starts with an episode older than the bound, and asserts _pressure_episode_spent is True before the member starts. After the start and another pump pass, it asserts the two members behind it are still held, none is in _pressure_hold_expired, and _pressure_episode_spent is False. The incident's quoted end line, "for its whole wait", is the per-row bound's expired_by=wait message. The episode's message is "ended without waiting".

Also on this head: a missing wave stamp is no longer a 0.0 sentinel. It reverts to the first-hold clock, which fixed five existing expiry tests that a young CI runner held (see the body).

First-pass check: R1-R8 OK, with one process miss: R8.7 (deterministic tests). A sentinel compared against monotonic time depends on host uptime. It is now pinned by test_a_young_monotonic_clock_still_expires_a_row_with_no_wave_stamp.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles disposition, head cdcf824.

Punchline (worst case about batch_total × the bound): that worst case is acceptable. It needs a wave that starts exactly one member just before each bound runs out. In that case the wave is still draining, and every start is real work. A wave that stops starting ends its held members one bound after its last start, so a stalled wave is still bounded by agent.subagent_queue_max_wait_secs as before. The episode bound, user stop and agent.subagent_queue_max_wait_secs=0 behave as before. The body states the total-wait bound under Related Issues.

First-pass check: R1-R8 OK (R8.7 process miss noted in the Design disposition).

@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles disposition, head cdcf824.

Punchline (is the count sentence needed when each fail line carries the error): yes, and the issue is the evidence for it. The reporter's parent got a wave-close digest whose header said the wave finished, "All results delivered" and "This run is complete", and it treated the wave as done. The 28 never-started errors were listed further down among the fail lines, and the parent did not act on them. The header is what the parent acts on, so the count goes there, right after the tally. It is one sentence and is only added when the count is non-zero. Holding position; no change.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design disposition, head cdcf824.

Punchline (re-run a real macOS held wave): agreed it would be useful, and it needs a maintainer. A fork PR cannot apply the ci:macos label, and no Mac with kernel memory pressure is available on our side for the 84-task WARN repro. The tests drive the same platform_compat level reader and the same pick and release paths with a faked level, so a maintainer run with ci:macos is the remaining check.

CI on this head, none in our diff:

@jeeshofone
jeeshofone force-pushed the fix/18891-queue-wait-progress branch from cdcf824 to bc30cf0 Compare October 10, 2026 23:50
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Review fixes on head bc30cf0 (one commit, rebased on main e6ac772, which is past #18944).

An independent review of cdcf824 found two defects and a NIT. Each one is fixed:

  • Progress keyed on the queue lane, not the wave name (gate.py _pressure_lane_started_impl / _forget_pressure_clock_impl, pump.py release site, subagent.py state). A held row's bound now counts from the later of its first hold and the last root start in its lane. The lane is the parent session it waits behind (taskq.lanes.lane_key_for), the lane the pump's weighted pick serves FIFO. A second spawn_run wave, or an unbatched spawn, queued behind a draining wave in the same chat is no longer ended while that wave keeps starting. Another session's starts never extend a row, so the cross-chat bound stays. The maps hold one entry per held row. A start in a lane with no held row stamps nothing, and the lane's stamp leaves with its last held row. The batch_id key, and with it the prefix-collision NIT, is gone. Tests: test_a_row_behind_a_draining_wave_in_its_own_lane_is_not_expired[second-wave|unbatched] (the review's repro), test_a_root_start_in_its_lane_restarts_a_held_rows_bound, test_a_lane_keeps_no_stamp_once_it_holds_no_row. Mutation (key per wave): 9 tests and both repro cases red.
  • The stale prune no longer drops a live row's first-hold clock. It prunes by liveness: never a row still in the window, nor one whose lane started a root within the prune window. Only a row that left with no registration or refusal is dropped, together with its place in its lane. The expiry audit (waited_secs, the WARNING) keeps the whole wait. The spec sentence is corrected. Tests: test_a_live_held_rows_clock_survives_a_pause_in_a_progressing_lane (the review's repro), test_the_stale_prune_drops_only_rows_that_left. Mutation (age-only prune): 2 tests and the repro red.
  • NIT, spent-episode rows in the digest. These rows stay counted, and the line now reads "resubmit them once memory eases if it is still needed", because a resubmit during that episode is ended at once. subagents.md and the spec say the same.
  • Trade-off, documented in the body and the spec: a row has no absolute ceiling while its lane keeps progressing. The worst case is about the rows ahead of it times the bound. The bound protects against a lane that has stopped.

Main-side: test_approval_decide_owner_bound (#18943) is fixed on main by #18944. This head contains it.

First-pass check: R1-R8 OK. Every gate in the description ran on the final commit.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

GPT disposition, head bc30cf0.

The GPT review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on batch_id. The stale prune is liveness-based. The digest line now says "resubmit them once memory eases". The body and the spec state the deliberate trade-off (no absolute ceiling while a row's lane keeps progressing). The cross-chat bound from the earlier Design round is unchanged: another session's starts never extend a row.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Opus disposition, head bc30cf0.

The Opus review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on batch_id. The stale prune is liveness-based. The digest line now says "resubmit them once memory eases". The body and the spec state the deliberate trade-off (no absolute ceiling while a row's lane keeps progressing). The cross-chat bound from the earlier Design round is unchanged: another session's starts never extend a row.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Design disposition, head bc30cf0.

The Design review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on batch_id. The stale prune is liveness-based. The digest line now says "resubmit them once memory eases". The body and the spec state the deliberate trade-off (no absolute ceiling while a row's lane keeps progressing). The cross-chat bound from the earlier Design round is unchanged: another session's starts never extend a row.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

First Principles disposition, head bc30cf0.

The First Principles review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on batch_id. The stale prune is liveness-based. The digest line now says "resubmit them once memory eases". The body and the spec state the deliberate trade-off (no absolute ceiling while a row's lane keeps progressing). The cross-chat bound from the earlier Design round is unchanged: another session's starts never extend a row.

First-pass check: R1-R8 OK.

…me never-started wave members

The macOS memory-pressure hold ended a held root start once
agent.subagent_queue_max_wait_secs had passed since its first hold, so the
tail of a large held wave expired as never started while the queue was
still starting members. Count the bound from the later of a row's first
hold and the last root start in its lane, the parent session the row waits
behind in the pump's FIFO pick: a lane that keeps starting roots no longer
expires the rows behind them, whichever spawn_run wave they belong to, and
a lane where no root starts for the bound still ends what it holds.
Another lane's starts never extend a row (automation keys share one
lane, as they share one queue), and nested starts skip the hold, so they
do not count as progress. A lane keeps its stamp only while
it holds a row. The stale-clock prune drops a row's clock only once the
row is out of the window and its lane has stopped, so a live row's audit
keeps its whole wait.

The wave-close digest the parent reads now says how many of its failures
never started, after the tally sentence the card parsers anchor on, and
to resubmit them once memory eases, instead of reading a finished wave.

Closes kirodotdev#18891
@jeeshofone
jeeshofone force-pushed the fix/18891-queue-wait-progress branch from bc30cf0 to 67c7d8e Compare October 12, 2026 04:15
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Delta since bc30cf039: new head 67c7d8e9e, rebased onto main cbf2b3614, still one commit.

First-pass check: R1-R8 OK.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Fixed in 67c7d8e9e. The constant comment in subagent.py now says the wait counts with no root start in the row's lane, from the later of its first hold and the lane's last root start. The test docstring that said "its wave's last start" says the same.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Taken in 67c7d8e9e. Both stale per-wave descriptions now describe the lane: the subagent.py constant comment and the test_admission_gate.py docstring. The unused batch_id= argument at the gate's _memory_pressure_holds call is gone; the function reads batch_id only under relabel, which that call never passes.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Removed in 67c7d8e9e. The batch_id=batch_id hunk at the gate's _memory_pressure_holds call was left over from the per-wave head. That call never sets relabel, so the value was never read. 877 admission and queue tests pass without it.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Accepted and documented in 67c7d8e9e, which is this item's first clear condition. The stamp stays keyed on the queue lane. A held row's bound measures how long the queue it waits in has gone without a start. Automation keys all map to SYSTEM_LANE, and the pump serves that lane as one FIFO queue (taskq/lanes.py lane_key_for), so a start from another cron or hook job really is a row ahead of the held one leaving that queue. Keying system-lane rows per session would expire an automation row while its own queue keeps moving, which is the #18891 bug.

  • docs/system-specs/modules/subagent.md and src/kiro_crew/docs/subagents.md now say that scheduled and background jobs share one progress lane and that this is accepted.
  • The code comments and the PR body no longer claim "another session's starts never extend a row". They say "another lane's".
  • test_automation_keys_share_one_progress_lane pins this. A chat's start does not extend a held cron row, and another cron job's start does. Keying per session turns it red.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

Held, and it needs a maintainer. The hold engages only on macOS (platform_compat.memory_pressure_level), and the author has no macOS host. That means the first clear condition, one real macOS run of a held wave that drains for longer than agent.subagent_queue_max_wait_secs, cannot be met from this side. The Manual verification section now states the gap plainly instead of saying N/A.

What does cover it: the new clock logic is platform-neutral, and it runs through the real gate and the real pump in test_a_row_behind_a_draining_wave_in_its_own_lane_is_not_expired[second-wave|unbatched] and test_a_held_wave_draining_slower_than_the_bound_never_expires. Only the pressure level and the monotonic clock are faked. A maintainer with a Mac can close this by running the #18891 wave (84 short tasks, WARN pressure, held cap 4) and checking that no member reads never started: waiting for memory while starts continue. Otherwise, please accept the faked-level tests as enough, which is the item's second clear condition.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

On the PASS punchline, that a held row has no upper limit while its lane keeps starting: this is intended, and both the spec and the body's "Deliberate trade-off" paragraph say so. The worst case is about the rows ahead of it times the bound. The bound protects against a lane that has stopped, and that lane still ends what it holds after one bound. An absolute ceiling would bring back #18891, where a healthy but slow queue expires its tail. Nothing in this PR is gated on it.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On macOS, a start held by memory pressure used to expire after a fixed wait even while its chat's queue kept starting. That is the #18891 tail of rows that "never started". This PR measures the wait from the later of the row's first hold and the last root start in its lane (taskq.lanes.lane_key_for, the same FIFO lane the pump serves), so the root cause is fixed. The wave digest also says "K of the ❌ never started" so the parent resubmits. I checked the Design lane's worry that a real start could skip the stamp. Every root start registers through the gate's registration block (gate.py:1549-1552), including a ClaimPoint re-entry, and an approval-released start stamps at pump.py:1390-1397. Both use the same time.monotonic clock that _memory_pressure_holds_impl compares against, and a nested start is excluded on both sides. The per-lane maps are dropped together with each lane's last held row, so they cannot grow without bound. The Cross-Platform Portability run on 67c7d8e was cancelled during actions/checkout on attempt 2. That is an infra cancel, PR Readiness passed, and the diff has no platform-specific calls, so a rerun before merge is enough. The remaining Design watch (no real macOS run yet) needs a maintainer to apply ci:macos, and the author has already said so. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 12, 2026 17:20
@iamwhatever
iamwhatever merged commit d808fc9 into kirodotdev:main Oct 12, 2026
103 of 104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

subagent: the memory-wait bound starts at submission, so the tail of a large wave times out while the queue is still draining (follow-up to #16480)

2 participants