Repository navigation
fix(subagent): bound the pressure-hold wait by queue progress - #18910
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The new per-lane bound has never run on macOS, the only platform where the hold engages; every test fakes the pressure level and the clocks. Watch
Suggestions
[DESIGN-REVIEWED] 67c7d8e Description read[DESCRIPTION-READ] 5f6687fd0f33be30ed1b28e8321b20399da2bd350cea7e2f6920bffb629e8441 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Not verified on macOS, the only platform where the hold applies. A maintainer should rerun the #18891 84-task held wave and confirm none of the tail ends "never started". What this change shipsInventory (7 items) — 7 justifiedIntent: stop a held subagent wave that is still draining from dropping its tail as "never started", and tell the parent which members never ran. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 67c7d8e Description read[DESCRIPTION-READ] 16422a1a40bb01b95c40fa25a3f6f742b0487592b58391853e1114b88f6974ee That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
1bb1805 to
25a2cb6
Compare
|
Design disposition, head 25a2cb6. Watch 1 (refusals counted as resubmittable): fixed. The digest now counts only members whose error opens with Watch 2 (bounded wait becomes bounded stall): fixed by the second option you named. The clock now resets only on a start from the row's own wave: First-pass check: R1-R8 OK. |
|
GPT disposition, head 25a2cb6. FINDING pump.py:1392 (an approved nested entry leaving the scan through the child First-pass check: R1-R8 OK. |
|
Opus disposition, head 25a2cb6. FINDING configuration.md (it said "no other subagent start", but the code moves only on root starts): fixed. The bound is now per wave, and both docs say exactly that: "with no other top-level start of the same First-pass check: R1-R8 OK. |
|
First Principles disposition, head 25a2cb6. Punchline (episode bound in the 84-task wave): the episode bound is out of scope here by design and listed under Not a goal. It ends new starts only while a hold has applied without a break for the whole bound. In the reported wave the hold was not continuous: the log shows starts admitted at 01:04 and later, and a start is only admitted when the hold does not apply. Each such break resets the episode in First-pass check: R1-R8 OK. |
25a2cb6 to
cdcf824
Compare
|
Design disposition, head cdcf824. Watch (the episode bound can still cause the incident): this is answered by a test on this head, and the code needs no change. A held root is only picked on a pump pass where Also on this head: a missing wave stamp is no longer a 0.0 sentinel. It reverts to the first-hold clock, which fixed five existing expiry tests that a young CI runner held (see the body). First-pass check: R1-R8 OK, with one process miss: R8.7 (deterministic tests). A sentinel compared against monotonic time depends on host uptime. It is now pinned by |
|
First Principles disposition, head cdcf824. Punchline (worst case about First-pass check: R1-R8 OK (R8.7 process miss noted in the Design disposition). |
|
First Principles disposition, head cdcf824. Punchline (is the count sentence needed when each fail line carries the error): yes, and the issue is the evidence for it. The reporter's parent got a wave-close digest whose header said the wave finished, "All results delivered" and "This run is complete", and it treated the wave as done. The 28 never-started errors were listed further down among the fail lines, and the parent did not act on them. The header is what the parent acts on, so the count goes there, right after the tally. It is one sentence and is only added when the count is non-zero. Holding position; no change. |
|
Design disposition, head cdcf824. Punchline (re-run a real macOS held wave): agreed it would be useful, and it needs a maintainer. A fork PR cannot apply the CI on this head, none in our diff:
|
cdcf824 to
bc30cf0
Compare
|
Review fixes on head bc30cf0 (one commit, rebased on main e6ac772, which is past #18944). An independent review of cdcf824 found two defects and a NIT. Each one is fixed:
Main-side: First-pass check: R1-R8 OK. Every gate in the description ran on the final commit. |
|
GPT disposition, head bc30cf0. The GPT review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on First-pass check: R1-R8 OK. |
|
Opus disposition, head bc30cf0. The Opus review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on First-pass check: R1-R8 OK. |
|
Design disposition, head bc30cf0. The Design review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on First-pass check: R1-R8 OK. |
|
First Principles disposition, head bc30cf0. The First Principles review on cdcf824 passed with no findings. This head carries the independent-review fixes summarised in the comment above. Progress is now keyed on the queue lane (the parent session) and not on First-pass check: R1-R8 OK. |
…me never-started wave members The macOS memory-pressure hold ended a held root start once agent.subagent_queue_max_wait_secs had passed since its first hold, so the tail of a large held wave expired as never started while the queue was still starting members. Count the bound from the later of a row's first hold and the last root start in its lane, the parent session the row waits behind in the pump's FIFO pick: a lane that keeps starting roots no longer expires the rows behind them, whichever spawn_run wave they belong to, and a lane where no root starts for the bound still ends what it holds. Another lane's starts never extend a row (automation keys share one lane, as they share one queue), and nested starts skip the hold, so they do not count as progress. A lane keeps its stamp only while it holds a row. The stale-clock prune drops a row's clock only once the row is out of the window and its lane has stopped, so a live row's audit keeps its whole wait. The wave-close digest the parent reads now says how many of its failures never started, after the tally sentence the card parsers anchor on, and to resubmit them once memory eases, instead of reading a finished wave. Closes kirodotdev#18891
bc30cf0 to
67c7d8e
Compare
|
Delta since
First-pass check: R1-R8 OK. |
|
Fixed in |
|
Taken in |
|
Removed in |
|
Accepted and documented in
|
|
Held, and it needs a maintainer. The hold engages only on macOS ( What does cover it: the new clock logic is platform-neutral, and it runs through the real gate and the real pump in |
|
On the PASS punchline, that a held row has no upper limit while its lane keeps starting: this is intended, and both the spec and the body's "Deliberate trade-off" paragraph say so. The worst case is about the rows ahead of it times the bound. The bound protects against a lane that has stopped, and that lane still ends what it holds after one bound. An absolute ceiling would bring back #18891, where a healthy but slow queue expires its tail. Nothing in this PR is gated on it. |
iamwhatever
left a comment
There was a problem hiding this comment.
On macOS, a start held by memory pressure used to expire after a fixed wait even while its chat's queue kept starting. That is the #18891 tail of rows that "never started". This PR measures the wait from the later of the row's first hold and the last root start in its lane (taskq.lanes.lane_key_for, the same FIFO lane the pump serves), so the root cause is fixed. The wave digest also says "K of the ❌ never started" so the parent resubmits. I checked the Design lane's worry that a real start could skip the stamp. Every root start registers through the gate's registration block (gate.py:1549-1552), including a ClaimPoint re-entry, and an approval-released start stamps at pump.py:1390-1397. Both use the same time.monotonic clock that _memory_pressure_holds_impl compares against, and a nested start is excluded on both sides. The per-lane maps are dropped together with each lane's last held row, so they cannot grow without bound. The Cross-Platform Portability run on 67c7d8e was cancelled during actions/checkout on attempt 2. That is an infra cancel, PR Readiness passed, and the diff has no platform-specific calls, so a rerun before merge is enough. The remaining Design watch (no real macOS run yet) needs a maintainer to apply ci:macos, and the author has already said so. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A held subagent wave that keeps starting members no longer ends its tail as "never started", and members that do end that way are named as not run in the wave digest the parent reads.
On macOS under memory pressure WARN, a
spawn_runwave of 84 short tasks drained at a held cap of 4. A new task started every 30 to 90 s the whole time, yet 28 tasks ended "never started" after 1844 to 2025 s, which isagent.subagent_queue_max_wait_secs(1800 s) counted from submission. The wave-close digest then read as a finished wave, and the parent only found the gap when output files were missing.Why it matters
Any held wave that takes longer than the bound to drain silently loses its tail: work is dropped while the queue is healthy, and the parent believes it ran.
Not a goal
batch_finished/ wave meta tallies.What changed (motivation → approach → change)
Root cause:
_memory_pressure_holds_impl(subagent_manager/admission/gate.py) stamps a row's clock on its first hold and expires it once the bound has passed since that stamp. The pump's pick classifies every root row in the window, so all rows are stamped at about submission.Change:
taskq.lanes.lane_key_for), the lane the pump's weighted pick serves FIFO. A root start in that lane is a row ahead of this one leaving the queue, whicheverspawn_runwave it came from, so a second wave or an unbatched spawn behind a draining wave waits its turn too. Another lane's starts never extend a row. Automation keys (cron, hook, heartbeat, background) all map toSYSTEM_LANE, one FIFO queue, so they share one progress lane; accepted and documented. A missing stamp is its own case (no 0.0 sentinel; a young runner's monotonic clock is below the bound).SubagentManager:_pressure_hold_lanes(row → lane),_pressure_lane_held(lane → held rows) and_pressure_lane_started_at(lane → last root start)._pressure_lane_startedstamps only a lane that holds a row;_forget_pressure_clock(every exit via_forget_pending_start, a floor wait taking over, the prune) drops the row, and the lane's stamp with its last row. Stamped where a root registers in the gate and where the pump releases an approved root (nested starts skip the hold, so are not progress).waited_secs, the WARNING) keeps its whole wait.slack/gateway.py): a failed member whose error opens withQUEUED_WAIT_EXPIRED_TEXTis counted. After the tally sentence (the card and notice regexes anchor on... of N agents.) the line adds "K of the ❌ never started, so their work was not done: resubmit them once memory eases if it is still needed." Rows the spent episode ended at once are counted too, hence "once memory eases": a resubmit during that episode is ended at once. A refusal is not counted.Why per lane, not per wave: the previous head keyed on
batch_id, so a row behind a draining wave in its own FIFO lane got no credit and expired while the queue moved (the issue, one wave over). Keying on the lane also drops the reliance on a caller-supplied id. Why not one manager-wide stamp: a row whose chat keeps losing its lane share would wait with no limit while other chats start. Why not clock from the head of the queue: the window is a weighted round-robin, so there is no single head.Deliberate trade-off: a row has no absolute ceiling while its lane keeps progressing (worst case about the rows ahead of it times the bound). What the bound protects against is a lane that has stopped; that still ends what it holds after one bound.
Design note (R3, R5.1): the maps are written only on the event loop by the hold check and the two start sites; no await is added. They hold at most one entry per held row, keyed by the parent key that row already carries, and a lane entry leaves with its last row.
Backwards compatibility
Compatible: no key, field, or event changes; the digest gains one sentence after the existing tally sentence, and every parser anchors on the unchanged prefix (
website/src/pages/chat/subagentCompletion.ts,src/kiro_crew/dashboard/system_notices.py). Removes nothing: the expiry still ends a stalled lane and a spent episode.Tests
test_a_root_start_in_its_lane_restarts_a_held_rows_bound;test_a_row_behind_a_draining_wave_in_its_own_lane_is_not_expired[second-wave|unbatched](the independent review's repro, end to end).test_a_held_wave_draining_slower_than_the_bound_never_expires: also starts with an episode older than the bound; the lane stamp leaves with its last held row.test_a_lane_keeps_no_stamp_once_it_holds_no_row,test_the_stale_prune_drops_only_rows_that_left,test_a_live_held_rows_clock_survives_a_pause_in_a_progressing_lane(review repro).test_automation_keys_share_one_progress_lane,test_a_young_monotonic_clock_still_expires_a_row_with_no_lane_stamp,test_the_gate_judges_a_held_start_by_its_own_lane,test_a_nested_start_is_not_progress_of_a_held_lane,test_releasing_an_approved_root_member_is_its_lanes_progress[root|child].test_wave_close_digest_names_members_that_never_started[False|True]: one ran, one failed after running, one never started, one refused; "1 of the ❌ never started" and "resubmit them once memory eases"; the card regex still parses the tally.Mutation checks, each red: key the stamp per wave (9 tests and both repro cases); prune by age alone (2 tests and the repro); plus the earlier ones (drop the
max(...), 0.0 sentinel, stamp nested starts).Run locally on the final commit, rebased on main cbf2b36 (CI on main green there; clears the stale-base ui-index, GuideLayer and Windows spawn-publication reds):
test_admission_gate.py,test_subagent_scale.pyand the admission/queue neighbours (fairness lanes, verdict census, queue max wait, memory wake, startup pressure, taskq admission, spawn gate, per-task overrides, subagent coverage): 877 passed.test_*ratchet*.py,test_agent_refactor_facade.py,test_error_code_contract.py,test_docs_*.py: 652 passed.docs_lint.py, comment-history from the merge-base, brand and feature-map: all pass.test_boot_smoke.py(15 passed).Manual verification
Not run on macOS: the hold engages only there and the author has no macOS host. Tests fake the level and clocks, with no sleeps. A maintainer run of the #18891 held wave would close this.
Related Issues
Closes #18891. Follow-up to #16480 (bounded wait).
Pattern harvest
Rule candidate: review-prompt
Pattern: a per-item timeout in a shared queue measured from enqueue, or keyed on a grouping name, rather than from the last progress of the queue the item actually waits in, so a slowly draining queue expires its tail while healthy.
Checklist
Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.