Repository navigation
feat: bound how long a subagent spawn waits for memory - #16347
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the parent handles a duplicate "never started: waiting for memory" terminal: replay after a restart can deliver it twice. [DESIGN-REVIEWED] 9a4d7f1 Description read[DESCRIPTION-READ] f797e435a39a9e9e06d9bfe64c2f4eeb3581801811db961d0c19a1f2506dbb03 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Verify that the at-least-once replay of an owed expiry report is acceptable to parents: after a crash a parent can get "never started: waiting for memory" twice. What this change shipsInventory (8 items) — 8 justifiedIntent: a parent whose subagent spawn waits for memory eventually learns that the spawn will never start, instead of waiting forever. This is an ADDITION; base RFC overload-resilience (
[FIRST-PRINCIPLES-REVIEWED] 9a4d7f1 Description read[DESCRIPTION-READ] 30c381ebfca14b77191707784ddd3150f04cb8354f5363effef9ee36892d43b2 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 5.6 finding F1 on d1b227b: fixed in ebc13caspan=c65eacf3d1b8
Specs updated in the same commit: |
UX Review (Opus 5.5) — 🟡 CONCERNSUX-level review of UX-Verdict: CONCERNS The banner now says starts give up after "the wait limit", a limit the user can't see or find; the blind reader had no idea how long it lasts. Watch
Suggestions
[UX-REVIEWED] 9a4d7f1 Description read[DESCRIPTION-READ] 5b60f73194bd8e4f6b2d80da9c816441fbb2ebf2d7cf2709102eed0c2cf6bb9d That sha256 names everything this verdict read: the PR description, plus the 9 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
ebc13ca to
c6a13db
Compare
GPT 5.6 finding on ebc13ca: fixed in c6a13dbspan=c65eacf3d1b8
Post-push review (should_fix): every path that needs the replay reaches it. The replay must run on every store attach, and it does:
Nit: the latch was set before the read. Fixed by the change above. Comment History Gate ( Specs updated in the same commit: |
design review concern on ebc13ca: fixed in the descriptionThe concern was that the description had drifted from the diff. The PR description now covers the shipped recovery machinery.
The replay serves the owner's A1 and Q10 direction that an expired spawn's parent is told why it ended. Without the replay, a process lost between the FAILED commit and the delivery would leave the parent silent, which is the outcome the Goal removes. |
first-principles review concern on ebc13ca: fixed in the descriptionThe concern was that the description had drifted from the diff. The PR description now covers the shipped recovery machinery.
The replay serves the owner's A1 and Q10 direction that an expired spawn's parent is told why it ended. Without the replay, a process lost between the FAILED commit and the delivery would leave the parent silent, which is the outcome the Goal removes. |
c6a13db to
65abdf5
Compare
GPT 6.1 finding on 65abdf5: not changed, rebuttedspan=c65eacf3d1b8
The owed-report debt (
The pinned contract is what the docstring says: a report that ran (not cancelled) clears the debt, and a report cancelled with its process stays owed. |
first-principles concern on 65abdf5: replay kept, rebuttedspan=604746f0b1a8
Clears when: the owner decides the expiry should be at most once. Until then, the replay is the fix for a blocking finding, not speculative surface. |
GPT 5.6 finding F2 on d1b227b: fixed in ebc13caspan=c65eacf3d1b8
Specs updated in the same commit: |
65abdf5 to
2ad6f67
Compare
|
Rebased onto origin/main (ecb30c2) to clear the conflict with #16357 in src/kiro_crew/subagent.py (a constants-import union: DEFAULT_SUBAGENT_QUEUE_MAX_WAIT_SECS alongside DENY_CAUSE_POLICY / DENY_CAUSE_SURFACE_POLICY). Folded two doc nits into the first commit: the subagent.md Durable task queue text now says the queued depth is re-published by _report_queued_stop itself (once per row, before the terminal record; the sweep emits none of its own), and the docstring of test_the_expiry_publishes_the_depth_it_changed no longer claims a sweep-owned emit. No behaviour change. Static gates, mypy and 3151 subagent/taskq/admission/config tests pass locally. New head 2ad6f67. |
A spawn the memory floor or the posture gate defers is re-parked every admit wait for as long as the host stays short, so it could wait forever. The owner chose a finite bound: agent.subagent_queue_max_wait_secs, default 1800 s, 0 for no bound, read live with no restart mark. The pump now ends such a row at the end of each pass. TaskStore.deferred_longer_than finds queued rows still parked that have spent the bound parked in their current wait (each deferral since the last claim or re-queue counted from its time to its until), and each is failed under the generation it was read at. Time a row spends eligible, queued for a slot, is not counted, so a row held back once and then queued behind a full cap is not ended by its next memory re-check. The parent receives the queued-stop report with the error "never started: waiting for memory", and its queued depth is re-published, reaching 0. A parent-end teardown stamps the parent key in this process, so an expired row of a conversation that ended is not injected into it (the injector would recreate it). The macOS kernel memory-pressure hold (#15876) bounded itself with a constant carrying this key's planned default. It now reads the key at each check, for a held start's own wait and for the episode, so a reload moves both and 0 lifts both; a bound raised past the episode un-spends it. The hold's chip no longer names 30 minutes: the gateway sends it no figure, so it now says the starts give up once the pressure outlasts the wait limit, in every catalog. The settle that follows the pre-written terminal is logged at DEBUG when the reporting record never held a claim and the row already holds that state; a claimed run in that position still warns. Specs, the RFC rows that called the wait unbounded, the packaged docs and config-baseline.json are updated with it.
ff339bc to
bd16386
Compare
The expiry's failure commits on the writer thread before its report runs on the loop, so a process lost in between (or a report the shutdown drain cancelled) left the parent never told. The failure is now written with finish(report_owed=True), which records the owing incarnation on the terminal transition in the same transaction. The mark is cleared once the report has reached the parent; a report that returned False (the injection timed out or failed) stays owed. So does one the gateway gave up on: a channel or cron parent's failed attempts are swallowed inside _on_done, so notify_injection_failed (and the cron arm that swallows a failure with no notice) marks the record _report_undelivered, and the clear skips it. A stage-boundary retry that later delivers a latched expiry clears the mark as the first report would have, so the next start does not report it twice. The next start reports every row owed_reports names, once per manager, exactly as a live expiry is reported. A report the gateway parks has not reached the parent when its task returns: a wave member held for its digest, an announce queued behind a busy slot, or one handed to an idle slot's turn lives only in this process's memory until that digest or that turn is delivered. The synthetic record carries _report_owed, the gateway hands the debt on with the digest or the announce as SubagentDelivery(report_owed=True), and the settle that delivers it clears the store mark instead of writing a delivered tombstone (the row has no run folder). That settle runs whenever _on_done returns, so when the gateway gave up on a cron or channel wave's digest (the last member's or the reaper's forced flush) the flusher's _report_undelivered keeps the held expiries owed instead of clearing them. A digest whose flusher reports to a parent a teardown retired clears it too. The replay reads owed_reports a page at a time behind an (updated_at, id) cursor and is marked done only once a page comes back short, so a backlog past one page is reported in full. A read the store refuses leaves it unfinished; each reaper sweep retries it from the cursor, and one in flight is never started twice. No replay runs while the gateway holds dispatch for its memory barrier: a report injected then fails on MemoryStartupUnavailable, which the injector swallows, so the report would look run and its owed mark would be cleared undelivered. release_queue_dispatch schedules the replay instead. A sweep's finish calls commit one by one, so a store refusal on a later row no longer discards the rows already failed: they are reported, and the refused row and the rest stay queued for the next sweep. The in-process retired-parent stamp is removed. Since #16395 a parent-end teardown's store sweep cancels the parent's waiting rows, so they no longer expire. Between the snapshot and that sweep, an expired row is gated by the teardown fence #16395 opened: a row whose parent has a teardown open and that the fence does not record ends its card without injecting. A sweep whose store read is refused (the teardown's reading lets TaskStoreUnavailable propagate) keeps that fence open and is retried by each reaper sweep until the read lands, so the retired parent's rows are still stopped rather than left queued to start or expire into the key's next conversation. TaskStore.oldest_unstarted_by_session, which only the stamp read, goes with it. Bound a floor wait with no durable row too. Main now queues an incognito, temporary or no-store start that does not fit the floor in the in-memory window instead of refusing it, and drops the posture gate for spawns. That wait has no store row for the sweep, so the gate keeps its parked time in _floor_waits (each park cut at its planned end, so a slot wait between parks is not counted) and ends it with the same "never started: waiting for memory" once the live agent.subagent_queue_max_wait_secs is reached.
bd16386 to
9a4d7f1
Compare
|
span=8262655f0adc First Principles item 7 on bd16386: the never_claimed narrowing — fixed in 9a4d7f1
|
|
span=bd4b0fed318d Opus 5.5 advisory finding on bd16386: the help text overstated the slot exclusion — fixed in 9a4d7f1
|
|
span=61ce9aaf8582 UX Watch on bd16386: "never started: waiting for memory" reads as live — accepted-and-deferred
|
|
span=bd2855dc9cae UX Watch on bd16386: "the wait limit" states no figure — accepted-and-deferred
|
|
span=94b3fa474451 UX suggestion on bd16386: the SettingRef sentence has no value — rebutted
|
|
span=fc608c89d64d UX Watch on 9a4d7f1: "the wait limit" is undefined — accepted-and-deferred
|
|
span=c1a9034bbb00 UX Watch on 9a4d7f1: the banner is one long semicolon sentence — accepted-and-deferred
|
|
span=e8a83767a05e UX suggestion on 9a4d7f1: "…give up after {{duration}}" — accepted-and-deferred
|
Problem / Motivation
Goal: a subagent spawn held back for memory stops waiting after a bounded time and its parent hears why, instead of the spawn sitting in the queue for as long as the host stays short.
When the memory floor (
agent.spawn_min_memory_gb) defers a spawn, every re-check re-parks it for another admit wait. Nothing ends that cycle. On a host that stays short, the spawn never starts and never fails. Its parent sees "1 waiting" indefinitely, and a parent blocked on it is never woken.Why it matters
Under persistent memory pressure, a fan-out can stall silently for hours. The parent model cannot tell a slow queue from one that will never move, so it can neither re-plan nor tell the user. The owner chose a finite wait (RFC overload-resilience Q10, accepted on main: default 1800 s, live-reloadable key, delivered terminal "never started: waiting for memory", queued depth back to 0).
Not a goal
queued_wait_textalready say why a row waits.What changed (motivation → approach → change)
Mainline-plan unit U5(a) (owner decision A1, RFC overload-resilience Q10, accepted on main). Detail:
subagent.md("A memory deferral has a max wait") andtaskq.md.agent.subagent_queue_max_wait_secs(default 1800 s,0= no bound, 0..86400). It has no restart mark:apply_limitsadopts a change live.TaskStore.deferred_longer_thancounts parked time in the current wait (eachdeferredevent fromtstountilsince the last claim). A re-check does not restart the clock, and time spent eligible for a slot is not counted. Time since the first deferral would count slot wait.error="never started: waiting for memory", so batch accounting, the digest and the parent wake are the stop's. The depth goes back to 0.finish(report_owed=True). The mark is cleared only once the report has reached the parent. A wave member held for its digest, or an announce queued behind a busy slot, carries the debt asSubagentDelivery(report_owed=True). The settle that delivers the digest or the turn then clears it. At the next store attach,taskq_replay_owed_expiries_asyncreports whatever is still owed: it pages by(updated_at, id)and waits for the memory barrier. Delivery is at least once.retry_owed_teardown_sweeps). A successor's rows are spared and delivered._PRESSURE_HOLD_MAX_WAIT_SECS. The chip copy no longer names "30 minutes" in any of the 13 catalogs: nothing ever shows the figure, so a non-default bound would have been described wrong.row_settled=True: the sweep'sfailedis the row's terminal write, so the report's settle writes no secondfinish. The refused-settle log rule is main's, unchanged.Backwards compatibility
Compatible. The key is new and finite by default, as the owner directed. Setting it to
0restores today's unbounded wait for memory deferrals. It also lifts the macOS pressure hold's bound, which main always fixed at 1800 s: one key with one meaning for both waits, documented inconfiguration.mdandsubagent.mdand pinned bytest_the_hold_is_bounded_by_the_live_queue_max_wait_key. Under the default, the macOS hold keeps its 1800 s bound. No existing key, field or API changes shape.SubagentDeliverygainsreport_owed, which defaults toFalse.Removes nothing a caller reads: two private constants (
_PRESSURE_HOLD_MAX_WAIT_SECS,_PRESSURE_HOLD_PRUNE_SECS) are replaced by the key and a prune factor. Their only readers aregate.pyandtest_admission_gate.py, both updated.Tests
test/test_subagent_queue_max_wait.pydrives the real manager pump on a real store with a virtual clock and the production 30 s admit wait.0for it, slot wait not counted, the inline pump, live reload,0, restored rows.TestADigestHeldExpiryStaysOwedruns through the real gateway_subagent_done:TestARefusedTeardownSweepIsRetried: the reaper retries a refused sweep and spares a successor's rows, and an expiry before the retry injects nothing.TestAnEndedParent,TestARefusedWriteKeepsWhatLanded,TestAnExpiryOutlivesItsProcess(replay, barrier, refused read, 257 rows), plustest_taskq_store.py,test_subagent_dependency_mark.py,test_admission_gate.py.Mutation-checked: removing any one of the new guards fails at least one test.
Manual verification
The 30-minute wait itself is covered by the unit tests on a virtual clock. The UI states were rendered from the real components (below).
Screenshots / video
The real
ActivityVieweron the real chat reducer, fed the gateway's frame (scratch capture entry, not committed).Queued banner,
subagent_queuedwith reasonmemory_pressure(the changed chip copy):The card a max-wait expiry ends in (
subagent_done, outcomefailed, errornever started: waiting for memory):The key has no Settings row: no agent-queue or memory key is in the settings registry (
admit_wait_secsandspawn_min_memory_gbare file and CLI only too). Its dashboard surface is theSettingRefchip:Light theme
Related Issues
Part of #16480 (no closing keyword: #16480 tracks more than this PR). Related: #15876 (merged; its hold bound now reads this key).
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Rebased onto #16355
#16355 drops the posture gate for spawns and queues a non-durable start that misses the floor in the in-memory window instead of refusing it. So this PR no longer bounds posture deferrals (gone), and it now bounds that in-memory floor wait too: the gate keeps its parked time in
_floor_waitsand ends it with the samenever started: waiting for memoryat the live bound. Spec, user docs and the verdict census are updated to match.