Skip to content

feat: bound how long a subagent spawn waits for memory - #16347

Merged
bolichen97 merged 2 commits into
mainfrom
feat/subagent-queue-max-wait
Oct 5, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
feat/subagent-queue-max-wait

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: a subagent spawn held back for memory stops waiting after a bounded time and its parent hears why, instead of the spawn sitting in the queue for as long as the host stays short.

When the memory floor (agent.spawn_min_memory_gb) defers a spawn, every re-check re-parks it for another admit wait. Nothing ends that cycle. On a host that stays short, the spawn never starts and never fails. Its parent sees "1 waiting" indefinitely, and a parent blocked on it is never woken.

Why it matters

Under persistent memory pressure, a fan-out can stall silently for hours. The parent model cannot tell a slow queue from one that will never move, so it can neither re-plan nor tell the user. The owner chose a finite wait (RFC overload-resilience Q10, accepted on main: default 1800 s, live-reloadable key, delivered terminal "never started: waiting for memory", queued depth back to 0).

Not a goal

  • Waking memory-queued spawns on events (child terminal, RSS settle) and per-lane memory shares: the second half of the same plan unit, landing separately.
  • A separate "still waiting" notice. None exists on main to keep. The queued card and queued_wait_text already say why a row waits.
  • Suppressing a REPLAYED expiry for a parent torn down before a restart. The teardown fence is in-process, so such a report reaches that key like any restored row does on main when it starts and completes. Accepted.
  • Owed-report replay for the other queued-stop writers. Each has an actor who already holds the outcome (the model's stop call returned it, or the operator pressed Stop). The expiry is an unattended timer, so its injected terminal is the only way the parent learns the spawn will never run.

What changed (motivation → approach → change)

Mainline-plan unit U5(a) (owner decision A1, RFC overload-resilience Q10, accepted on main). Detail: subagent.md ("A memory deferral has a max wait") and taskq.md.

  • Key. agent.subagent_queue_max_wait_secs (default 1800 s, 0 = no bound, 0..86400). It has no restart mark: apply_limits adopts a change live.
  • Measure. TaskStore.deferred_longer_than counts parked time in the current wait (each deferred event from ts to until since the last claim). A re-check does not restart the clock, and time spent eligible for a slot is not counted. Time since the first deferral would count slot wait.
  • Sweep and report. At the end of each pump pass, one writer-thread call fails each row past the bound under its generation. Each row gets the queued-stop report with error="never started: waiting for memory", so batch accounting, the digest and the parent wake are the stop's. The depth goes back to 0.
  • The report outlives its process. The failure commits first, with finish(report_owed=True). The mark is cleared only once the report has reached the parent. A wave member held for its digest, or an announce queued behind a busy slot, carries the debt as SubagentDelivery(report_owed=True). The settle that delivers the digest or the turn then clears it. At the next store attach, taskq_replay_owed_expiries_async reports whatever is still owed: it pages by (updated_at, id) and waits for the memory barrier. Delivery is at least once.
  • Ended parent. A parent-end teardown's store sweep cancels the parent's waiting rows. An expiry in the window before that sweep lands ends its card without injecting. A store read the sweep is refused now keeps the fence open, and each reaper sweep retries it (retry_owed_teardown_sweeps). A successor's rows are spared and delivered.
  • macOS pressure hold reads the live key for both its per-start bound and its episode bound, replacing _PRESSURE_HOLD_MAX_WAIT_SECS. The chip copy no longer names "30 minutes" in any of the 13 catalogs: nothing ever shows the figure, so a non-default bound would have been described wrong.
  • Settle. The expiry report passes row_settled=True: the sweep's failed is the row's terminal write, so the report's settle writes no second finish. The refused-settle log rule is main's, unchanged.

Backwards compatibility

Compatible. The key is new and finite by default, as the owner directed. Setting it to 0 restores today's unbounded wait for memory deferrals. It also lifts the macOS pressure hold's bound, which main always fixed at 1800 s: one key with one meaning for both waits, documented in configuration.md and subagent.md and pinned by test_the_hold_is_bounded_by_the_live_queue_max_wait_key. Under the default, the macOS hold keeps its 1800 s bound. No existing key, field or API changes shape. SubagentDelivery gains report_owed, which defaults to False.

Removes nothing a caller reads: two private constants (_PRESSURE_HOLD_MAX_WAIT_SECS, _PRESSURE_HOLD_PRUNE_SECS) are replaced by the key and a prune factor. Their only readers are gate.py and test_admission_gate.py, both updated.

Tests

test/test_subagent_queue_max_wait.py drives the real manager pump on a real store with a virtual clock and the production 30 s admit wait.

  • The bound: one terminal and depth 0, an in-memory (temporary) floor wait and 0 for it, slot wait not counted, the inline pump, live reload, 0, restored rows.
  • TestADigestHeldExpiryStaysOwed runs through the real gateway _subagent_done:
    • A two-member wave with one live sibling: the expiry is held, still owed, and replayed by a restarted manager.
    • The digest that carries it clears the mark and writes no tombstone.
    • A queued announce carries the debt.
    • A held expiry of an ended parent is cleared.
  • TestARefusedTeardownSweepIsRetried: the reaper retries a refused sweep and spares a successor's rows, and an expiry before the retry injects nothing.
  • TestAnEndedParent, TestARefusedWriteKeepsWhatLanded, TestAnExpiryOutlivesItsProcess (replay, barrier, refused read, 257 rows), plus test_taskq_store.py, test_subagent_dependency_mark.py, test_admission_gate.py.

Mutation-checked: removing any one of the new guards fails at least one test.

Manual verification

The 30-minute wait itself is covered by the unit tests on a virtual clock. The UI states were rendered from the real components (below).

Screenshots / video

The real ActivityViewer on the real chat reducer, fed the gateway's frame (scratch capture entry, not committed).

Queued banner, subagent_queued with reason memory_pressure (the changed chip copy):

Queued banner, memory pressure, dark

The card a max-wait expiry ends in (subagent_done, outcome failed, error never started: waiting for memory):

Never-started card, dark

The key has no Settings row: no agent-queue or memory key is in the settings registry (admit_wait_secs and spawn_min_memory_gb are file and CLI only too). Its dashboard surface is the SettingRef chip:

SettingRef chip for the key

Light theme

Queued banner, light
Never-started card, light
SettingRef chip, light

Related Issues

Part of #16480 (no closing keyword: #16480 tracks more than this PR). Related: #15876 (merged; its hold bound now reads this key).

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Rebased onto #16355

#16355 drops the posture gate for spawns and queues a non-durable start that misses the floor in the in-memory window instead of refusing it. So this PR no longer bounds posture deferrals (gone), and it now bounds that in-memory floor wait too: the gate keeps its parked time in _floor_waits and ends it with the same never started: waiting for memory at the live bound. Spec, user docs and the verdict census are updated to match.

@bolichen97
bolichen97 requested a review from a team as a code owner October 3, 2026 01:09
@bolichen97
bolichen97 requested a review from buluoray October 3, 2026 01:09
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 3, 2026
@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 9a4d7f184e035433ceae52d6a24f8943b00d1e12 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the parent handles a duplicate "never started: waiting for memory" terminal: replay after a restart can deliver it twice.

[DESIGN-REVIEWED] 9a4d7f1

Description read

[DESCRIPTION-READ] f797e435a39a9e9e06d9bfe64c2f4eeb3581801811db961d0c19a1f2506dbb03

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 9a4d7f184e035433ceae52d6a24f8943b00d1e12 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Verify that the at-least-once replay of an owed expiry report is acceptable to parents: after a crash a parent can get "never started: waiting for memory" twice.

What this change ships

Inventory (8 items) — 8 justified

Intent: a parent whose subagent spawn waits for memory eventually learns that the spawn will never start, instead of waiting forever. This is an ADDITION; base RFC overload-resilience (partial) Q10 records the shape: default 1800 s, a live key, a delivered terminal.

  1. A spawn held for memory now fails after 1800 s of parked time, and its parent gets "never started: waiting for memory" — justified
  2. New live-reloadable key agent.subagent_queue_max_wait_secs; 0 restores the unbounded wait — justified
  3. The clock counts only time parked by deferrals; a re-check doesn't restart it and slot wait doesn't count (deferred_longer_than) — justified
  4. An undelivered expiry report survives a restart and is replayed at the next store attach (report_owed, owed_reports, mark_reported) — justified
  5. An expiry for a parent that has already ended no longer injects into the retired conversation — justified
  6. A teardown store sweep whose read the store refused is retried on each reaper pass (retry_owed_teardown_sweeps) — justified
  7. The macOS pressure hold's bound now follows the key instead of the fixed _PRESSURE_HOLD_MAX_WAIT_SECS — justified
  8. The queued-chip copy in all 13 catalogs drops "30 minutes" — justified

[FIRST-PRINCIPLES-REVIEWED] 9a4d7f1

Description read

[DESCRIPTION-READ] 30c381ebfca14b77191707784ddd3150f04cb8354f5363effef9ee36892d43b2

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 9a4d7f184e035433ceae52d6a24f8943b00d1e12 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 9a4d7f1

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 9a4d7f184e035433ceae52d6a24f8943b00d1e12: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 9a4d7f184e035433ceae52d6a24f8943b00d1e12 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 9a4d7f1

Verdict parsed from the review's SHA-scoped output markers for commit 9a4d7f184e035433ceae52d6a24f8943b00d1e12.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 9a4d7f184e035433ceae52d6a24f8943b00d1e12: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 3, 2026
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT 5.6 finding F1 on d1b227b: fixed in ebc13ca

span=c65eacf3d1b8

  • F1 taskq_bridge.py:716: committed expiry is not replayed after a crash. Fixed.

The expiry is now written with finish(..., report_owed=True). In the same transaction, the terminal transition event records which incarnation still owes the report. When the report task completes without being cancelled, its done-callback posts TaskStore.mark_reported, which appends a reported event. On the next start, taskq_replay_owed_expiries_async runs once per manager. It is scheduled from taskq_boot_dispatch, or from _initialize_taskq when the store attaches after the reaper has started. It reports every failed row that TaskStore.owed_reports returns, through the same path a live expiry uses. Two kinds of row are covered: rows a lost process owed, and rows whose report the shutdown drain cancelled. Rows the current incarnation owes are never returned, because their reports are already in flight. Delivery is at least once: if the clear itself is lost, the next start reports the expiry again; it is never skipped. Tests: TestAnExpiryOutlivesItsProcess (3 tests) and test_an_owed_report_outlives_the_process_that_owed_it. All four fail on d1b227b.

Specs updated in the same commit: subagent.md § Durable task queue and taskq.md.

@github-actions

github-actions Bot commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — 🟡 CONCERNS

UX-level review of 9a4d7f184e035433ceae52d6a24f8943b00d1e12 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: CONCERNS

The banner now says starts give up after "the wait limit", a limit the user can't see or find; the blind reader had no idea how long it lasts.

Watch

  • "the wait limit" is undefined. The diff replaces "has lasted 30 minutes" with "outlasts the wait limit" in all 13 catalogs. The banner has no number, no link and no setting name, so it now warns that starts will give up without saying when (shot-01/04). The blind reader said: "I don't know what 'the wait limit' is or how long it lasts… Nothing tells me how long until they 'give up'." Severity is moderate: this shows on every macOS memory-pressure wait, every time it happens, and leaves the user unable to tell whether to wait or act. Smallest fix: have subagent_queued carry the live bound and fill it into the string ("…give up after {{duration}}"). Dropping the clause is also a fix, since the expiry card already explains the outcome.
  • The banner is still one long sentence joined by semicolons. The reader called it "long and joined with semicolons, so it's hard to read". This PR only edits the last clause, but that clause is the one carrying the deadline. Reordering it so the deadline sits next to "waiting to start" would put the important part where people read first.

Suggestions

  • memory_pressure (en.json): "waiting to start — macOS is low on memory; starts resume once it eases and give up after {{duration}}". This removes "outlasts", which the reader said was a hard word.

[UX-REVIEWED] 9a4d7f1

Description read

[DESCRIPTION-READ] 5b60f73194bd8e4f6b2d80da9c816441fbb2ebf2d7cf2709102eed0c2cf6bb9d

That sha256 names everything this verdict read: the PR description, plus the 9 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97
bolichen97 force-pushed the feat/subagent-queue-max-wait branch from ebc13ca to c6a13db Compare October 3, 2026 04:07
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT 5.6 finding on ebc13ca: fixed in c6a13db

span=c65eacf3d1b8

  • F1 taskq_bridge.py:887: the one-shot replay can strand owed expiry reports. Fixed.

taskq_replay_owed_expiries_async now sets _taskq_owed_replayed only after it has read every owed row. It reads TaskStore.owed_reports a page at a time (_OWED_REPLAY_PAGE = 256). owed_reports takes a new after=(updated_at, id) keyset cursor, and its order is now updated_at, id. The replay is marked done only when a page comes back short.

  • If the store refuses a read with TaskStoreUnavailable, the replay is left unfinished. The cursor stays on the manager, so a retry resumes after the last page that was already reported, and this process never reports a row twice.
  • Each reaper sweep now calls taskq_schedule_owed_replay, so the replay is retried. After the replay finishes, the call does nothing.
  • A replay already in flight is never started a second time (_taskq_owed_replaying).
  • The first refusal logs at WARNING. Later retries log at DEBUG.

Tests. These two fail on ebc13ca:

  • test_a_refused_read_leaves_the_replay_to_the_next_sweep: the first read raises, and the next sweep delivers the report exactly once.
  • test_every_owed_report_is_replayed_past_one_page: 257 owed rows are each delivered once, and none is left owed.

test_owed_reports_pages_after_a_cursor pins the cursor, including a tie on updated_at.

Post-push review (should_fix): every path that needs the replay reaches it. The replay must run on every store attach, and it does:

  • Sync open before start_reaper. taskq_boot_dispatch schedules it.
  • Off-loop open finishing after start_reaper. The gateway constructs SubagentManager and then calls start_reaper() on the same tick, so the store is attached only after that. _initialize_taskq schedules the replay because the reaper is already running.
  • Re-open after a refused open. taskq_reopen_if_due goes through _initialize_taskq while the reaper is running.
  • A replay that failed. It is retried by the reaper sweep.

test_the_gateway_boot_order_replays_when_the_store_attaches builds the manager the way the gateway does: it pre-seeds an owed row, then calls start_reaper() while the open is still in flight and awaits wait_taskq_ready(). The test fails if the schedule call is removed from _initialize_taskq. I checked this by mutation.

Nit: the latch was set before the read. Fixed by the change above.

Comment History Gate (taskq_bridge.py:714, "no longer"): reworded to present tense.

Specs updated in the same commit: subagent.md (Durable task queue) and taskq.md (owed_reports(kind, *, limit, after)).

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

design review concern on ebc13ca: fixed in the description

The concern was that the description had drifted from the diff. The PR description now covers the shipped recovery machinery.

  • The "What changed" section now has a bullet, "A lost report is replayed". It covers finish(report_owed=True), TaskStore.owed_reports and mark_reported, taskq_replay_owed_expiries_async and the paths that reach it, the paged cursor, the retry on each sweep, and at-least-once delivery.
  • The "kept 86400 s" text is replaced with the stamp lifetime as implemented: no age limit, kept while an unstarted row it gates still waits, and dropped by the sweep once it gates nothing.
  • The "Tests" section now names TestAnExpiryOutlivesItsProcess (6 tests), the two stamp-lifetime tests and the store's owed-report tests.

The replay serves the owner's A1 and Q10 direction that an expired spawn's parent is told why it ended. Without the replay, a process lost between the FAILED commit and the delivery would leave the parent silent, which is the outcome the Goal removes.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

first-principles review concern on ebc13ca: fixed in the description

The concern was that the description had drifted from the diff. The PR description now covers the shipped recovery machinery.

  • The "What changed" section now has a bullet, "A lost report is replayed". It covers finish(report_owed=True), TaskStore.owed_reports and mark_reported, taskq_replay_owed_expiries_async and the paths that reach it, the paged cursor, the retry on each sweep, and at-least-once delivery.
  • The "kept 86400 s" text is replaced with the stamp lifetime as implemented: no age limit, kept while an unstarted row it gates still waits, and dropped by the sweep once it gates nothing.
  • The "Tests" section now names TestAnExpiryOutlivesItsProcess (6 tests), the two stamp-lifetime tests and the store's owed-report tests.

The replay serves the owner's A1 and Q10 direction that an expired spawn's parent is told why it ended. Without the replay, a process lost between the FAILED commit and the delivery would leave the parent silent, which is the outcome the Goal removes.

@bolichen97
bolichen97 force-pushed the feat/subagent-queue-max-wait branch from c6a13db to 65abdf5 Compare October 3, 2026 05:30
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

GPT 6.1 finding on 65abdf5: not changed, rebutted

span=c65eacf3d1b8

  • F1 taskq_bridge.py:877: a report that returns False still clears the owed debt. Kept as is, deliberately.

The owed-report debt (finish(..., report_owed=True) / mark_reported) covers one gap: the process is lost between committing the expiry and running its report. It is not a delivery outbox. Once the report has run, delivery failure is handled by the shared terminal-report path, the same way it is for every other subagent terminal:

  • _report_terminal_impl returns False only after _ON_DONE_TIMEOUT (1200 s, subagent.py:1183) or an exception (terminal.py:436-458). On the timeout it resets the parent session and calls notify_injection_failed. That call (terminal.py:1337) appends the failure to the dashboard slot and queues it into slot._pending_subagent_failures, so the LLM still gets the notice on the parent's next turn. So the expiry is not silently lost. It is delivered by the fallback channel.
  • Keeping the debt on False would deliver the notice twice: once by that fallback, and again on the next restart into the same parent, which might still be busy. A replay of an expiry that already reached the parent's failure queue is the double report this mechanism exists to prevent.
  • No other terminal report in the codebase retries on False. Making only the memory-wait expiry stricter than a normal completion would be a new delivery contract, and it belongs to the terminal-report path, not this PR.
  • The row stays FAILED in the store either way, so state is not lost or corrupted.

The pinned contract is what the docstring says: a report that ran (not cancelled) clears the debt, and a report cancelled with its process stays owed. test_a_report_cancelled_with_its_process_stays_owed and test_a_reported_expiry_owes_nothing cover it.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor Author

first-principles concern on 65abdf5: replay kept, rebutted

span=604746f0b1a8

  • Item 6, "the lost-report replay is oversized; keep the expiry at-most-once like the queued stop." Kept, deliberately.
  1. A blocking lane asked for it. GPT 5.6 blocked d1b227b on this exact window: "committed expiry is not replayed after a crash". The disposition is in this thread (issuecomment-5964708971). Removing the replay re-opens a finding a blocking lane already raised. The paging that looks like over-generalisation came from the next blocking GPT round on ebc13ca ("the one-shot replay can strand owed expiry reports", issuecomment-5965377667). An unbounded single read was the defect there.
  2. The queued stop is not the same loss. A queued stop always has an actor who already knows the outcome. Either the LLM's own stop call returned the result to it, or the operator pressed stop in the dashboard. The expiry has no actor. It is an unattended timer, and the injected terminal is the ONLY way the parent learns its spawn will never run. A parent that lost it keeps its plan built on a subagent that will never report. The asymmetry is the point.
  3. The window is not just a crash. The shutdown drain cancels report tasks that have not finished, and that happens on every gateway restart and update, not only on a crash. test_a_report_cancelled_with_its_process_stays_owed pins that case.
  4. owed_reports(kind, ...) takes a kind because every TaskStore read does (count_pending, fetch_dispatchable, deferred_longer_than, oldest_unstarted_by_session). It does not widen the surface. It still has one consumer.

Clears when: the owner decides the expiry should be at most once. Until then, the replay is the fix for a blocking finding, not speculative surface.

@bolichen97

Copy link
Copy Markdown
Contributor Author

GPT 5.6 finding F2 on d1b227b: fixed in ebc13ca

span=c65eacf3d1b8

  • F2 taskq_bridge.py:814: a pruned retirement stamp can route an expiry into a successor. Fixed.

_RETIRED_PARENT_TTL_SECS and the age-based prune are removed. A stamp is now kept for as long as the store holds an unstarted row of that parent that was accepted at or before the stamp. Each sweep runs its reports first, then drops the stamps that gate nothing, using TaskStore.oldest_unstarted_by_session on the writer thread. A key that was re-stamped in the meantime is kept. Without a store, nothing is stamped, so the dict cannot grow without bound. Tests: test_the_stamp_is_kept_for_as_long_as_its_rows_wait fails on d1b227b with the exact injection the finding describes. test_a_stamp_that_gates_nothing_is_dropped covers the cleanup.

Specs updated in the same commit: subagent.md § Durable task queue and taskq.md.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 3, 2026
@bolichen97
bolichen97 force-pushed the feat/subagent-queue-max-wait branch from 65abdf5 to 2ad6f67 Compare October 3, 2026 07:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased onto origin/main (ecb30c2) to clear the conflict with #16357 in src/kiro_crew/subagent.py (a constants-import union: DEFAULT_SUBAGENT_QUEUE_MAX_WAIT_SECS alongside DENY_CAUSE_POLICY / DENY_CAUSE_SURFACE_POLICY). Folded two doc nits into the first commit: the subagent.md Durable task queue text now says the queued depth is re-published by _report_queued_stop itself (once per row, before the terminal record; the sweep emits none of its own), and the docstring of test_the_expiry_publishes_the_depth_it_changed no longer claims a sweep-owned emit. No behaviour change. Static gates, mypy and 3151 subagent/taskq/admission/config tests pass locally. New head 2ad6f67.

@github-actions github-actions Bot removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 3, 2026
A spawn the memory floor or the posture gate defers is re-parked every
admit wait for as long as the host stays short, so it could wait
forever. The owner chose a finite bound:
agent.subagent_queue_max_wait_secs, default 1800 s, 0 for no bound,
read live with no restart mark.

The pump now ends such a row at the end of each pass.
TaskStore.deferred_longer_than finds queued rows still parked that
have spent the bound parked in their current wait (each deferral since
the last claim or re-queue counted from its time to its until), and
each is failed under the generation it was read at. Time a row spends
eligible, queued for a slot, is not counted, so a row held back once
and then queued behind a full cap is not ended by its next memory
re-check. The parent receives the queued-stop report with the error
"never started: waiting for memory", and its queued depth is
re-published, reaching 0. A parent-end teardown stamps the parent key
in this process, so an expired row of a conversation that ended is
not injected into it (the injector would recreate it).

The macOS kernel memory-pressure hold (#15876) bounded itself with a
constant carrying this key's planned default. It now reads the key at
each check, for a held start's own wait and for the episode, so a
reload moves both and 0 lifts both; a bound raised past the episode
un-spends it. The hold's chip no longer names 30 minutes: the gateway
sends it no figure, so it now says the starts give up once the pressure
outlasts the wait limit, in every catalog.

The settle that follows the pre-written terminal is logged at DEBUG
when the reporting record never held a claim and the row already holds
that state; a claimed run in that position still warns. Specs, the RFC
rows that called the wait unbounded, the packaged docs and
config-baseline.json are updated with it.
@bolichen97
bolichen97 force-pushed the feat/subagent-queue-max-wait branch from ff339bc to bd16386 Compare October 5, 2026 03:58
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 5, 2026
The expiry's failure commits on the writer thread before its report
runs on the loop, so a process lost in between (or a report the
shutdown drain cancelled) left the parent never told. The failure is
now written with finish(report_owed=True), which records the owing
incarnation on the terminal transition in the same transaction. The
mark is cleared once the report has reached the parent; a report that
returned False (the injection timed out or failed) stays owed. So does
one the gateway gave up on: a channel or cron parent's failed attempts
are swallowed inside _on_done, so notify_injection_failed (and the cron
arm that swallows a failure with no notice) marks the record
_report_undelivered, and the clear skips it. A stage-boundary retry
that later delivers a latched expiry clears the mark as the first
report would have, so the next start does not report it twice. The
next start reports every row owed_reports names, once per manager,
exactly as a live expiry is reported.

A report the gateway parks has not reached the parent when its task
returns: a wave member held for its digest, an announce queued behind
a busy slot, or one handed to an idle slot's turn lives only in this
process's memory until that digest or that turn is delivered. The
synthetic record carries _report_owed, the gateway hands the debt on
with the digest or the announce as SubagentDelivery(report_owed=True),
and the settle that delivers it clears the store mark instead of
writing a delivered tombstone (the row has no run folder). That settle
runs whenever _on_done returns, so when the gateway gave up on a cron
or channel wave's digest (the last member's or the reaper's forced
flush) the flusher's _report_undelivered keeps the held expiries owed
instead of clearing them. A digest whose flusher reports to a parent a
teardown retired clears it too.

The replay reads owed_reports a page at a time behind an (updated_at,
id) cursor and is marked done only once a page comes back short, so a
backlog past one page is reported in full. A read the store refuses
leaves it unfinished; each reaper sweep retries it from the cursor, and
one in flight is never started twice. No replay runs while the gateway
holds dispatch for its memory barrier: a report injected then fails on
MemoryStartupUnavailable, which the injector swallows, so the report
would look run and its owed mark would be cleared undelivered.
release_queue_dispatch schedules the replay instead.

A sweep's finish calls commit one by one, so a store refusal on a later
row no longer discards the rows already failed: they are reported, and
the refused row and the rest stay queued for the next sweep.

The in-process retired-parent stamp is removed. Since #16395 a
parent-end teardown's store sweep cancels the parent's waiting rows,
so they no longer expire. Between the snapshot and that sweep, an
expired row is gated by the teardown fence #16395 opened: a row whose
parent has a teardown open and that the fence does not record ends
its card without injecting. A sweep whose store read is refused (the
teardown's reading lets TaskStoreUnavailable propagate) keeps that
fence open and is retried by each reaper sweep until the
read lands, so the retired parent's rows are still stopped rather than
left queued to start or expire into the key's next conversation.
TaskStore.oldest_unstarted_by_session, which only the stamp read, goes
with it.

Bound a floor wait with no durable row too. Main now queues an incognito,
temporary or no-store start that does not fit the floor in the in-memory
window instead of refusing it, and drops the posture gate for spawns. That
wait has no store row for the sweep, so the gate keeps its parked time in
_floor_waits (each park cut at its planned end, so a slot wait between parks
is not counted) and ends it with the same "never started: waiting for
memory" once the live agent.subagent_queue_max_wait_secs is reached.
@bolichen97
bolichen97 force-pushed the feat/subagent-queue-max-wait branch from bd16386 to 9a4d7f1 Compare October 5, 2026 04:35
@bolichen97

Copy link
Copy Markdown
Contributor Author

span=8262655f0adc

First Principles item 7 on bd16386: the never_claimed narrowing — fixed in 9a4d7f1

Outcome: fixed in 9a4d7f1, by the lane's own subtraction.
The never_claimed parameter, its two call-site arguments, the rewritten comment and test_a_same_state_refusal_is_quiet_only_for_a_record_that_never_claimed are gone; taskq_report_refused_settle is main's rule again. The expiry report now passes row_settled=True (the sweep's failed is the row's terminal write), so its settle writes no second finish. The description and the subagent spec say so.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=bd4b0fed318d

Opus 5.5 advisory finding on bd16386: the help text overstated the slot exclusion — fixed in 9a4d7f1

Outcome: fixed in 9a4d7f1 (the finding's second option).
The macOS hold's per-start clock (_pressure_holds.setdefault) is main's; this PR only swaps its constant for the live bound. Resetting it on release would change main's hold, which is out of this PR's goal. The help text in sections.py and config-baseline.json now says: "Waiting for a free slot does not count, except on macOS: a start held for memory pressure is timed from its first hold."

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=61ce9aaf8582

UX Watch on bd16386: "never started: waiting for memory" reads as live — accepted-and-deferred

Outcome: accepted-and-deferred to #16845 (the copy task already there).
QUEUED_WAIT_EXPIRED_TEXT is also the terminal the parent agent reads and the owed-report replay re-sends, so it changes in the same copy pass as the memory_pressure chip, translated once across the 13 catalogs.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=bd2855dc9cae

UX Watch on bd16386: "the wait limit" states no figure — accepted-and-deferred

Outcome: accepted-and-deferred to #16845.
Showing the live figure needs the bound on the subagent_queued frame and a plural-aware {{limit}} in all 13 catalogs; that is the first task in #16845. "30 minutes" was dropped because it is wrong for any non-default agent.subagent_queue_max_wait_secs.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=94b3fa474451

UX suggestion on bd16386: the SettingRef sentence has no value — rebutted

Outcome: rebutted.
The Settings row this PR adds (AgentConfig.subagent_queue_max_wait_secs) is labelled "Subagent Queue Max Wait (seconds)", shows its value (default 1800) in the field, and its help ends "0 waits forever; the most is 86400 (one day)". The unit and the number are on screen beside the text, and configuration.md names 1800 (30 min).

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=fc608c89d64d

UX Watch on 9a4d7f1: "the wait limit" is undefined — accepted-and-deferred

Outcome: accepted-and-deferred to #16845.
Filling the live figure needs the bound on the subagent_queued frame and a plural-aware {{duration}} in all 13 catalogs; that is the first task in #16845. "30 minutes" was dropped because it is wrong for any non-default agent.subagent_queue_max_wait_secs, and dropping the clause would hide that starts end at all.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=c1a9034bbb00

UX Watch on 9a4d7f1: the banner is one long semicolon sentence — accepted-and-deferred

Outcome: accepted-and-deferred to #16845 (the copy task already there).
The sentence shape predates this PR, which edits only its last clause. Reordering it so the deadline sits next to "waiting to start" lands in the same copy pass as the {{duration}} rewording, so the 13 catalogs are translated once.

@bolichen97

Copy link
Copy Markdown
Contributor Author

span=e8a83767a05e

UX suggestion on 9a4d7f1: "…give up after {{duration}}" — accepted-and-deferred

Outcome: accepted-and-deferred to #16845.
The suggested wording needs {{duration}}, which needs the bound on the subagent_queued frame. It lands with that protocol change, together with the reorder above.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants