Repository navigation
fix(subagent): keep a queued notice recoverable until it drains - #19415
bolichen97 merged 2 commits into
Conversation
A notice for the parent agent waits in slot._pending_subagent_failures, which lives only in memory, until the parent's next turn takes it. Two paths tombstoned the run as delivered before that, so a restart in between lost the notice and the next start never announced the run: - a report the gateway gave up on (_report_undelivered) still got the run loop's delivered tombstone; - a boot orphan's notice was tombstoned delivered as soon as the slot queued it. The run loop now skips the delivered mark for a given-up report, and both notices record the tombstone they owe on the slot's delivery ledger under the queued text. The parent's turn settles the debt when it drains the text (settle_drained_subagent_notices), writing the carried record, so a cut-off or failed orphan is not relabelled completed. A restart before the drain leaves the folder to the next start, which announces the run once.
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: keep a parent agent's sub-agent notice from being lost when the gateway restarts before the parent's next turn reads it. FIX.
[FIRST-PRINCIPLES-REVIEWED] 1a3613b Description read[DESCRIPTION-READ] 4f2c961f299cc6625589367e59cad5c41e23e1ade6492c3bed6ac426fdd352c8 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that [DESIGN-REVIEWED] 1a3613b Description read[DESCRIPTION-READ] e7e70b12ea26d2a3b0850d4608ac4e4e6c0a891d5b1e4b5aece08d0ec43c0872 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
The record a queued orphan notice owes rides the slot ledger in memory until the parent's next turn drains it. The ledger's entry cap bounds how many records it holds, not how large each is, and the record copied last_tool, pid and turns from the run's state.json as they were stored. last_tool is the backend's tool title, which for a shell tool is the whole command, and tool diagnostics persist it uncut. orphan_carried_fields now builds those three fields for both orphan tombstones (the one a delivered notice writes or owes, and the one the boot scan writes): last_tool is flattened to one line, redacted over the whole value and cut to the 80 characters the notice already shows, and a pid or turns that is not the integer the run writes reads as absent. The write after a delivered notice passes cause="gateway_restart" as an explicit keyword again. test_every_tombstone_cause_has_a_terminal_state reads the causes from the keywords of those calls, and a ** spread of the record hid this one from it. The notice's delivery record carries the same cause for the drain's write.
|
Two things on the previous head, both fixed in the new commit 1a3613b (on top of 253471c; origin/main is still c4c51aa, so there is no rebase).
Tests ( |
bolichen97
left a comment
There was a problem hiding this comment.
Maintainer approval after a full diff read on head 1a3613bb86: diff matches the description, no regression to existing behaviour found, CI and readiness green.
Problem / Motivation
Goal: A sub-agent notice the parent agent has not yet taken (a report the gateway gave up on, or a restart orphan's notice) leaves its run visible to the next start, so a restart before the parent's next turn announces the run again instead of losing it.
Both notices reach the parent agent only through
slot._pending_subagent_failures, an in-memory list that the parent's next turn empties (dashboard/chat_runner.py:11395-11397on c4c51aa). The run's folder is tombstoneddeliveredbefore that, and a tombstone excludes the folder from every later reconciliation (list_orphans()), so a restart in between loses the notice and nothing announces the run.notify_injection_failedsetsinfo._report_undelivered(subagent_manager/terminal.py:1420on c4c51aa) and thesubagent_injection_failedhandler queues the failure text in that list withslot._pending_subagent_failures.append(slack/gateway.py:11127on c4c51aa). The run loop'sdeliveredgate does not read the flag (subagent_manager/terminal.py:399-402on c4c51aa), so the run still gets its tombstone. The leaked-process branch beside it does read it (subagent_manager/terminal.py:434on c4c51aa)._orphan_notifyonly queues the text and returns True (slack/gateway.py:11206-11209on c4c51aa), and_notify_orphan_implthen writesrecovery_action="delivered"(subagent_manager/monitoring.py:826on c4c51aa).#18802 made the reconciler tombstone an orphan only after its notice is recorded, and named this follow-up in its Not a goal.
Why it matters
A parent agent that is never told a sub-agent finished, or was cut off, may wait for it, spawn it again or report its result as missing. The user still sees the transcript card, so what is lost is the agent's own copy, which is the one that drives its next step.
Not a goal
_arm_queued_delivery_settlement,dashboard/chat_runner.py:6612). The drain settles when it takes the text, so a restart between the drain and the model reading it still loses the notice, as before._pending_subagent_failuresitself, or changing what the parent is told: the text, the card and the owner DM are as they were._retrigger_recoveryinslack/gateway.py, which empties the same list. Nothing on main calls it except its own done-callback (git grep -n _retrigger_recoveryfinds its definition and that call), so it is left alone.What changed (motivation → approach → change)
Root cause: the
deliveredtombstone is written when a notice is queued in memory, not when the parent takes it.Approach: the tombstone a queued notice would have written becomes a debt on the slot's delivery ledger (
note_pending_subagent_deliveryandtake_pending_subagent_deliveries), keyed on the queued text. A queued announce already uses that ledger, which holds at most 128 entries per slot; an entry evicted past that leaves its run for the next start to announce. The drain that takes the text settles the debt through the manager (settle_queued_delivery).subagent_manager/terminal.py:407: the run loop'sdeliveredgate also requiresnot info._report_undelivered, as the leaked-process branch does (:439).slack/gateway.py:11132: thesubagent_injection_failedhandler notes a completed run's debt under the queued text. A failed or stopped run was never going to get thedeliveredtombstone.SubagentDeliverygainstombstone(subagent.py:1095), the record a restart orphan's notice owes (the boot fields, outcome included), andORPHAN_NOTICE_OWED(subagent.py:1101) is whaton_orphan_notifyreturns when it queued the notice and took that record._orphan_notify(slack/gateway.py:11188) accepts the record, notes it under the queued text and returnsORPHAN_NOTICE_OWED. The record is built once in_notify_orphan_impland passed on (subagent_manager/monitoring.py:869), then shared with the write that follows an injection that took no debt (subagent_manager/monitoring.py:883, which passescause="gateway_restart"as a keyword, the formtest_every_tombstone_cause_has_a_terminal_statereads from the call)._reconcile_orphans_impl(subagent_manager/monitoring.py:545) sends no owner DM and writes no tombstone for an owed notice.orphan_carried_fields(subagent_manager/monitoring.py:91) builds the three fields that record copies from the run'sstate.json,pid,turnsandlast_tool, for the record a queued notice owes and for the boot scan's own tombstone. The slot ledger caps how many records it holds (128), not how large each one is, andlast_toolis the backend's tool title, which for a shell tool is the whole command. It is flattened to one line, redacted over the whole value and cut to the 80 characters the notice already shows (_ORPHAN_LAST_TOOL_CHARS,subagent_manager/monitoring.py:88), and apidorturnsthat is not an integer reads as absent.settle_drained_subagent_notices(dashboard/chat_runner.py:6719) is called where_run_chatempties the list (dashboard/chat_runner.py:11439).settle_queued_delivery_impl(subagent_manager/waves.py:467) writes a carried record in place of thedeliveredmark, so a cut-off or failed orphan is not relabelled completed.docs/system-specs/modules/subagent.md:3465states the rule.The one behaviour change: the run loop no longer writes the
deliveredtombstone of a report the gateway gave up on. The parent's turn writes it when it drains the notice. A restart before then, or a parent with no tab to drain it, leaves the folder to the next start, which announces the run once.Backwards compatibility
Compatible: no persisted format changes. Every tombstone written has the fields it had before, and a normal boot still announces each orphan once.
on_orphan_notifygains an optionaldeliveryargument; the gateway's callback is its only implementation insrc.Removes nothing: no route, field, file or config key is removed or renamed.
Tests
In short: on main (c4c51aa) the four test files that carry the new cases fail 16 of them and pass 565 other cases, the same sixteen in each of 3 runs; on this branch the same files pass 581, 3 of 3 runs. Mutations: 12 of 12 fail only their own rule's cases.
New cases, by rule:
deliveredtombstone, and a restart announces it againtest_a_restart_before_the_drain_announces_a_given_up_report_againtest_the_parent_turn_settles_the_notice_it_drains,test_a_drained_notice_settles_the_tombstone_and_is_not_announced_againtest_an_owed_notice_is_reported_as_owed_not_as_delivered,test_a_restart_before_the_drain_announces_a_boot_orphan_againtest_a_drained_boot_orphan_notice_keeps_its_outcome_and_is_announced_once[finished|lost]test_subagent_event_injection_failed_notes_the_debt_of_a_completed_run,test_subagent_event_injection_failed_notes_no_debt_for_an_unfinished_run[failed|stopped]_orphan_notifytakes the record under the queued texttest_notify_takes_the_notice_tombstone_under_the_queued_texttest_a_delivered_report_is_still_tombstoned_by_the_run_looplast_toolis held at 80 on the slot ledgertest_the_record_a_queued_notice_owes_is_bounded_where_it_is_builtcausegateway_restarttest_a_tombstone_the_scan_writes_carries_the_bounded_title[injected|digest]pid, a booleanturns, a listlast_toolread as absent)test_the_carried_fields_keep_only_the_types_the_run_stores[as-stored|strings-and-lists|booleans-and-numbers|absent]srcstill has a terminal task state, and the scan seesgateway_restartin the write after a delivered noticetest_every_tombstone_cause_has_a_terminal_state(test_taskq_reconcile.py, existing)test_subagent_undelivered_notice_restart.pyruns the real terminal report, the real reconciler and_run_chaton a tmp_path home; only the model and the next start's notifiers are stand-ins. Every wait carries a 30 s ceiling.Red: the four files on main (the three that changed copied onto it), three runs, 16 failed and 565 passed each time. Of the 16, nine are the first revision's. Three are behavioural: the given-up run already has a
deliveredtombstone where none is expected, the same for the parent-turn case, and "a notice queued for the parent agent's next turn is lost with the restart and not announced again". Four fail on the new names (settle_drained_subagent_notices,ORPHAN_NOTICE_OWED). Two are the gateway cases: the callback takes no record on main (TypeError), and the handler notes no debt. The other seven are this revision's bound cases: on main the ledger holds the whole command (assert 200005 <= 80), the scan's tombstone carries it on both routes, andorphan_carried_fieldsdoes not exist (fourImportError). On this branch's previous head (253471c) the new module andtest_every_tombstone_cause_has_a_terminal_statefail 8 cases and pass 8, 3 of 3 runs: the seven bound cases, and the scan, which readscausefrom the keywords of awrite_tombstonecall and no longer sawgateway_restartonce the call spread the record (assert {'delivered', 'reaped', 'timeout', 'gateway_restart'} <= causes).Mutations, one per rule: dropping the gate term in
terminal.pyfails the two given-up cases; ignoring the record in_orphan_notifyfails only its gateway case; skipping the note in the injection-failed handler fails only its case; removing the call in_run_chatfails only the parent-turn case; dropping the carried-record branch inwaves.pyfails only the two drained-orphan cases; skipping the owed branch in the reconciler fails the three cases that boot through it; not passing the record, or not returning the owed value, fails the two orphan cases that look for it; an emptysettle_drained_subagent_noticesfails the four drained cases. For the bound: building the notice's record from the stored fields fails the ledger case and theinjectedscan case; building the boot scan's tombstone from them fails thedigestcase only; spreading the record without the explicitcausekeyword fails onlytest_every_tombstone_cause_has_a_terminal_state.Test doubles updated: the two
_injectdoubles intest_subagent_boot_orphan_announcement.pyandtest_subagent_owed_tombstones.pytakedelivery=None, because the reconciler now passes it; the_imploffload pin intest_subagent_panel_durable_replay.pygoes from 6 to 7 for the carried-record write.Neighbours: the 36 test files that name the symbols this change edits (
SubagentDelivery,settle_queued_delivery,_pending_subagent_failures,on_orphan_notify,_report_terminal,_reconcile_orphans,orphan_resume_hint, the manager's_implbinding and the like), run serially: 3579 passed, 1 skipped. black, isort, flake8 (including F821), mypy on the six source files,check_comment_history,check_brand_name,docs-lint, the harness-parity gate, and the wait-discipline and flake-pattern ratchets on the head (it sits directly on origin/main; 92 passed) also pass.Manual verification
N/A: unit coverage runs the run loop's report, the reconciler, the gateway callbacks and
_run_chatfor real; it was not run against a live gateway.Related Issues
subagent.pyand the same_imploffload pin intest_subagent_panel_durable_replay.py;git merge-treeof the two heads conflicts in those two files, so whichever lands second rebases.Pattern harvest
Rule candidate: review-prompt
Pattern: a durable "delivered" mark written when a message is queued in process memory, not when its consumer takes it, turns a restart into a silent loss; keep the debt where the next start can see it until the consumer's drain settles it.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.