Skip to content

fix(subagent): keep a queued notice recoverable until it drains - #19415

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/undelivered-subagent-notice-survives-a-restart
Oct 12, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/undelivered-subagent-notice-survives-a-restart

Conversation

@javenciu

@javenciu javenciu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent notice the parent agent has not yet taken (a report the gateway gave up on, or a restart orphan's notice) leaves its run visible to the next start, so a restart before the parent's next turn announces the run again instead of losing it.

Both notices reach the parent agent only through slot._pending_subagent_failures, an in-memory list that the parent's next turn empties (dashboard/chat_runner.py:11395-11397 on c4c51aa). The run's folder is tombstoned delivered before that, and a tombstone excludes the folder from every later reconciliation (list_orphans()), so a restart in between loses the notice and nothing announces the run.

  • A report the gateway gave up on. notify_injection_failed sets info._report_undelivered (subagent_manager/terminal.py:1420 on c4c51aa) and the subagent_injection_failed handler queues the failure text in that list with slot._pending_subagent_failures.append (slack/gateway.py:11127 on c4c51aa). The run loop's delivered gate does not read the flag (subagent_manager/terminal.py:399-402 on c4c51aa), so the run still gets its tombstone. The leaked-process branch beside it does read it (subagent_manager/terminal.py:434 on c4c51aa).
  • A restart orphan. _orphan_notify only queues the text and returns True (slack/gateway.py:11206-11209 on c4c51aa), and _notify_orphan_impl then writes recovery_action="delivered" (subagent_manager/monitoring.py:826 on c4c51aa).

#18802 made the reconciler tombstone an orphan only after its notice is recorded, and named this follow-up in its Not a goal.

Why it matters

A parent agent that is never told a sub-agent finished, or was cut off, may wait for it, spawn it again or report its result as missing. The user still sees the transcript card, so what is lost is the agent's own copy, which is the one that drives its next step.

Not a goal

  • Settling after the model has read the notice, as the queued-announce path does (_arm_queued_delivery_settlement, dashboard/chat_runner.py:6612). The drain settles when it takes the text, so a restart between the drain and the model reading it still loses the notice, as before.
  • Persisting _pending_subagent_failures itself, or changing what the parent is told: the text, the card and the owner DM are as they were.
  • The owner-digest ordering of the reconciler, which Evaluate tombstoning restart-orphan digest runs only after delivery is confirmed #19171 (open) asks about.
  • _retrigger_recovery in slack/gateway.py, which empties the same list. Nothing on main calls it except its own done-callback (git grep -n _retrigger_recovery finds its definition and that call), so it is left alone.

What changed (motivation → approach → change)

Root cause: the delivered tombstone is written when a notice is queued in memory, not when the parent takes it.

Approach: the tombstone a queued notice would have written becomes a debt on the slot's delivery ledger (note_pending_subagent_delivery and take_pending_subagent_deliveries), keyed on the queued text. A queued announce already uses that ledger, which holds at most 128 entries per slot; an entry evicted past that leaves its run for the next start to announce. The drain that takes the text settles the debt through the manager (settle_queued_delivery).

  • subagent_manager/terminal.py:407: the run loop's delivered gate also requires not info._report_undelivered, as the leaked-process branch does (:439).
  • slack/gateway.py:11132: the subagent_injection_failed handler notes a completed run's debt under the queued text. A failed or stopped run was never going to get the delivered tombstone.
  • SubagentDelivery gains tombstone (subagent.py:1095), the record a restart orphan's notice owes (the boot fields, outcome included), and ORPHAN_NOTICE_OWED (subagent.py:1101) is what on_orphan_notify returns when it queued the notice and took that record.
  • _orphan_notify (slack/gateway.py:11188) accepts the record, notes it under the queued text and returns ORPHAN_NOTICE_OWED. The record is built once in _notify_orphan_impl and passed on (subagent_manager/monitoring.py:869), then shared with the write that follows an injection that took no debt (subagent_manager/monitoring.py:883, which passes cause="gateway_restart" as a keyword, the form test_every_tombstone_cause_has_a_terminal_state reads from the call). _reconcile_orphans_impl (subagent_manager/monitoring.py:545) sends no owner DM and writes no tombstone for an owed notice.
  • orphan_carried_fields (subagent_manager/monitoring.py:91) builds the three fields that record copies from the run's state.json, pid, turns and last_tool, for the record a queued notice owes and for the boot scan's own tombstone. The slot ledger caps how many records it holds (128), not how large each one is, and last_tool is the backend's tool title, which for a shell tool is the whole command. It is flattened to one line, redacted over the whole value and cut to the 80 characters the notice already shows (_ORPHAN_LAST_TOOL_CHARS, subagent_manager/monitoring.py:88), and a pid or turns that is not an integer reads as absent.
  • settle_drained_subagent_notices (dashboard/chat_runner.py:6719) is called where _run_chat empties the list (dashboard/chat_runner.py:11439).
  • settle_queued_delivery_impl (subagent_manager/waves.py:467) writes a carried record in place of the delivered mark, so a cut-off or failed orphan is not relabelled completed.
  • docs/system-specs/modules/subagent.md:3465 states the rule.

The one behaviour change: the run loop no longer writes the delivered tombstone of a report the gateway gave up on. The parent's turn writes it when it drains the notice. A restart before then, or a parent with no tab to drain it, leaves the folder to the next start, which announces the run once.

Backwards compatibility

Compatible: no persisted format changes. Every tombstone written has the fields it had before, and a normal boot still announces each orphan once. on_orphan_notify gains an optional delivery argument; the gateway's callback is its only implementation in src.

Removes nothing: no route, field, file or config key is removed or renamed.

Tests

In short: on main (c4c51aa) the four test files that carry the new cases fail 16 of them and pass 565 other cases, the same sixteen in each of 3 runs; on this branch the same files pass 581, 3 of 3 runs. Mutations: 12 of 12 fail only their own rule's cases.

New cases, by rule:

Rule Cases
A report the gateway gave up on keeps no delivered tombstone, and a restart announces it again test_a_restart_before_the_drain_announces_a_given_up_report_again
The parent's turn settles the notice it drains, and the next start is quiet test_the_parent_turn_settles_the_notice_it_drains, test_a_drained_notice_settles_the_tombstone_and_is_not_announced_again
A restart orphan's notice is reported as owed, and a restart announces it again test_an_owed_notice_is_reported_as_owed_not_as_delivered, test_a_restart_before_the_drain_announces_a_boot_orphan_again
A drained orphan notice keeps its recorded outcome (finished, lost) and is announced once test_a_drained_boot_orphan_notice_keeps_its_outcome_and_is_announced_once[finished|lost]
The gateway notes a completed run's debt, and only that run's test_subagent_event_injection_failed_notes_the_debt_of_a_completed_run, test_subagent_event_injection_failed_notes_no_debt_for_an_unfinished_run[failed|stopped]
_orphan_notify takes the record under the queued text test_notify_takes_the_notice_tombstone_under_the_queued_text
Control: a delivered report is still tombstoned by the run loop test_a_delivered_report_is_still_tombstoned_by_the_run_loop
The record a queued notice owes is bounded where it is built: a 200,000-character last_tool is held at 80 on the slot ledger test_the_record_a_queued_notice_owes_is_bounded_where_it_is_built
The boot scan's own tombstone, on the injected and on the digest route, carries the bounded title and cause gateway_restart test_a_tombstone_the_scan_writes_carries_the_bounded_title[injected|digest]
The carried fields keep only the types the run stores (a string pid, a boolean turns, a list last_tool read as absent) test_the_carried_fields_keep_only_the_types_the_run_stores[as-stored|strings-and-lists|booleans-and-numbers|absent]
Every tombstone cause in src still has a terminal task state, and the scan sees gateway_restart in the write after a delivered notice test_every_tombstone_cause_has_a_terminal_state (test_taskq_reconcile.py, existing)

test_subagent_undelivered_notice_restart.py runs the real terminal report, the real reconciler and _run_chat on a tmp_path home; only the model and the next start's notifiers are stand-ins. Every wait carries a 30 s ceiling.

Red: the four files on main (the three that changed copied onto it), three runs, 16 failed and 565 passed each time. Of the 16, nine are the first revision's. Three are behavioural: the given-up run already has a delivered tombstone where none is expected, the same for the parent-turn case, and "a notice queued for the parent agent's next turn is lost with the restart and not announced again". Four fail on the new names (settle_drained_subagent_notices, ORPHAN_NOTICE_OWED). Two are the gateway cases: the callback takes no record on main (TypeError), and the handler notes no debt. The other seven are this revision's bound cases: on main the ledger holds the whole command (assert 200005 <= 80), the scan's tombstone carries it on both routes, and orphan_carried_fields does not exist (four ImportError). On this branch's previous head (253471c) the new module and test_every_tombstone_cause_has_a_terminal_state fail 8 cases and pass 8, 3 of 3 runs: the seven bound cases, and the scan, which reads cause from the keywords of a write_tombstone call and no longer saw gateway_restart once the call spread the record (assert {'delivered', 'reaped', 'timeout', 'gateway_restart'} <= causes).

Mutations, one per rule: dropping the gate term in terminal.py fails the two given-up cases; ignoring the record in _orphan_notify fails only its gateway case; skipping the note in the injection-failed handler fails only its case; removing the call in _run_chat fails only the parent-turn case; dropping the carried-record branch in waves.py fails only the two drained-orphan cases; skipping the owed branch in the reconciler fails the three cases that boot through it; not passing the record, or not returning the owed value, fails the two orphan cases that look for it; an empty settle_drained_subagent_notices fails the four drained cases. For the bound: building the notice's record from the stored fields fails the ledger case and the injected scan case; building the boot scan's tombstone from them fails the digest case only; spreading the record without the explicit cause keyword fails only test_every_tombstone_cause_has_a_terminal_state.

Test doubles updated: the two _inject doubles in test_subagent_boot_orphan_announcement.py and test_subagent_owed_tombstones.py take delivery=None, because the reconciler now passes it; the _impl offload pin in test_subagent_panel_durable_replay.py goes from 6 to 7 for the carried-record write.

Neighbours: the 36 test files that name the symbols this change edits (SubagentDelivery, settle_queued_delivery, _pending_subagent_failures, on_orphan_notify, _report_terminal, _reconcile_orphans, orphan_resume_hint, the manager's _impl binding and the like), run serially: 3579 passed, 1 skipped. black, isort, flake8 (including F821), mypy on the six source files, check_comment_history, check_brand_name, docs-lint, the harness-parity gate, and the wait-discipline and flake-pattern ratchets on the head (it sits directly on origin/main; 92 passed) also pass.

Manual verification

N/A: unit coverage runs the run loop's report, the reconciler, the gateway callbacks and _run_chat for real; it was not run against a live gateway.

Related Issues

Pattern harvest

Rule candidate: review-prompt
Pattern: a durable "delivered" mark written when a message is queued in process memory, not when its consumer takes it, turns a restart into a silent loss; keep the debt where the next start can see it until the consumer's drain settles it.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

A notice for the parent agent waits in slot._pending_subagent_failures,
which lives only in memory, until the parent's next turn takes it. Two
paths tombstoned the run as delivered before that, so a restart in
between lost the notice and the next start never announced the run:

- a report the gateway gave up on (_report_undelivered) still got the
  run loop's delivered tombstone;
- a boot orphan's notice was tombstoned delivered as soon as the slot
  queued it.

The run loop now skips the delivered mark for a given-up report, and
both notices record the tombstone they owe on the slot's delivery
ledger under the queued text. The parent's turn settles the debt when it
drains the text (settle_drained_subagent_notices), writing the carried
record, so a cut-off or failed orphan is not relabelled completed. A
restart before the drain leaves the folder to the next start, which
announces the run once.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 12:11
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 1a3613bb86b8460b2a09d16379e41b9a9b180bbc touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 1a3613bb86b8460b2a09d16379e41b9a9b180bbc via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

settle_drained_subagent_notices copies the take-and-settle body of _arm_queued_delivery_settlement._on_turn_done almost line for line, so one settlement path now has two copies.

Not justified as shipped

  • Item 3 — duplicate of src/kiro_crew/dashboard/chat_runner.py:6675-6711: the new function (chat_runner.py:6719 in the diff) repeats the same take_pending_subagent_deliveries → settle_queued_delivery → iscoroutine → _background_tasks sequence. Only the trigger differs.
    Subtraction: keep one body. Hoist the _on_turn_done tail into a single function and have both the drain and the callback call it.
    Clears when: one copy of the take-and-settle sequence remains.
  • Item 6 — rides along: orphan_carried_fields now also changes the boot scan's own on-disk tombstone (monitoring.py:516). last_tool is cut to 80 characters and a non-integer pid or turns is dropped. The stated harm, "the ledger's entry cap bounds how many records it holds", covers only the in-memory record. Separately, "Every tombstone written has the fields it had before" is true of the fields but not of their values.

What this change ships

Inventory (6 items) — 4 justified

Intent: keep a parent agent's sub-agent notice from being lost when the gateway restarts before the parent's next turn reads it. FIX.

  1. A report the gateway gave up on is no longer tombstoned until the parent's turn takes its notice — justified
  2. A restart orphan's notice queued for the parent: no tombstone and no owner DM until the parent's turn takes it — justified
  3. The parent's turn now settles the notices it takes — duplicate of _arm_queued_delivery_settlement body
  4. SubagentDelivery.tombstone field and ORPHAN_NOTICE_OWED return value — justified
  5. A settled orphan notice keeps its recorded outcome instead of reading as completed — justified
  6. The boot scan's own tombstone now has a shortened last_tool and type-checked pid/turns — rides along, no harm named for the on-disk write

[FIRST-PRINCIPLES-REVIEWED] 1a3613b

Description read

[DESCRIPTION-READ] 4f2c961f299cc6625589367e59cad5c41e23e1ade6492c3bed6ac426fdd352c8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 1a3613bb86b8460b2a09d16379e41b9a9b180bbc via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that _retrigger_recovery (slack/gateway.py:9575) really is dead: it empties _pending_subagent_failures without settling the debt, so if it is live, drained notices get announced again on restart.

[DESIGN-REVIEWED] 1a3613b

Description read

[DESCRIPTION-READ] e7e70b12ea26d2a3b0850d4608ac4e4e6c0a891d5b1e4b5aece08d0ec43c0872

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 1a3613bb86b8460b2a09d16379e41b9a9b180bbc via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 1a3613b

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 1a3613bb86b8460b2a09d16379e41b9a9b180bbc via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 1a3613b

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
The record a queued orphan notice owes rides the slot ledger in memory
until the parent's next turn drains it. The ledger's entry cap bounds how
many records it holds, not how large each is, and the record copied
last_tool, pid and turns from the run's state.json as they were stored.
last_tool is the backend's tool title, which for a shell tool is the whole
command, and tool diagnostics persist it uncut.

orphan_carried_fields now builds those three fields for both orphan
tombstones (the one a delivered notice writes or owes, and the one the
boot scan writes): last_tool is flattened to one line, redacted over the
whole value and cut to the 80 characters the notice already shows, and a
pid or turns that is not the integer the run writes reads as absent.

The write after a delivered notice passes cause="gateway_restart" as an
explicit keyword again. test_every_tombstone_cause_has_a_terminal_state
reads the causes from the keywords of those calls, and a ** spread of the
record hid this one from it. The notice's delivery record carries the
same cause for the drain's write.
@javenciu

Copy link
Copy Markdown
Contributor Author

Two things on the previous head, both fixed in the new commit 1a3613b (on top of 253471c; origin/main is still c4c51aa, so there is no rebase).

  1. test_every_tombstone_cause_has_a_terminal_state (Backend Tests s2 and Windows s2). _notify_orphan_impl wrote the post-injection tombstone with **record, and that test reads each cause from the keywords of the write_tombstone calls, so it no longer saw gateway_restart. The call passes cause="gateway_restart" as a keyword again (src/kiro_crew/subagent_manager/monitoring.py:883), and the record the queued notice owes carries the same cause for the drain's write (monitoring.py:870). The test passes on this head.

  2. "Carried tombstones bypass the delivery ledger's memory bound" (monitoring.py, the last_tool field). Agreed: the slot ledger caps how many records it holds (128), not how large each is, and last_tool is the backend's tool title, which for a shell tool is the whole command. orphan_carried_fields (monitoring.py:91) now builds pid, turns and last_tool for the record a queued notice owes and for the boot scan's own tombstone. last_tool is flattened to one line, redacted over the whole value and cut to the 80 characters the notice already shows (_ORPHAN_LAST_TOOL_CHARS in monitoring.py:88), and a pid or turns that is not an integer reads as absent. I cut the field instead of leaving oversized records untombstoned: with the cut no record is oversized, so every orphan still gets its tombstone and its single announcement.

Tests (test/test_subagent_undelivered_notice_restart.py, three new tests, seven cases): the ledger holds a 200,000-character command at 80 characters, the scan's tombstone carries the cut title on the injected and on the digest route, and the helper keeps only the types the run stores. On the previous head the new module plus test_every_tombstone_cause_has_a_terminal_state fail 8 cases and pass 8, 3 of 3 runs; on this head they pass 16, 3 of 3. The four files that carry the cases have 581 cases: 16 fail on main and all pass here, 3 of 3 runs each; the 36 neighbour files pass 3579 (1 skipped); the wait-discipline and flake-pattern ratchets on the head pass (92).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maintainer approval after a full diff read on head 1a3613bb86: diff matches the description, no regression to existing behaviour found, CI and readiness green.

@bolichen97
bolichen97 merged commit fe80076 into kirodotdev:main Oct 12, 2026
106 of 108 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants