Repository navigation
fix: tombstone a boot orphan only after its notice is recorded - #18802
iamwhatever merged 1 commit into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Merge order: whichever of this PR and the owed-tombstone companion lands second must add What this change shipsInventory (6 items) — 6 justifiedIntent: when the gateway restarts mid-scan, no finished sub-agent run should end up announced to nobody. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 4b4382c Description read[DESCRIPTION-READ] 1b5aecece6df86251a32bf8cd529516689874adf59d4da11e4ce4536f035895f That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the reorder still holds once rebased onto current [DESIGN-REVIEWED] 4b4382c Description read[DESCRIPTION-READ] 226e9310217ca0e6e8c9c6823cbaf08203c1439ed779022a602e809bfcaddb5d That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
The boot orphan reconciler wrote each orphan's gateway_restart tombstone before it called _notify_orphan. The tombstone excludes the folder from every later reconciliation, so a gateway that died in between left the finished run announced to no one, ever. For a parent with no open tab the window spanned the whole scan, because its notice waits for one digest DM at the end. The reconciler now notifies first. A notice handed to a slot keeps its post-injection tombstone, which now states the outcome itself, and the boot's own tombstone is written when none landed. A notice bound for the owner DM has its tombstone written after the digest DM. A notifier or DM that raises still tombstones the run, so a broken notifier does not make it re-announce at every start. Only a process that dies between the notice and its tombstone leaves the run to the next start, which announces it again rather than never.
53fd2e3 to
4b4382c
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
The boot orphan reconciler wrote each orphan's gateway_restart tombstone before announcing it. A gateway that died in between therefore left a finished sub-agent run announced to no one, ever, because list_orphans skips tombstoned folders. This PR reverses the order at the root. A slot-injected notice is tombstoned right after injection (by _notify_orphan_impl, which now also states outcome=completed itself). A DM-bound notice is tombstoned only after the digest DM. A notifier that raises still tombstones after the scan, so it cannot loop at every boot. I read the whole monitoring.py change against the loop body. _notify_orphan_impl returns None only after a successful injection, so the _tombstone_unless_present branch runs only then, and it backfills the boot's own tombstone when the notifier wrote none. Every exit path that tombstoned before still tombstones, and the kill/audit path before it is unchanged. The only cost is a duplicate announcement if the process dies between notice and tombstone, which the spec prefers and the PR lists as a non-goal. The 7 tests run the real reconciler, and the two loss cases fail on main. CI is green and every lane passes. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A sub-agent run found at boot is tombstoned only after its notice has been recorded, so a gateway that dies during the scan cannot leave a finished run announced to no one.
The boot orphan reconciler (
subagent_manager/monitoring.py,_reconcile_orphans_impl) wrote each orphan'sgateway_restarttombstone before it called_notify_orphan. The tombstone excludes the folder from every later reconciliation (list_orphans()skips tombstoned folders), so a gateway that died between the two left the finished run announced to no one, ever. For a parent with no open tab the window was the whole scan, because that notice waits for one digest DM at the end.The spec's rule is the opposite: "An undelivered report abandoned at shutdown is made RECOVERABLE, not silently dropped" (
docs/system-specs/modules/subagent.md:2148).Why it matters
A parent agent that never hears that a sub-agent finished may wait for it, spawn it again, or report its result as missing. A duplicate announcement after a crash at the wrong moment is visible and harmless by comparison.
Not a goal
slot._pending_subagent_failuresuntil that turn, and the tombstone written after the injection already saysdelivered. That is a separate change: it needs a durable debt for the notice and a settle at both drains, and it follows in its own pull request.What changed (motivation → approach → change)
_reconcile_orphans_implbuilds the tombstone's fields and calls_notify_orphanfirst._notify_orphan_implwrites the tombstone right after the injection, as it already did, and now states the outcome itself, since no earlier tombstone carries it forward. If no tombstone landed (a notifier that took the notice but wrote none), the reconciler writes the boot's own, as before.cause="gateway_restart",recovery_action(deliveredafter an injection, the recovery action otherwise),pid,turns,last_tool, andoutcome="completed"for a finished run. The writes stay off the event loop.Backwards compatibility
Compatible: no persisted format changes; a tombstone written in the new order holds the same fields and values as before, and a normal boot still announces each orphan exactly once (
test_a_normal_boot_announces_each_orphan_exactly_once).Removes nothing: every orphan that was tombstoned before is still tombstoned, after its notice instead of before it; no route, field, file or config key is removed or renamed.
Tests
test/test_subagent_boot_orphan_announcement.pyruns the real reconciler on a tmp_path home with its notifiers recorded.main(843abe3) with only the test file added, three serial runs, the same each time: 2 failed, 5 passed.AssertionError: a finished run whose notice was never created is announced to no one at the next bootAssertionError: a run bound for the owner DM was tombstoned before the DM was sent and is announced to no onegateway_restart/delivered/completed;result_available/completed);result_available).test_runtime_reconcile.py::test_the_orphan_reconcile_withholds_a_leased_pidandtest_subagent_persistence.py::TestOrphanNotification::test_notification_failure_doesnt_crashassert that every reconciled orphan ends tombstoned, including when the notifier raises or takes the notice without writing one. An earlier revision of this change failed them; they pass unchanged.main.deliveryargument):git merge-treeof the two heads is clean. On that merged tree this file's_injecttest double, which takes(parent_session, msg, meta=None), is called withdelivery=, so three tests here fail until the double gainsdelivery=None. With that one-line change, this file, that change's test file andtest_subagent_persistence.pypass together on the merged tree: 150 passed. Whichever change lands second makes that edit.list_orphansorwrite_tombstone, run serially with this file first: 1995 passed.Manual verification
Not run against a live gateway.
Related Issues
git merge-treewith its current head is clean, and the merged reconciler keeps this change's order. If fix(subagents): end a leaked process at restart without re-announcing it #18078 lands first, this branch rebases onto it.monitoring.py;git merge-treereports a conflict there that this change adds.Pattern harvest
Rule candidate: review-prompt
Pattern: a durable "handled" mark (a tombstone or a delivered flag) written before the side effect it records turns a crash between the two into a silent loss; write the mark after the effect.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)