Skip to content

fix: tombstone a boot orphan only after its notice is recorded - #18802

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/boot-orphan-announcement-is-never-lost
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/boot-orphan-announcement-is-never-lost

Conversation

@javenciu

@javenciu javenciu commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent run found at boot is tombstoned only after its notice has been recorded, so a gateway that dies during the scan cannot leave a finished run announced to no one.

The boot orphan reconciler (subagent_manager/monitoring.py, _reconcile_orphans_impl) wrote each orphan's gateway_restart tombstone before it called _notify_orphan. The tombstone excludes the folder from every later reconciliation (list_orphans() skips tombstoned folders), so a gateway that died between the two left the finished run announced to no one, ever. For a parent with no open tab the window was the whole scan, because that notice waits for one digest DM at the end.

The spec's rule is the opposite: "An undelivered report abandoned at shutdown is made RECOVERABLE, not silently dropped" (docs/system-specs/modules/subagent.md:2148).

Why it matters

A parent agent that never hears that a sub-agent finished may wait for it, spawn it again, or report its result as missing. A duplicate announcement after a crash at the wrong moment is visible and harmless by comparison.

Not a goal

  • Exactly-once announcement. A crash between the notice and its tombstone announces the run once more at the next boot, which the spec prefers to losing it.
  • A notice lost with a restart before the parent's next turn. The agent's copy of a slot-injected notice waits in the in-memory slot._pending_subagent_failures until that turn, and the tombstone written after the injection already says delivered. That is a separate change: it needs a durable debt for the notice and a settle at both drains, and it follows in its own pull request.
  • Changing what the notice says, when it is delivered, or the user's transcript card.

What changed (motivation → approach → change)

  • Motivation: record that an orphan was handled only after it was handled.
  • Approach: notify first, tombstone after.
  • Change: _reconcile_orphans_impl builds the tombstone's fields and calls _notify_orphan first.
    • A notice handed to a slot: _notify_orphan_impl writes the tombstone right after the injection, as it already did, and now states the outcome itself, since no earlier tombstone carries it forward. If no tombstone landed (a notifier that took the notice but wrote none), the reconciler writes the boot's own, as before.
    • A notice bound for the owner DM: its tombstone is held and written only after the digest DM is sent.
    • A notifier or DM that raises still tombstones, as before, since it would raise again at every start. That tombstone is written after the scan, with the DM-bound ones, so the reconciler keeps a single write site. Only a process that dies before the write leaves the run to the next start, which announces it again.
    • The tombstone's content is unchanged: cause="gateway_restart", recovery_action (delivered after an injection, the recovery action otherwise), pid, turns, last_tool, and outcome="completed" for a finished run. The writes stay off the event loop.

Backwards compatibility

Compatible: no persisted format changes; a tombstone written in the new order holds the same fields and values as before, and a normal boot still announces each orphan exactly once (test_a_normal_boot_announces_each_orphan_exactly_once).

Removes nothing: every orphan that was tombstoned before is still tombstoned, after its notice instead of before it; no route, field, file or config key is removed or renamed.

Tests

test/test_subagent_boot_orphan_announcement.py runs the real reconciler on a tmp_path home with its notifiers recorded.

  • Red on the current main (843abe3) with only the test file added, three serial runs, the same each time: 2 failed, 5 passed.
    • AssertionError: a finished run whose notice was never created is announced to no one at the next boot
    • AssertionError: a run bound for the owner DM was tombstoned before the DM was sent and is announced to no one
  • Controls:
    • a normal boot announces each orphan exactly once, and its tombstone reads gateway_restart / delivered / completed;
    • a DM-bound orphan is announced once and tombstoned after the DM (result_available / completed);
    • a run delivered before the restart is not announced;
    • a digest DM that raises still tombstones the run, so the next boot is quiet;
    • a notifier that takes the notice but writes no tombstone still leaves the run tombstoned (result_available).
  • Existing tests, unchanged: test_runtime_reconcile.py::test_the_orphan_reconcile_withholds_a_leased_pid and test_subagent_persistence.py::TestOrphanNotification::test_notification_failure_doesnt_crash assert that every reconciled orphan ends tombstoned, including when the notifier raises or takes the notice without writing one. An earlier revision of this change failed them; they pass unchanged.
  • Green: 7 passed, in three consecutive runs on this branch rebased onto that main.
  • With the companion owed-tombstone change (the one that records an undelivered notice on the slot's ledger and passes the notifier a delivery argument): git merge-tree of the two heads is clean. On that merged tree this file's _inject test double, which takes (parent_session, msg, meta=None), is called with delivery=, so three tests here fail until the double gains delivery=None. With that one-line change, this file, that change's test file and test_subagent_persistence.py pass together on the merged tree: 150 passed. Whichever change lands second makes that edit.
  • Neighbours: the 27 test files that name the reconciler, the orphan notifiers, list_orphans or write_tombstone, run serially with this file first: 1995 passed.

Manual verification

Not run against a live gateway.

Related Issues

Pattern harvest

Rule candidate: review-prompt
Pattern: a durable "handled" mark (a tombstone or a delivered flag) written before the side effect it records turns a crash between the two into a silent loss; write the mark after the effect.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 10, 2026 09:08
@javenciu
javenciu requested a review from cruisercohen October 10, 2026 09:08
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 10, 2026
@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 4b4382c673b11ca03405d74e9cce92e351ddd193 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 4b4382c673b11ca03405d74e9cce92e351ddd193 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Merge order: whichever of this PR and the owed-tombstone companion lands second must add delivery=None to this file's _inject double.

What this change ships

Inventory (6 items) — 6 justified

Intent: when the gateway restarts mid-scan, no finished sub-agent run should end up announced to nobody. This is a FIX.

  1. A boot orphan whose notice a slot took is now tombstoned after the injection, not before — justified
  2. An orphan whose notice goes to the owner DM is now tombstoned only after the digest DM is sent — justified
  3. A run whose notifier raised is now tombstoned after the scan instead of inline — justified
  4. Fallback tombstone when a slot took the notice but no tombstone landed — justified
  5. The tombstone written after an injection now records outcome=completed itself — justified
  6. New test file pins the order: 2 cases fail on base, plus controls — justified

[FIRST-PRINCIPLES-REVIEWED] 4b4382c

Description read

[DESCRIPTION-READ] 1b5aecece6df86251a32bf8cd529516689874adf59d4da11e4ce4536f035895f

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 4b4382c673b11ca03405d74e9cce92e351ddd193 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the reorder still holds once rebased onto current main, where the reconciler has been restructured (kills now go through _end_orphan_process).

[DESIGN-REVIEWED] 4b4382c

Description read

[DESCRIPTION-READ] 226e9310217ca0e6e8c9c6823cbaf08203c1439ed779022a602e809bfcaddb5d

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 4b4382c673b11ca03405d74e9cce92e351ddd193 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4b4382c

@github-actions

github-actions Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 4b4382c673b11ca03405d74e9cce92e351ddd193 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4b4382c

The boot orphan reconciler wrote each orphan's gateway_restart
tombstone before it called _notify_orphan. The tombstone excludes the
folder from every later reconciliation, so a gateway that died in
between left the finished run announced to no one, ever. For a parent
with no open tab the window spanned the whole scan, because its notice
waits for one digest DM at the end.

The reconciler now notifies first. A notice handed to a slot keeps its
post-injection tombstone, which now states the outcome itself, and the
boot's own tombstone is written when none landed. A notice bound for
the owner DM has its tombstone written after the digest DM. A notifier
or DM that raises still tombstones the run, so a broken notifier does
not make it re-announce at every start. Only a process that dies
between the notice and its tombstone leaves the run to the next start,
which announces it again rather than never.
@javenciu
javenciu force-pushed the fix/boot-orphan-announcement-is-never-lost branch from 53fd2e3 to 4b4382c Compare October 11, 2026 06:47
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The boot orphan reconciler wrote each orphan's gateway_restart tombstone before announcing it. A gateway that died in between therefore left a finished sub-agent run announced to no one, ever, because list_orphans skips tombstoned folders. This PR reverses the order at the root. A slot-injected notice is tombstoned right after injection (by _notify_orphan_impl, which now also states outcome=completed itself). A DM-bound notice is tombstoned only after the digest DM. A notifier that raises still tombstones after the scan, so it cannot loop at every boot. I read the whole monitoring.py change against the loop body. _notify_orphan_impl returns None only after a successful injection, so the _tombstone_unless_present branch runs only then, and it backfills the boot's own tombstone when the notifier wrote none. Every exit path that tombstoned before still tombstones, and the kill/audit path before it is unchanged. The only cost is a duplicate announcement if the process dies between notice and tombstone, which the spec prefers and the PR lists as a non-goal. The 7 tests run the real reconciler, and the two loss cases fail on main. CI is green and every lane passes. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 18:06
@iamwhatever
iamwhatever merged commit dfd2d31 into kirodotdev:main Oct 11, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants