Skip to content

fix: retry a failed tombstone write instead of re-reporting the run - #18976

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/no-duplicate-completion-after-failed-tombstone
Oct 11, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
javenciu:fix/no-duplicate-completion-after-failed-tombstone

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent run whose tombstone write failed is reported to its parent once, not again at every later boot.

A sub-agent's tombstone.json is what keeps its folder out of the next boot's orphan reconciliation. Reconciliation announces every untombstoned run to its parent.

When that write fails, for example on a full disk, write_tombstone logs a WARNING and returns. No caller learns of the failure, so the folder stays untombstoned:

  • A delivered run. The result reaches the parent, then the delivered tombstone (mark_delivered, from terminal.py, waves.py and the batch settle) fails. The next boot announces the finished result to the parent a second time.
  • A boot's own tombstone. Reconciliation announces an orphan, then its gateway_restart tombstone fails. The following boot announces the same run again.

Why it matters

The parent conversation receives a completion it has already collected, and nothing in it tells the repeat apart from a real new completion. A parent agent may act on it twice.

Not a goal

What changed (motivation → approach → change)

  • Motivation: a failed tombstone write should be finished later rather than forgotten.
  • Approach: one hook inside write_tombstone, which every writer goes through: the delivered writes and the boot reconciler's write.
  • Change:
    • subagent_persistence.py, recording a failure: on an OSError the WARNING stays as it is, and the run id is recorded in a capped in-memory map together with the fields the write was given and its died. The cap is 4,096 ids. An entry over 16 KiB is not recorded on any write, including one that replaces an earlier record for the same run; that earlier record is dropped too, because it describes an older ending.
      • Past the cap the write is not retried, so the folder stays an orphan and is announced again at the next start. It is never lost.
      • Nothing is recorded when a tombstone is already on disk, because the folder is already out of reconciliation.
      • write_tombstone's signature is unchanged; its body moves to _write_tombstone, which also takes the retry flag.
    • subagent_persistence.py, the retry: retry_owed_tombstones() replays each record unchanged.
      • It creates the file exclusively: a temp file is hard-linked into place, so the content appears whole. The temp file's name is unique to the call (tempfile.mkstemp), so one left behind by a process that died mid-write never blocks a later retry.
      • It never replaces a tombstone another path wrote in the meantime, and never recreates a removed run folder. In both cases it drops the record.
      • A retry that fails again keeps the record and logs at DEBUG only. A retry that lands logs once at INFO.
      • One retry runs at a time: a lock is held for the whole retry. It takes a stop callable, read before each write, and a budget in seconds that bounds both its wait for a retry already running and the writes after that. Records still owed at the deadline stay owed, so their runs are reported again at the next start, as without this change.
    • subagent_manager/monitoring.py: the reaper calls the retry on the maintenance executor each tick, only when a record exists and the manager is not shutting down. It passes the manager's shutdown flag as stop, so a retry still writing when shutdown begins ends after the write it is in.
    • subagent_manager/cancellation.py: cancel_all calls the retry once in a worker thread with budget=_STATE_DRAIN_TIMEOUT (5 s, the bound the other off-loop writers get at shutdown) and awaits it for at most twice that, so a wedged disk cannot hold shutdown there. It does so before its report drain, and the lock makes it wait for a reaper retry still writing, so a straggler the drain re-admits to orphan recovery gets its tombstone cleared after every retry write, not before one.

Backwards compatibility

  • Nothing persisted changes shape. A retried tombstone holds exactly the fields, and the died, that the failed write would have held.
  • A write that succeeds behaves as before and leaves no record.
  • write_tombstone's callers are unchanged.

Tests

New file test/test_subagent_owed_tombstones.py. All state is under tmp_path, the reconciler's notifiers are recorders, and no prune runs.

  • Red on 843abe3, this branch's base, with only the new test file copied in: 13 failed, 1 passed, in each of 3 serial runs. The 3 behaviour tests fail for the stated reasons:
    • a run whose result already reached its parent was reported again at the next boot, although space came back before the restart (reaper tick);
    • the shutdown did not write the delivered tombstone it owed (graceful shutdown);
    • a run announced at one boot was announced again at the next, because the boot's own tombstone write had failed (boot tombstone).
      The 9 controls that use the new functions fail on the base with AttributeError, and so does the shutdown-race test, which patches the retry's exclusive create. That test stalls a reaper retry's first write, runs cancel_all, clears a second run's tombstone as the drain does for a straggler, then lets the write finish. Against the retry without the lock and stop, it fails with a retry that began before shutdown wrote a tombstone after the shutdown cleared it, 3 of 3 runs. The control that an undelivered orphan is announced exactly once passes there, as it should.
  • Controls:
    • a write that lands owes nothing;
    • an undelivered orphan is announced exactly once across two boots;
    • a retry never recreates a removed folder;
    • a retry never replaces a tombstone written in the meantime;
    • a retry killed after it created its temp file leaves that file behind, and the next retry still writes the tombstone;
    • the retry keeps the original fields and died;
    • a retry that fails again keeps its record and logs no WARNING;
    • past the cap, the failed write is not retried and the folder stays an orphan;
    • an oversized second failed write for a run already recorded is not stored, and the run's earlier record is not retried;
    • a retry past its budget writes nothing and keeps its records.
  • Green: 14 passed, in each of 3 serial runs.
  • Neighbour files (the 52 that name write_tombstone, mark_delivered, list_orphans, the reaper loop, _reconcile_orphans, cancel_all or the retry): 4,316 passed on the final commit.
    • An earlier revision failed test_taskq_reconcile.py::test_every_tombstone_cause_has_a_terminal_state. That test reads the cause argument of every call named _write_tombstone, and an earlier name for the new body function collided with that name. The body is now _write_tombstone_file; the test is unchanged.

Manual verification

Not run against a live gateway.

Screenshots / video

None; no UI change.

Related Issues

Pattern harvest

Rule candidate: review-prompt
Pattern: a durable write whose failure is logged and swallowed inside the function that performs it, which leaves every caller believing the state was recorded.

Other durable writes in subagent_persistence.py that log and carry on are not examined here.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 00:02
@javenciu
javenciu requested a review from buluoray October 11, 2026 00:02
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 4ab04a2b8d6da4cafb53bf421cf97bb7d1eb1aae touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 4ab04a2b8d6da4cafb53bf421cf97bb7d1eb1aae via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that no path brings a delivered run back to life without going through _LIVE_RUN_STATES (continue, resume or a taskq replay). If one does, a stale owed record can tombstone the newer run.

[DESIGN-REVIEWED] 4ab04a2

Description read

[DESCRIPTION-READ] f4cba8c84219178f1a37023ffe9d0e08b5c1788fcb1d3ae9df21798136bf355f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 4ab04a2b8d6da4cafb53bf421cf97bb7d1eb1aae via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The 16 KiB per-entry bound, and its rule that drops the earlier record, guard against a field size that no shipped write_tombstone caller has been shown to produce.

Not justified as shipped

  • Item 3 — oversized: the diff adds "An entry over 16 KiB is not recorded on any write… that earlier record is dropped too" without naming a caller whose fields get that big. The richest caller, SubagentManager._write_tombstone (subagent.py:6121), sends detail already cut to MAX_ERROR_DETAIL_LEN (2,000, process_identity.py:100), plus scalar fields and last_tool. I read the call sites in the 4 non-test files (subagent.py, run.py, terminal.py, cancellation.py) plus mark_delivered, and none passes unbounded extras.
    Subtraction: delete _OWED_TOMBSTONE_MAX_BYTES, the json.dumps size probe and its drop-on-replace branch in _owe_tombstone, and keep only the 4,096-count cap.
    Clears when: a write_tombstone caller whose extras can go over 16 KiB is named.

What this change ships

Inventory (7 items) — 6 justified

Intent: a finished sub-agent run whose tombstone write failed gets reported to its parent once, not again at every boot after. This is a FIX.

  1. A failed tombstone write is remembered in memory, not just logged — justified
  2. The reaper writes remembered tombstones each tick once space returns — justified
  3. A remembered write with fields over 16 KiB is not retried, and drops its earlier record — oversized
  4. Graceful shutdown makes one last retry, with a time limit — justified
  5. A retry never replaces a newer tombstone and never recreates a deleted folder — justified
  6. Only one retry runs at a time, and a reaper retry stops once shutdown begins — justified
  7. A retry that lands logs once at INFO; a retry that fails again logs at DEBUG — justified

[FIRST-PRINCIPLES-REVIEWED] 4ab04a2

Description read

[DESCRIPTION-READ] e6518701bb51972e23da266cffe4e274dc91deee7c65599c58a7d31457bbe3c6

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 4ab04a2b8d6da4cafb53bf421cf97bb7d1eb1aae via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4ab04a2

@github-actions

github-actions Bot commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 4ab04a2b8d6da4cafb53bf421cf97bb7d1eb1aae via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 4ab04a2

When a sub-agent's tombstone write failed (a full disk), write_tombstone
logged a warning and carried on, so the folder stayed in the next boot's
orphan reconciliation and the parent was sent a result it had already
collected, or a run already announced at one boot was announced again
at the next.

A failed write is now recorded with the fields and the moment it was
given, in a capped in-memory map. The reaper retries the records off the
event loop each tick and graceful shutdown once more, before it
re-admits undelivered runs to orphan recovery. A retry creates the
tombstone only if none exists and the run folder is still there, and a
retry that fails again logs at DEBUG. Past the cap the folder stays an
orphan: reported again at the next start, never lost.
@javenciu
javenciu force-pushed the fix/no-duplicate-completion-after-failed-tombstone branch from 1a62c5e to 4ab04a2 Compare October 11, 2026 10:26
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this PR, a failed tombstone.json write (a full disk) was only logged, so the next boot's orphan reconciliation announced an already-delivered run to its parent a second time. The PR fixes this at the single write site: _write_tombstone_file records the failed write's fields and its died time in a capped in-memory map. The reaper retries those records off-loop each tick, and cancel_all makes one bounded final retry. I read subagent_persistence.py, monitoring.py and cancellation.py in full and tried to make a retry do harm. A tombstone that already exists blocks recording (_owe_tombstone checks for it). The retry uses an exclusive link-create, so it can never replace a tombstone another path wrote, and a removed folder gives FileNotFoundError, so it never recreates one. A newer failure for the same run replaces the record, and the delete compares by identity. The Design lane asked whether a continued run could get a stale tombstone. It cannot: continuation mints a new agent id (continuation.py _mint_agent_id), so the old folder is never revived. The shutdown retry is bounded by budget plus wait_for and waits on the retry lock, so a reaper write cannot land after the drain clears a straggler. The test covers that race, and it fails without the lock. The First Principles CONCERNS about the 16 KiB per-entry cap is over-engineering, not a defect, and is harmless: an oversized entry falls back to today's behaviour of announcing the run again. The tests are red on the base and green on the head, checks are green, and there are no change requests. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever enabled auto-merge (squash) October 11, 2026 18:07
@iamwhatever
iamwhatever merged commit 2a2fcc9 into kirodotdev:main Oct 11, 2026
104 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 11, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants