Repository navigation
fix: retry a failed tombstone write instead of re-reporting the run - #18976
iamwhatever merged 1 commit into
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that no path brings a delivered run back to life without going through [DESIGN-REVIEWED] 4ab04a2 Description read[DESCRIPTION-READ] f4cba8c84219178f1a37023ffe9d0e08b5c1788fcb1d3ae9df21798136bf355f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The 16 KiB per-entry bound, and its rule that drops the earlier record, guard against a field size that no shipped Not justified as shipped
What this change shipsInventory (7 items) — 6 justifiedIntent: a finished sub-agent run whose tombstone write failed gets reported to its parent once, not again at every boot after. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 4ab04a2 Description read[DESCRIPTION-READ] e6518701bb51972e23da266cffe4e274dc91deee7c65599c58a7d31457bbe3c6 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
When a sub-agent's tombstone write failed (a full disk), write_tombstone logged a warning and carried on, so the folder stayed in the next boot's orphan reconciliation and the parent was sent a result it had already collected, or a run already announced at one boot was announced again at the next. A failed write is now recorded with the fields and the moment it was given, in a capped in-memory map. The reaper retries the records off the event loop each tick and graceful shutdown once more, before it re-admits undelivered runs to orphan recovery. A retry creates the tombstone only if none exists and the run folder is still there, and a retry that fails again logs at DEBUG. Past the cap the folder stays an orphan: reported again at the next start, never lost.
1a62c5e to
4ab04a2
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
Before this PR, a failed tombstone.json write (a full disk) was only logged, so the next boot's orphan reconciliation announced an already-delivered run to its parent a second time. The PR fixes this at the single write site: _write_tombstone_file records the failed write's fields and its died time in a capped in-memory map. The reaper retries those records off-loop each tick, and cancel_all makes one bounded final retry. I read subagent_persistence.py, monitoring.py and cancellation.py in full and tried to make a retry do harm. A tombstone that already exists blocks recording (_owe_tombstone checks for it). The retry uses an exclusive link-create, so it can never replace a tombstone another path wrote, and a removed folder gives FileNotFoundError, so it never recreates one. A newer failure for the same run replaces the record, and the delete compares by identity. The Design lane asked whether a continued run could get a stale tombstone. It cannot: continuation mints a new agent id (continuation.py _mint_agent_id), so the old folder is never revived. The shutdown retry is bounded by budget plus wait_for and waits on the retry lock, so a reaper write cannot land after the drain clears a straggler. The test covers that race, and it fails without the lock. The First Principles CONCERNS about the 16 KiB per-entry cap is over-engineering, not a defect, and is harmless: an oversized entry falls back to today's behaviour of announcing the run again. The tests are red on the base and green on the head, checks are green, and there are no change requests. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A sub-agent run whose tombstone write failed is reported to its parent once, not again at every later boot.
A sub-agent's
tombstone.jsonis what keeps its folder out of the next boot's orphan reconciliation. Reconciliation announces every untombstoned run to its parent.When that write fails, for example on a full disk,
write_tombstonelogs a WARNING and returns. No caller learns of the failure, so the folder stays untombstoned:deliveredtombstone (mark_delivered, fromterminal.py,waves.pyand the batch settle) fails. The next boot announces the finished result to the parent a second time.gateway_restarttombstone fails. The following boot announces the same run again.Why it matters
The parent conversation receives a completion it has already collected, and nothing in it tells the repeat apart from a real new completion. A parent agent may act on it twice.
Not a goal
fsynccannot be interrupted from inside the process, and on such a disk the drain's ownclear_tombstone, a synchronous unlink, blocks as well.What changed (motivation → approach → change)
write_tombstone, which every writer goes through: the delivered writes and the boot reconciler's write.subagent_persistence.py, recording a failure: on anOSErrorthe WARNING stays as it is, and the run id is recorded in a capped in-memory map together with the fields the write was given and itsdied. The cap is 4,096 ids. An entry over 16 KiB is not recorded on any write, including one that replaces an earlier record for the same run; that earlier record is dropped too, because it describes an older ending.write_tombstone's signature is unchanged; its body moves to_write_tombstone, which also takes the retry flag.subagent_persistence.py, the retry:retry_owed_tombstones()replays each record unchanged.tempfile.mkstemp), so one left behind by a process that died mid-write never blocks a later retry.stopcallable, read before each write, and abudgetin seconds that bounds both its wait for a retry already running and the writes after that. Records still owed at the deadline stay owed, so their runs are reported again at the next start, as without this change.subagent_manager/monitoring.py: the reaper calls the retry on the maintenance executor each tick, only when a record exists and the manager is not shutting down. It passes the manager's shutdown flag asstop, so a retry still writing when shutdown begins ends after the write it is in.subagent_manager/cancellation.py:cancel_allcalls the retry once in a worker thread withbudget=_STATE_DRAIN_TIMEOUT(5 s, the bound the other off-loop writers get at shutdown) and awaits it for at most twice that, so a wedged disk cannot hold shutdown there. It does so before its report drain, and the lock makes it wait for a reaper retry still writing, so a straggler the drain re-admits to orphan recovery gets its tombstone cleared after every retry write, not before one.Backwards compatibility
died, that the failed write would have held.write_tombstone's callers are unchanged.Tests
New file
test/test_subagent_owed_tombstones.py. All state is under tmp_path, the reconciler's notifiers are recorders, and no prune runs.a run whose result already reached its parent was reported again at the next boot, although space came back before the restart(reaper tick);the shutdown did not write the delivered tombstone it owed(graceful shutdown);a run announced at one boot was announced again at the next, because the boot's own tombstone write had failed(boot tombstone).The 9 controls that use the new functions fail on the base with
AttributeError, and so does the shutdown-race test, which patches the retry's exclusive create. That test stalls a reaper retry's first write, runscancel_all, clears a second run's tombstone as the drain does for a straggler, then lets the write finish. Against the retry without the lock andstop, it fails witha retry that began before shutdown wrote a tombstone after the shutdown cleared it, 3 of 3 runs. The control that an undelivered orphan is announced exactly once passes there, as it should.died;write_tombstone,mark_delivered,list_orphans, the reaper loop,_reconcile_orphans,cancel_allor the retry): 4,316 passed on the final commit.test_taskq_reconcile.py::test_every_tombstone_cause_has_a_terminal_state. That test reads the cause argument of every call named_write_tombstone, and an earlier name for the new body function collided with that name. The body is now_write_tombstone_file; the test is unchanged.Manual verification
Not run against a live gateway.
Screenshots / video
None; no UI change.
Related Issues
write_tombstoneandmark_delivered, and rewrites the reconciler, on different lines.git merge-treeof this branch with fix(subagents): end a leaked process at restart without re-announcing it #18078's head: no conflict.write_tombstonereceived, so fix(subagents): end a leaked process at restart without re-announcing it #18078'sleaked_processflag carries through if it lands first.Pattern harvest
Rule candidate: review-prompt
Pattern: a durable write whose failure is logged and swallowed inside the function that performs it, which leaves every caller believing the state was recorded.
Other durable writes in
subagent_persistence.pythat log and carry on are not examined here.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement