Skip to content

fix(subagent): keep a finished result whole when the restart loses its flag - #15670

Merged
iamwhatever merged 1 commit into
mainfrom
fix/subagent-result-complete-reconcile-15221
Oct 2, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/subagent-result-complete-reconcile-15221

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A subagent run that finished and wrote its whole answer to disk is reported to its parent as a finished result after a gateway restart, not as a fragment cut off mid-turn.

At a gateway restart, a subagent run that finished and wrote its whole result to disk minutes earlier can be reconciled as partial_result and announced to its parent as "cut off mid-turn by gateway restart ... read it as an unfinished fragment, not as the agent's answer". The label is driven entirely by whether one flag, result_complete, was persisted — and that flag lives in state.json, written in a step after the one that finalizes result.txt. A restart falling in that gap leaves a complete answer on disk with the flag never written, and the reconciler then demotes a whole finding to an opening sentence.

Why it matters

The parent agent acts on the notification's wording. A completed subagent answer relabeled a fragment is read as an incomplete opening sentence, so the parent discards or re-runs work that was already done — silently losing a finished result whenever a restart happens to land in a sub-second write window. The failure is invisible: nothing is corrupt, the bytes are all present, only the label is wrong.

Not a goal

  • Does not change the safe-direction guarantee for a genuinely interrupted run: a run that never reached its completion event still writes neither signal and is still announced as cut off.
  • Does not touch delivery, batching, or the digest-DM path for orphan notifications.
  • Does not change where result.txt or state.json are written, only adds a second, atomic completeness signal beside them.

What changed (motivation → approach → change)

Observed symptom: a finished subagent run announced to its parent as "cut off mid-turn" after a gateway restart, its complete answer read as a fragment.

Root cause: completeness and the bytes it describes live in two files written at two times. write_result_chunk streams result.txt in chunk by chunk and the completion path caps it; result_complete is recorded only later, in a separate state.json write. tombstone_recovery_action read completeness off state.json alone, so a restart between the cap and the flag write classified a whole answer partial_result.

The specific change makes the completeness signal land atomically with the file it describes. The completion path drops a durable marker (result.complete) in the same step that caps result.txt, before the separate state.json write, via temp-file + rename so a torn write is never read as half-present. The classifier treats the marker or the state flag as proof the answer is whole:

  • subagent_persistence.py: mark_result_complete(agent_id) writes the atomic marker (skipped for transient in-memory runs, matching write_result_chunk); result_marked_complete(agent_id) reads it.
  • subagent_manager/run.py: the completion path writes the marker right after cap_result_file and before the _write_state_off_loop flag write, under the same _complete_event is not None and _stop.is_success condition. The marker write goes off the event loop via asyncio.to_thread (the filesystem work must not block the gateway's single loop on a network-backed data home), and is shielded and drained on cancellation like the sibling _write_state_off_loop: a cancel during the rename holds until the worker settles, so the marker lands before any terminal tombstone a cancel arm writes and the two never disagree on disk.
  • subagent_manager/monitoring.py: tombstone_recovery_action returns partial_result only when neither state["result_complete"] nor result_marked_complete(agent_id) is set.
  • subagent.py: the new helper is threaded onto the two component-binding surfaces so run.py resolves it at runtime.

Backwards compatibility

Compatible: adds a new signal (result.complete) and widens the completion test to an OR. A run with the old state.json flag and no marker is still classified result_available; a run with neither is still partial_result. No existing field is rejected, renamed, or removed, and the new marker file is additive beside the agent directory's existing files.

Tests

Added to test/test_runtime_reconcile.py, each registered in the module's mutation-target docstring:

  • test_the_durable_marker_rescues_a_finished_result_whose_flag_never_wrote — a whole result with the marker present but no state.json flag classifies result_available.
  • test_a_result_without_either_signal_still_under_claims_as_a_fragment — the complement: no flag and no marker stays partial_result, preserving the safe direction.
  • test_the_reconcile_calls_a_marker_only_finished_run_orphaned_not_cut_off — end to end over the real orphan reconcile: the notification reads "orphaned by gateway restart" (not "cut off mid-turn") and the tombstone's recovery_action is result_available.

Added to test/test_subagent_persistence.py (TestResultCompleteMarker), covering every branch of the new helpers: the write→read round trip, a transient (incognito/temporary) run writing no marker, a failed atomic rename swallowed with its temp file cleaned up, and the reader answering False for an unusable agent id.

Added to test/test_subagent_stop_reason_consistency.py, driving the real SubagentManager.spawn → _run → _run_inner_impl path (only the ACP session faked): a clean end_turn completion drops the durable marker on disk, and a stream that dies with no complete event drops none — pinning the production invariant that the marker is created by successful completion, not only by a direct helper call.

Each was confirmed to fail without the fix (reverting the classifier's marker clause reds the first and third) and pass with it. Full test/test_runtime_reconcile.py (127 tests) and the subagent persistence / component-globals / continuable / result-view suites pass.

Manual verification

N/A — unit coverage sufficient: the defect is a disk-state classification reproduced directly by seeding result.txt + the marker with no state.json flag, both at the classifier and end to end over the reconcile.

Related Issues

Fixes #15221

Pattern harvest

Rule candidate: review-prompt
Pattern: a completeness/validity flag persisted in a different file (or a later write) than the bytes it describes can diverge across a crash — the signal must land atomically with, or no later than, the data it certifies.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 1, 2026 01:59
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5) — 🟡 CONCERNS

Design-level review of 142a8f4a4b231f1edf6039fa1fb6b278146e4377 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The owning spec still pins the old contract — subagent.md:2182 says reconciliation classifies "on that flag alone", which this diff falsifies.

Watch

  • The repo makes docs/system-specs/modules/subagent.md the source of truth loaded before any subagent change, and it still reads "reconciliation classifies the file on that flag alone — result_available with it, partial_result without" (lines 2178–2184). The diff adds a second, equal completeness signal (result.complete, OR-ed in tombstone_recovery_action) with no spec change, so the next contributor reasoning from the spec will treat the marker as removable or redundant and can reintroduce the exact crash window this PR closes; AGENTS.md requires the spec update in the same commit, and the checklist's "Documentation updated" claim has no backing hunk.
    Clears when: docs/system-specs/modules/subagent.md's classification paragraph is updated in this PR to name both signals and the crash window the marker closes.

[DESIGN-REVIEWED] 142a8f4

Description read

[DESCRIPTION-READ] cfe90cc9cd2d3899d834e189637a72180b4e68a859c9a711ba72d50e0ea8c1e9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review — ✅ no blocking findings

GPT 5.6 completed its review of 142a8f4a4b231f1edf6039fa1fb6b278146e4377 and found no blocking issues.

This comment is updated in place on each push.

Review details

FINDING -- src/kiro_crew/subagent_manager/run.py:2660 -- A restart after cap_result_file but before the marker rename still yields partial_result, so “cannot lag ... across a crash” contradicts the code -> Fix: describe this as narrowing the crash window.
[GPT-REVIEWED] 142a8f4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 142a8f4a4b231f1edf6039fa1fb6b278146e4377: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5) — 🟡 CONCERNS

Premise-level review of 142a8f4a4b231f1edf6039fa1fb6b278146e4377 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The bodyless marker ships temp-file+rename torn-write machinery a zero-byte file cannot need — touch() creates the dirent atomically; there is nothing to tear.

Not justified as shipped

  • Item 2 — oversized: mark_result_complete justifies its mkstemp/Path.replace body with "Atomic (temp file + rename) so a torn write is never read as a half-present marker", yet the same diff says "Its presence is the whole signal — it carries no body". An empty file's creation is atomic at the directory entry, so the torn-write premise has no content to tear; the machinery buys only an extra failure branch, its cleanup, and test_a_failed_rename_is_swallowed_and_leaves_no_temp_file — and a crash between mkstemp and replace still leaves the .tmp litter the in-process cleanup cannot reach.
    Subtraction: shrink the body to marker.touch(), deleting the temp-file branch and its cleanup test.
    Clears when: the body is shrunk to touch(), or a filesystem the data home supports is named where bare O_CREAT creation is not atomic but rename is.

What this change ships

Inventory (6 items) — 5 justified

Intent: FIX — a finished subagent result that survived a gateway restart is reported to its parent as a whole answer, not a mid-turn fragment (linked issue #15221).

  1. A finished run caught in the flag-write gap is announced "orphaned by gateway restart", not "cut off mid-turn" — justified
  2. Every completed run leaves a result.complete marker beside result.txt — oversized: torn-write machinery for a bodyless file
  3. The orphan classifier accepts marker OR old state.json flag (old on-disk runs still classify) — justified
  4. Two new persistence helpers, one counted consumer each (result_marked_complete → monitoring.py:140; mark_result_complete → run.py:2672) — justified
  5. The comment pinning the gap as "the safe direction" is rewritten; the reversal is declared, its rationale kept for genuine fragments, and the genuinely-interrupted case still under-claims — justified
  6. Completion performs one extra shielded off-loop disk write before the state write — justified

[FIRST-PRINCIPLES-REVIEWED] 142a8f4

Description read

[DESCRIPTION-READ] 8b36d5c85b522aedf756e818c007626df046acc03d4cf2e1b57a40dfac91b643

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review — ✅ no blocking findings

Reviewed 142a8f4a4b231f1edf6039fa1fb6b278146e4377 — this comment is updated in place on each push.

Review details

Nothing blocks; the new completion-marker drain is the one off-loop writer in this module that is neither bounded nor repeated-cancel-safe.

FINDING — src/kiro_crew/subagent_manager/run.py:2677 — the drain await _marker_writer awaits the worker task directly, so it becomes the run task's _fut_waiter and a second cancel (wait_for timeout followed by cancel_all, the sequence the sibling at run.py:180 and run.py:276 explicitly codes for — "asyncio.wait never cancels its members") cancels _marker_writer itself, swallows it via except BaseException: pass and re-raises before line 2685, so _run_impl's terminal arm runs _write_tombstone → tombstone_recovery_action with the result_complete flag never written and the detached rename still in flight → the finished answer is tombstoned partial_result, which the pre-PR path guaranteed against because that cancel landed on _write_state_off_loop's bounded, repeat-cancel-tolerant drain; the same await is also deadline-free, unlike _STATE_DRAIN_TIMEOUT whose own comment claims to bound "every off-loop writer", so a wedged data home holds cancel_all's untimed gather (cancellation.py:1136) open indefinitely → Fix: drain with the sibling shape — while not _marker_writer.done(): remaining = deadline - time.monotonic(); ...; try: await asyncio.wait({_marker_writer}, timeout=remaining) except asyncio.CancelledError: pass under _STATE_DRAIN_TIMEOUT (already imported at run.py:28).

FINDING — src/kiro_crew/subagent_manager/monitoring.py:140 — the classifier now accepts two proofs of completeness, but every statement of that contract still says one: the calling function's own docstring at monitoring.py:424 ("only a run that recorded result_complete has a whole answer on disk, and anything else is a fragment"), the owning spec at docs/system-specs/modules/subagent.md:2182 ("reconciliation classifies the file on that flag alone"), and the delivered-notice contract at docs/system-specs/common/injected-messages.md:180 ("only when the run recorded result_complete") are all false for an orphan rescued by the marker → Fix: in the same commit, update the two spec paragraphs and _reconcile_orphans_impl's docstring to state that either the state.json flag or the durable result.complete marker proves the answer is whole (requires editing files this PR does not touch, so advisory).

[OPUS-REVIEWED] 142a8f4

Verdict parsed from the review's SHA-scoped output markers for commit 142a8f4a4b231f1edf6039fa1fb6b278146e4377.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 142a8f4a4b231f1edf6039fa1fb6b278146e4377: <one-sentence reason>

@chenmingwei23
chenmingwei23 force-pushed the fix/subagent-result-complete-reconcile-15221 branch from 1e7ab32 to 970624b Compare October 1, 2026 02:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/subagent-result-complete-reconcile-15221 branch from 970624b to 7e4ee10 Compare October 1, 2026 02:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 1, 2026
…s flag

A subagent run finalizes result.txt in one step and records result_complete
in state.json in a later step. A gateway restart landing between the two left
a whole answer on disk with the flag never written, so the orphan reconciler
read the completed answer as a fragment and told the parent it was cut off
mid-turn.

Drop a durable completion marker (result.complete) in the same completion step
that caps result.txt, before the separate state.json write. The reconciler's
classifier treats the marker or the state flag as proof the answer is whole, so
the crash window between the two writes no longer under-claims a finished run.
A run that never completed writes neither signal, which stays the safe
direction. The marker is atomic (temp file + rename) and transient runs that
never touch disk write no marker.
@chenmingwei23
chenmingwei23 force-pushed the fix/subagent-result-complete-reconcile-15221 branch from 7e4ee10 to 142a8f4 Compare October 2, 2026 07:59
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Follow-up opened as #16185, with @chenmingwei23 credited as Co-authored-by. This branch has maintainerCanModify off, so the change could not be pushed here.

#16185 fixes #15221 with a different design. This PR adds a result.complete marker beside result.txt. #16185 instead has a finished run claim its ending once, synchronously, then rewrite result.txt whole from memory (capped, fsynced) before it records result_complete. A restart can then never find the flag describing bytes the run did not write. It also covers the live-run side of the same window: a Stop, cancel or deadline landing during the ending. Every other ending caps result.txt in place. The restart reconcile, its notice and the task queue's boot probe all read one completeness fact.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed: result.complete is written atomically right after cap_result_file and only on a successful complete event, so a genuinely interrupted run is still reported as a fragment; continuations mint fresh ids so an old marker cannot vouch for later output. No deletion beyond the marker's own temp file, no kills. Nits: mark_result_complete's docstring says it never overwrites but Path.replace does; the cancel branch in run.py swallows non-OSError from the marker writer silently, a debug log would help.

@iamwhatever
iamwhatever enabled auto-merge (squash) October 2, 2026 16:44
@iamwhatever
iamwhatever merged commit 218eedf into main Oct 2, 2026
97 checks passed
@iamwhatever
iamwhatever deleted the fix/subagent-result-complete-reconcile-15221 branch October 2, 2026 16:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants