Skip to content

fix(subagent): claim a whole answer's ending and keep its result whole - #16185

Merged
chenmingwei23 merged 2 commits into
mainfrom
followup/15670-result-complete
Oct 4, 2026
Merged

chenmingwei23 merged 2 commits into
mainfrom
followup/15670-result-complete

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A subagent run that finished with its whole answer is reported to its parent as a finished result, whatever lands around its ending: a gateway restart, a Stop, a cancel or the deadline.

After a gateway restart, a subagent that finished minutes earlier can be announced to its parent as "cut off mid-turn by gateway restart ... read it as an unfinished fragment". The only completeness signal is result_complete in state.json, written in a later step than the one that finishes result.txt, so a restart in between demotes a whole answer to a fragment (#15221). The same window has a live-run variant: a Stop, a cancel or the deadline landing while a finished run records its ending can overwrite that ending or leave the run with none recorded. Separately, result.txt is appended per chunk across attempts, so a respawned run can glue two attempts' text together.

Why it matters

The parent agent acts on the notice's wording. A completed answer labelled a fragment gets discarded or re-run, so finished work is silently lost whenever a restart or a stop happens to land in a short write window. Nothing looks corrupt: the bytes are on disk and only the label is wrong, so nobody notices.

Not a goal

  • No change for a run that really was interrupted: a run that never reached a successful complete event is still announced as cut off.
  • No change to delivery, batching or the digest-DM path for orphan notices.
  • No change to where result.txt and state.json live, or to the result byte budget.
  • No new retry or recovery behaviour for failed runs.

What changed (motivation → approach → change)

Symptom: a finished run reported as a fragment after a restart, and a finished run whose ending a late stop could overwrite. Root cause: "finished" was never one fact. It was a flag written after the bytes it describes, and each stop path decided for itself whether the run had already ended.

Approach: make the ending ONE synchronous claim, and make the completeness fact describe a file written from memory.

  • One ending claim. A successful complete event claims the completed ending synchronously (SubagentInfo._ending_claimed), unless a stop got there first (done, a reap, a user stop, or an error a failed child or an expired wait stamped). From the claim on, Stop, a parent end, the reaper, fail_parent and wait expiry treat the run as done. _run's timeout, cancel and exception arms leave a recorded ending alone. settle_before_delete answers pending while a claimed run is still writing its result.
  • The whole answer is rewritten from memory (subagent_persistence.py). The claimed run rewrites result.txt whole from the streamed text, capped (cap_result_bytes, UTF-8 safe), fsynced with its folder, and only then records result_complete=True. Both happen in one drained worker (write_finished_result), which is bounded even when nothing cancels it. An empty answer leaves no result.txt.
  • Every other ending is capped in place. _run's finally caps the streamed file in place (context_management.py's cap_result_file, reading only the two ends) and records result_complete=False. That covers the endings that never reach the run's tail, and it never raises, so the report still spawns. A whole rewrite that fails falls back to the same cap.
  • Attempts never glue. An attempt's first chunk starts result.txt (write_result_chunk(fresh=True), a temp file and rename, LF on every platform) and the live partial afresh, on separate signals, so a failed write never costs the partial or the earlier attempt's file.
  • The usage row no longer holds an ending. Its inputs are read synchronously and the append runs on a task cancel_all drains.
  • One completeness fact for every reader. The restart reconcile, its notice (one builder: ✅ completed, ⚠️ partial, ❌ lost) and the task queue's boot probe all read result_is_whole(state), so fix(subagent): keep a finished result whole when the restart loses its flag #15670's result.complete marker is retired. A run that finished before the restart is announced ✅ finished before gateway restart. The live settle and the boot probe share one outcome-to-task-state table (outcome_task_state).
  • Every failure cause settles failed on boot. The boot probe's cause table (tombstone_terminal_state) maps reaped, startup_timeout and start_queue_saturated (a cause main added in fix(acp): give interactive chat starts priority over subagent starts #16020) to failed, as the live settle does, so every cause but gateway_restart proves an ending. test_taskq_reconcile.py::test_every_tombstone_cause_has_a_terminal_state pins that no written cause is unmapped.
  • subagent.py binds the new helpers. update_state reads via read_bytes_with_retry, and _atomic_write delegates to atomic_write(fsync=True, mode=0o600).
Ending Before After
Whole answer, restart before the flag write 🟥 announced as a fragment 🟩 rewritten whole, then flagged; announced ✅ completed
Whole answer, then Stop / cancel / deadline lands 🟥 the late stop could overwrite the ending 🟩 claim taken first; the stop is a no-op
Turn limit, raised error, deadline before the tail 🟥 result.txt left uncapped 🟩 capped in place from _run's finally
Respawn after a cancel 🟥 attempts glued in result.txt 🟩 first chunk starts the file and the live partial afresh
Textless whole answer 🟦 announced as lost 🟩 announced ✅ finished without text

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Follow-up to #15670 (@chenmingwei23, credited as Co-authored-by), merged with a result.complete marker. Here the flag lands right after the whole rewrite, in one worker job, so the marker covers nothing more.

Specs updated in the same commit: docs/system-specs/modules/subagent.md (lifecycle, ending claim, terminal-state contract, folder layout, write-site census, reconcile), docs/system-specs/modules/taskq.md (boot probe), docs/system-specs/common/injected-messages.md (orphan notice shapes), docs/request-for-change/rfc-resumable-subagent-sessions.md (cap helper names).

Backwards compatibility

Compatible: result_complete keeps its meaning and its key. A state written by an older build without the flag still reads as not whole, which is the same safe direction as today. Nothing that main accepts is rejected or renamed, except the two marker helpers #15670 added (mark_result_complete, result_marked_complete). One narrow case changes: a run that an earlier build on main finished, dropped the marker for, and lost the flag write for, before upgrading to this one, now reads as a partial (the pre-#15670 behaviour) instead of whole. A stale result.complete file is never read and goes away with its folder. The orphan notice gains a ✅ shape, and the card renders it from the glyph.

Reader: src/kiro_crew/subagent_manager/monitoring.py:tombstone_recovery_action -- subagent -- reads result_is_whole(state), the flag the claimed ending writes after the whole rewrite; test_runtime_reconcile.py::test_a_flagged_result_is_the_agents_answer
Reader: src/kiro_crew/subagent_manager/monitoring.py:_notify_orphan_impl -- subagent -- same fact, end to end over the reconcile; test_runtime_reconcile.py::test_the_reconcile_announces_a_finished_run_as_finished
Reader: src/kiro_crew/subagent_manager/admission/taskq_bridge.py:outcome_task_state -- subagent -- the boot probe reads the tombstone outcome then result_is_whole; test_taskq_reconcile.py
Reader: website/src/pages/chat/subagentCompletion.ts:parseSubagentCompletion -- chat -- the outcome is keyed on the glyph, so the reworded note parses as before; SubagentCompletionCard.test.tsx

Tests

  • test_subagent_stop_reason_consistency.py: the claimed ending is written after post-processing, whole and capped from memory. Every other ending caps, including endings that never reach the tail. A stamped error counts as a stop that got there first. A hung claimed write is bounded. Whatever lands after the claim, the run completes once. Every other stop path treats a claimed run as done. The usage row never holds an ending. A respawn never glues, and a cut-off respawn delivers the partial it left on disk. A failed append still delivers the whole partial. A textless whole answer leaves no file. The file is LF. A transient run touches no disk.
  • test_subagent_persistence.py: a fresh chunk the disk refuses keeps the earlier partial, the orphan notice builder (4 cases), the restart tombstone outcome, write_finished_result (including a refused rename that still caps), and update_state Windows contention on read and rename, plus the 0o600 mode.
  • test_runtime_reconcile.py: fix(subagent): keep a finished result whole when the restart loses its flag #15670's three marker tests now pin the flag rule (flagged reads whole, unflagged under-claims, the reconcile announces a finished run as finished).
  • test_session_start_gate.py: reads _running_count once the run task returned, since _run's finally now awaits the off-loop cap before releasing the slot.
  • test_taskq_reconcile.py: the boot probe reads the tombstone outcome, then the cause, then result_is_whole. A test pins that every outcome maps.
  • test_context_management.py: cap_result_bytes and the in-place cap_result_file (UTF-8 seams, shrink only).
  • Fixture updates: test_session, test_subagent_dependency_mark, test_subagent_panel_durable_replay, test_turn_duration_subagent, and others for the new record fields.
  • Frontend: SubagentCompletionCard.test.tsx for the ✅ orphan shape.

Manual verification

N/A. Unit coverage is sufficient: the restart reconcile, stop paths and file writes run against real SubagentManager instances and a temp data home. A live gateway restart cannot be scripted deterministically inside the write window this closes.

Screenshots / video

The three gateway-restart cards, seeded with the exact text _notify_orphan_impl composes and rendered by the built SPA (a scratch copy of website/scripts/capture-subagent-completion-card.mjs): a whole answer (✅ finished before gateway restart, Completed), a textless whole answer, and a run the restart cut off (⚠️, Interrupted). Collapsed and expanded, light and dark.

Restart cards, light, collapsed

Restart cards, light, expanded

Restart cards, dark, collapsed

Restart cards, dark, expanded

Related Issues

Fixes #15221
Follow-up to #15670

Pattern harvest

Rule candidate: review-prompt
Pattern: a completeness flag written in a later step than the bytes it certifies, or a terminal state that each stop path decides for itself instead of one synchronous first-arrival claim.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@bolichen97
bolichen97 requested a review from a team October 2, 2026 16:05
@bolichen97
bolichen97 requested a review from a team as a code owner October 2, 2026 16:05
@bolichen97
bolichen97 requested a review from iamwhatever October 2, 2026 16:05
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ human override accepted

@bolichen97 overrode this lane for 95e5a5dab124dcb3715ead723ec872c69c489c54 via /ai-review override (this commit only).

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 95e5a5dab124dcb3715ead723ec872c69c489c54 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The completed-ending claim is a second "ended" token next to done, so each stop path has to check done or _ending_claimed itself, and a missed check brings back the overwrite this PR fixes.

Watch

  • The claim has no single owner. The description says stop paths "treat the run as done", but the code does that by repeating the check at about ten call sites: cancellation.py:674,1231, monitoring.py:985, terminal.py:612, waits.py:689,829 and run.py:451,817,889,1023. Some test info.done or info._ending_claimed. Some test info._reap_started or info._ending_claimed. The new fail_parent guard tests not parent.done and not parent._ending_claimed. A future Stop, reap or expiry path that checks only info.done would still stamp an error on, or respawn, a run that already holds a whole answer, during the async write window. That is the A finished, result-bearing subagent run is relabeled "cut off mid-turn by gateway restart" when only the result_complete flag failed to persist before the crash #15221-class loss, and as the PR says, nothing would look corrupt.
    Clears when: one SubagentInfo predicate (e.g. ending_recorded) covering done/claim/reap/user_stop/error replaces the repeated checks, or a test enumerates every stop entry point and drives each against a claimed run.

  • Boot probe behaviour changes for reaped and startup_timeout. Both now map to FAILED where they used to be unmapped, and an unmapped row was recovered on boot. This sits outside the frozen Goal, which covers only whole answers. The description justifies it as matching the live settle, so the open question is whether the old unmapped state was deliberate or just a gap.
    Clears when: the git history or the author confirms that leaving these causes unmapped was an oversight and not a recover-on-boot decision.

[DESIGN-REVIEWED] 95e5a5d

Description read

[DESCRIPTION-READ] 819ebbb240469aef665f0dbd4957af7a363d563292e312c1ff1bd8391f5526a3

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 95e5a5dab124dcb3715ead723ec872c69c489c54 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

One thing to check: the reader found a "Completed" card that says "without writing any text" odd. Confirm parents and users treat a textless finish as a success.

[UX-REVIEWED] 95e5a5d

Description read

[DESCRIPTION-READ] 60e20bbede28dbf90a306f26b7e51dbe330f6904565b48867cf9c291cc17759d

That sha256 names everything this verdict read: the PR description, plus the 7 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 95e5a5dab124dcb3715ead723ec872c69c489c54 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 95e5a5d

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 95e5a5dab124dcb3715ead723ec872c69c489c54: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@github-actions

github-actions Bot commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 95e5a5dab124dcb3715ead723ec872c69c489c54 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 95e5a5d

Verdict parsed from the review's SHA-scoped output markers for commit 95e5a5dab124dcb3715ead723ec872c69c489c54.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 95e5a5dab124dcb3715ead723ec872c69c489c54: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the followup/15670-result-complete branch from fb94df3 to 96edcc9 Compare October 2, 2026 20:24
@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

mechanism: write_result_chunk(fresh=True) stages through atomic_write (temp file + rename), refusing to recreate a removed run folder

  • Fresh-write failure destroys the previous attempt span=38c76f4dfc8c — fixed in 96edcc9

Outcome: fixed. An attempt's first chunk no longer opens result.txt with O_TRUNC.
It is written to a temp file and renamed over result.txt (subagent_persistence._replace_in_run_folder -> atomic_write), so the earlier attempt's partial is replaced only once the new text is on disk.
A write the disk refuses (full disk, quota) leaves the earlier partial in place and returns False, so the next chunk retries the fresh write.
Test: test_subagent_persistence.py::TestWriteResultChunk::test_a_fresh_chunk_that_cannot_be_written_keeps_the_earlier_partial (a failing _write_all keeps the old bytes and leaves no .tmp).

@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • "✅ orphaned by gateway restart" is mixed-signal copy span=f49d72ecb570 — fixed in 96edcc9

Outcome: fixed. The ✅ note now reads "finished before gateway restart" (monitoring._notify_orphan_impl), so the green Completed card no longer says "orphaned".
The card's outcome is keyed on the glyph, so the parser needs no change; spec docs/system-specs/common/injected-messages.md and the backend and frontend fixtures carry the new wording.
Evidence gap closed: the PR description now attaches the three restart cards (whole, textless, cut off), collapsed and expanded, light and dark, rendered by the built SPA.

@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Reword the ✅ note to name the outcome span=4da0866c3464 — fixed in 96edcc9

Outcome: fixed as suggested: the note is "finished before gateway restart" (monitoring._notify_orphan_impl).
Pinned by test_subagent_persistence.py::test_the_notice_follows_the_run_record and test_runtime_reconcile.py::test_the_reconcile_announces_a_finished_run_as_finished; the frontend fixture in SubagentCompletionCard.test.tsx uses the same text.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 2, 2026
@bolichen97
bolichen97 force-pushed the followup/15670-result-complete branch from 96edcc9 to d644f78 Compare October 2, 2026 23:10
@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author

mechanism: a whole answer a reap got to first is left unrecorded by the tail, and _run writes the reap's ending (the reap-echo arm's record) and leaves the report to the reap

  • A stop-rejected claim still falls through as success span=src/kiro_crew/subagent_manager/run.py:2897 — fixed in d644f78

Outcome: fixed. Under a reap in flight the tail no longer sets done or counts a success. _run sees the reap and an unrecorded ending, then writes the reap's own record with _record_reap_ending (the helper the reap-echo arm now shares), so a deadline reap is a failure naming the deadline and a Stop stays neutral. The report goes to the reap, after its kill has decided.
A user stop with no reap yet takes the neutral user-stop branch ahead of the success branch. A run another path already ended counts no success.
Test: test_subagent_stop_reason_consistency.py::test_a_deadline_reap_that_got_there_first_owns_the_ending fails without the fix (outcome 'completed', record_success called). test_a_stop_that_got_there_first_owns_the_ending_and_no_flag_is_written now also asserts record_success is never called. Spec: docs/system-specs/modules/subagent.md, Ending claim.

@bolichen97

bolichen97 commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor Author
  • "Use the read tool to retrieve it." reads as addressed to someone else span=src/kiro_crew/subagent_manager/monitoring.py:608 — not changed (advisory CONCERNS)

Rationale: this sentence is part of an injected message, and the parent agent is the reader it is written for. That agent does have a read tool and acts on the instruction. The live completion envelope (terminal.py, "Use the read tool to retrieve it if needed.") uses the same wording and the same card already renders it. Rewording only the restart variant would split two shapes of one envelope that docs/system-specs/common/injected-messages.md documents together. If the human-facing card should say something different, the card should change its rendering for both shapes, and that belongs in its own change.

@bolichen97
bolichen97 force-pushed the followup/15670-result-complete branch from d644f78 to 4b7d742 Compare October 3, 2026 00:51
@bolichen97

Copy link
Copy Markdown
Contributor Author

mechanism: result_complete records one fact, "result.txt is whole", and a refused write leaves it False; the run's outcome is the tombstone's, not this flag's

  • Refused completeness write still publishes success span=src/kiro_crew/subagent_manager/run.py:2905 — not changed (rebutted; override requested)

Rationale: the state this write owns is "result.txt holds the whole answer", and a refused write leaves that state explicitly False, which is a terminal state, never a retryable one. Nothing in memory claims the opposite. What the tail publishes is the run's outcome: a successful complete event, with the whole answer held in info.result and delivered from memory in the completion. That outcome is owned by the report and its tombstone, not by result_complete.
After a restart before delivery, the notice reads the flag and says "cut off mid-turn". That direction is safe: the notice never vouches for bytes that may not be on disk, and the fallback cap in write_finished_result keeps whatever streamed. The other answers are worse. Failing the run, or holding the completion until a write succeeds, would throw away or strand a whole answer because of a disk error, and nothing else can end a claimed run. A "retryable" state has no retrier, because the claim makes the run done to every stop.
Pinned as designed by test_subagent_stop_reason_consistency.py::test_a_failed_result_write_records_no_flag_and_the_run_completes (file write and folder fsync both fail: outcome completed, result_complete False, WARNING logged). Spec: docs/system-specs/modules/subagent.md, Ending claim ("the flag lands late or not at all, the safe direction").
Also fixed in 4b7d742 (from a post-push review): two behaviours the commit claimed had no test. A user stop that lands before its reap starts records the neutral stop (test_a_user_stop_no_reap_has_reached_yet_records_the_neutral_stop). A run another path already ended counts no success (test_a_run_another_path_already_ended_counts_no_success). The reap-in-flight record now says "the runtime was torn down before the run finished" when the stream just stopped with no complete event (test_a_reap_in_flight_names_whether_the_answer_had_finished). Each test fails under its mutation: branch reorder, bare _count_success(), and an unconditional "finished its answer".

@bolichen97
bolichen97 force-pushed the followup/15670-result-complete branch from 4b7d742 to 6360ec7 Compare October 3, 2026 02:22
@bolichen97

Copy link
Copy Markdown
Contributor Author

mechanism: until an attempt's first result.txt write succeeds, each chunk starts the file with the attempt's accumulated text, so a refused first write leaves no head-missing fragment

  • A recovered fresh write drops earlier chunks span=src/kiro_crew/subagent_manager/run.py:2297 — fixed in 6360ec7

Outcome: fixed as suggested. While _result_file_started is False the chunk write is write_result_chunk(info.id, result_text, fresh=True) (the whole attempt so far); after the first success it appends event.text. Pinned by test_subagent_stop_reason_consistency.py::test_a_refused_first_write_leaves_no_hole_at_the_head_of_result_txt (first write refused, then recovers: result.txt is "alpha beta gamma "); against 4b7d742's run.py it fails with 'beta gamma ' == 'alpha beta gamma '. Spec: docs/system-specs/modules/subagent.md, the result.txt restart paragraph.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • "Use the read tool to retrieve it." addresses the parent agent, not the person reading the card span=src/kiro_crew/subagent_manager/monitoring.py:608 — not changed (advisory CONCERNS)

Rationale: same finding as at 96edcc9, same answer (#16185 (comment)). The sentence is part of an injected message whose reader is the parent agent, which has a read tool. The live completion envelope in terminal.py already uses this wording, and docs/system-specs/common/injected-messages.md documents both shapes together. Changing how the card renders that line for a human is a card change for both shapes, so it belongs in its own PR.

@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 4, 2026
@bolichen97

bolichen97 commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor Author
  • Item 5: boot settles reaped, startup_timeout and start_queue_saturated tombstones failed (rides along, inherited) span=src/kiro_crew/subagent_manager/admission/types.py:tombstone_terminal_state — not changed (advisory CONCERNS)
    mechanism: the boot probe reads a tombstone's recorded outcome first; the cause table only answers a tombstone with no outcome

Rationale: answered at bf3a5e9 (#16185 (comment)) and at 5937a46 (#16185 (comment)). This head does not touch it. Every tombstone written today records outcome (subagent_persistence.write_tombstone). So a reaped, startup-timeout or saturated run already settles failed through outcome_task_state before the cause table is read, and these keys stop no requeue that the outcome read does not already stop. The Goal needs the outcome-first read: a whole answer that a later stop's tombstone records as outcome: completed must settle done on boot. The keys reach only a tombstone with no outcome. Dropping them makes one cause settle failed or be re-dispatched depending on when its tombstone was written. test_every_tombstone_cause_has_a_terminal_state caught #16020's new cause on this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • On Windows, a whole rewrite whose rename keeps failing falls back to the in-place cap with result_complete=False, so the answer is announced as partial span=src/kiro_crew/subagent_persistence.py:write_finished_result — not changed (advisory note on a PASS)

Rationale: that is the intended safe direction, and the spec states it ("A failed write records result_complete=False and falls back to capping the streamed file in place"). The flag never vouches for bytes that did not reach the disk. The rename is atomic_write → replace_with_retry, which first retries the Windows sharing-violation window, so only a reader holding result.txt past that budget gets here. The cost is an under-claim: the parent reads the streamed text as a partial, never a torn or glued file. Pinned by test_a_failed_result_write_records_no_flag_and_the_run_completes. The 5937a46 note on the same rename is answered at #16185 (comment).

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The textless-finish copy leaves the outcome unclear span=src/kiro_crew/subagent_manager/monitoring.py — not changed (advisory CONCERNS, Watch)

Rationale: this is the sixth request to reword this line, answered at 2a62ba0 (#16185 (comment)), bf3a5e9 (#16185 (comment)), c273d8a and 96edcc9. The suggested "It made changes but wrote no reply" states something the reconcile does not know. It knows only that the run recorded a whole answer and that the answer had no text. It cannot know whether the run changed anything, so that sentence would be a guess shown as fact. The ✅ glyph and the green "Completed" chip state the outcome, and the sentence says why there is no result to read. The review rates it low frequency.

bolichen97 and others added 2 commits October 4, 2026 08:12
A finished subagent result could read as a fragment after a gateway
restart, and a stop landing while a finished run recorded its ending
could leave the run returning with no ending recorded.

A whole answer now claims its completed ending once, synchronously,
unless a stop got there first (an error a failed child or an expired
wait stamped counts as one), and every stop path treats the claim as
done. The claimed run rewrites result.txt whole from memory, capped and
fsynced, before recording result_complete; an empty answer leaves no
file. That write is bounded, since nothing else can end it. Every other
ending, including those that never reach the run's tail, caps the
streamed file in place from the run's finally, and a whole rewrite that
fails falls back to the same cap. An attempt's first chunk starts
result.txt and the live partial afresh, so a respawn never glues two
attempts; that first write is a temp file and rename, so a failed write
keeps the earlier partial. Until a write succeeds, and again after a
refused append, each chunk starts the file with everything the attempt
streamed, so a refused write leaves no hole for a restart to find; that
start write grows with the answer, so it runs off the loop in a worker
drained like a state write, after the live partial takes the chunk. The
usage row runs on a task cancel_all drains, so no ending waits for it.
The restart reconcile, its notice and the task queue's boot probe read
one completeness fact, so the result.complete marker is retired: the
flag is now written right after the whole rewrite it describes. A run
that finished before a restart is announced as "finished before gateway
restart".

Co-authored-by: Raymond Chen <45910466+chenmingwei23@users.noreply.github.com>
A successful complete event arriving while a reap tears the run down
was refused the ending claim, but the tail still set done and counted a
success. The reap's record guard then skipped, so a deadline-stopped run
was reported completed and cleared its session's failure count.

The tail now records nothing under a reap in flight; _run writes the
reap's ending with the reap-echo arm's own record and leaves the report
to the reap. That record says the answer finished only when a complete
event ended it; a stream that just stopped names the teardown. A user
stop no reap has reached yet records the neutral stop, and a run
another path already ended counts no success. Each of the three is
pinned by its own test.

A start_queue_saturated reap, a tombstone cause main added since,
settles failed on boot like startup_timeout, so the boot probe keeps
its rule that every cause but gateway_restart proves an ending.

_run's finally caps an unclaimed ending after the run is done, so
_conversation_busy does not count it, and a release landing while the
cap's worker runs wrote keep=False for that worker's whole-file
update_state to roll back. The cap now holds the conversation in
_abandoned_state_writers until its worker lands.

The hold counts a run's writers, not the run: a state write left
detached past its drain and the final cap can both be live for one id,
and the earlier one landing first released the conversation while the
cap was still writing. The hold now lasts until the run's last writer
lands, and a writer registered twice is held once.
@bolichen97
bolichen97 force-pushed the followup/15670-result-complete branch from d0252de to 95e5a5d Compare October 4, 2026 08:21
@bolichen97

Copy link
Copy Markdown
Contributor Author

Pushed 95e5a5d: rebased onto origin/main 3f960a1. It is still two commits, with authorship and the Co-authored-by trailer kept. The rebase had no conflicts.

  • Reds at d0252de were inherited from main. test_chat_handlers_composition_contract.py (the re pin) and test_review_repair_routing_skill.py (prepare-pr at 68077 > 68000 bytes) failed on Linux 3.12 shards 1/3 and Windows shards 1/3. This PR touches neither file. Main fixed them in test: drop re from the chat_handlers base-name pin #16695 and fix(skills): bring prepare-pr SKILL.md back under its 68000-byte budget #16717, and both now pass on the rebased tree. Coverage Gate was red only because those jobs failed.
  • The dedup test now fails under its mutation. test_the_hold_lasts_until_the_last_writer_of_a_run_lands claimed that a writer registered twice is held once, but removing the if writer in held: return early return changed nothing it observed. Now the cap fails, and the test asserts that its failure is logged exactly once. With the early return removed, the done-callback runs twice, the line is logged twice, and the test fails. I checked this locally.
  • Two comments outside the diff are now accurate for the per-writer hold. continuation.py's _conversation_busy docstring and SubagentManager.pending_work_count's comment both still said that one done-callback discards the run id. They now say that each writer removes itself and the id goes with the run's last writer. pending_work_count still counts runs, so its behavior is unchanged. The test_subagent_coverage.py fixture now uses the dict-of-sets shape instead of a bare set.

Local checks: black, subprocess-encoding, isort, flake8, mypy, comment-history and harness-parity gates all pass. The PR's 15 test files plus the continuable, composition-contract, skill-budget and install-aware files ran with 1923 passed.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rationale: the marker never closed this window on main either. It only moved it. Main appends result.txt per chunk, then runs cap_result_file on the loop, then hops to a thread for mark_result_complete (origin/main subagent_manager/run.py ~L3030, subagent_persistence.py L1126). That is a separate write in a later step than the bytes it certifies, so a kill after the last chunk and before the marker reads as partial on main too. Here the gap is narrower. The rename, fsync_dir and update_state run back to back inside one function, write_finished_result, with no event-loop step between them. A gateway restart, which is #15221's scenario, does not land in the gap. _drain_state_writer_impl shields the worker against the run's cancellation. Past the bound, the worker is held in _abandoned_state_writers, which pending_work_count counts, and GatewayRunner._in_flight_work_counts (slack/gateway.py L1164) reports that as background work a restart would interrupt. Restoring the marker cannot remove the hard-kill gap. To certify the rewritten bytes, it would have to follow the rename as its own write, which leaves the same gap. Written before the rewrite, it would certify the streamed file, which is exactly the file the rewrite stops trusting, because a failed best-effort append leaves a hole there. A marker before the rewrite could therefore vouch for a fragment as whole, which is the unsafe direction. Only one atomic write of the bytes and the flag together would close it, and no two-file layout provides that. This is the gap answered at c273d8a and 2a62ba0 (#16185 (comment)), where it was rated CONCERNS. The code it reviews has not changed since then. This round changed two comments and two tests, and rebased. The spec's Gateway Restart Reconciliation paragraph (docs/system-specs/modules/subagent.md) records the remaining gap.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 5: a respawned attempt starts result.txt afresh (rides along) span=src/kiro_crew/subagent_persistence.py:write_result_chunk — not changed (advisory)

Rationale: answered at 5937a46 (#16185 (comment)), unchanged since. The whole-answer rewrite reads the streamed text of the CURRENT attempt, and the in-place cap of every other ending reads result.txt. Both are wrong while two attempts' text is glued in one file, so fresh=True is part of what makes "the file the flag describes" a single attempt's bytes.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 6: boot settles reaped, startup_timeout and start_queue_saturated tombstones failed (rides along) span=src/kiro_crew/subagent_manager/admission/types.py:tombstone_terminal_state — not changed (advisory)

Rationale: answered at d0252de (#16185 (comment)), unchanged since. With the marker retired, the boot probe reads result_is_whole(state) together with the cause table, and test_every_tombstone_cause_has_a_terminal_state requires every written cause to be mapped. An unmapped cause would fall through to the restart arm and be announced as cut off by a restart it never had.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 7: update_state reads through a Windows sharing-violation retry; state writes fsync at 0600 (rides along) span=src/kiro_crew/subagent_persistence.py:update_state/_atomic_write — not changed (advisory)

Rationale: this is the flag's own write path, not a separate change. result_complete lands through update_state, and on a read error it returns False and skips the write. On Windows, a reader holding state.json during the rewrite's rename causes a sharing violation, which would skip the flag and leave a whole answer announced as a fragment. That is the Goal's symptom, reached through a different door. The fsync makes the flag durable to the same degree as the fsynced bytes it vouches for. Mode 0600 is the mode the delegated atomic_write applies, kept so state.json's permissions do not regress.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 8: the usage-row append is a drained task (rides along) span=src/kiro_crew/subagent_manager/run.py — not changed (advisory)

Rationale: answered at bf3a5e9 (#16185 (comment)), unchanged since. After the ending claim every stop is a no-op, so an awaited append between the claim and the ending record is a window in which a cancel would land in the record itself. Running the append on a task that cancel_all drains removes that await from the ending.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • The claim has no single owner: about ten stop sites repeat done or _ending_claimed span=cancellation.py/monitoring.py/terminal.py/waits.py/run.py — not changed (advisory Watch)

Rationale: answered at 1541be1 (#16185 (comment)) and 2a62ba0 (#16185 (comment)). This head adds no stop site. It changes two comments and two tests, and rebases. Each stop that can land while a claimed run is still writing checks the claim: cancel, parent end, the reaper, fail_parent, the waits, and settle_before_delete. test_subagent_stop_reason_consistency.py drives each of them against a claimed run, which meets the review's second "Clears when". The sites that read done alone run only after the run's task has finished, when a claimed ending has already set done. A shared predicate would rename those guards without changing behavior, so it belongs in a refactor PR of its own, not in this fix.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Boot probe maps reaped and startup_timeout to FAILED where they were unmapped (recovered on boot) span=src/kiro_crew/subagent_manager/admission/types.py:tombstone_terminal_state — not changed (advisory Watch)

Rationale: answered at bf3a5e9 (#16185 (comment)) and d0252de (#16185 (comment)). The keys change nothing for any tombstone written today. write_tombstone records outcome, and the probe reads that first through outcome_task_state, the same table as the live settle. A reaped or startup-timeout run already settles failed on boot, as it does live. Main's live settle marks these runs FAILED (info.error is set), so main's boot probe disagreeing with its own live settle on the same ending is the gap, not a recover-on-boot decision. A run the live path recorded as failed is not re-dispatched when the gateway happens to restart before its row settles. The keys reach only a tombstone with no outcome, so they keep one cause from settling failed or being re-dispatched depending on when its tombstone was written. test_every_tombstone_cause_has_a_terminal_state pins that every written cause except gateway_restart is mapped. It caught #16020's start_queue_saturated on this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override first-principles 95e5a5d: the hard-kill gap exists on main too. There the result.complete marker is a separate write in a later step than the bytes it certifies. Here the rename, fsync_dir and the flag run back to back in one function, so the gap is narrower, and it fails safe: it under-claims (partial) and never over-claims. Closing it fully needs one atomic write of the bytes and the flag together, which is a design change; the gap is recorded in the Gateway Restart Reconciliation spec.

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 marked the first-principles AI finding as false positive, not applicable, or explicitly accepted for 95e5a5dab124dcb3715ead723ec872c69c489c54.

the hard-kill gap exists on main too. There the result.complete marker is a separate write in a later step than the bytes it certifies. Here the rename, fsync_dir and the flag run back to back in one function, so the gap is narrower, and it fails safe: it under-claims (partial) and never over-claims. Closing it fully needs one atomic write of the bytes and the flag together, which is a design change; the gap is recorded in the Gateway Restart Reconciliation spec.

This decision applies only to this commit. A new push requires a new judgment.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 6: the usage row is appended on a held task instead of before done span=src/kiro_crew/subagent_manager/run.py — not changed (advisory CONCERNS)
    mechanism: after the ending claim every stop is a no-op, so any await between the claim and done must be bounded

Rationale: answered at bf3a5e9 (#16185 (comment)). The review suggests moving the claim above the usage-row await. That is the ordering the answer rules out. persist_token_record_async ends in an unbounded await asyncio.to_thread(...). From the claim on, Stop, a parent end and the reaper all treat the run as done and do nothing, so an unbounded await after the claim leaves a run that nothing can end on a wedged filesystem. The old pin ("done never visible with the usage row missing") has no reader that needs the row before done. cancel_all drains _report_tasks within its bound, so shutdown does not lose the row.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 7: boot fails reaped, startup-timeout and saturated tombstones span=src/kiro_crew/subagent_manager/admission/types.py:tombstone_terminal_state — not changed (advisory CONCERNS)

Rationale: answered at bf3a5e9 (#16185 (comment)). The boot probe reads a tombstone's recorded outcome first, which the Goal needs: a whole answer that a later stop's tombstone records as outcome: completed must settle done on boot. The cause rows reach only a tombstone with no outcome. Dropping them would make the same cause settle failed or be re-dispatched depending on when the tombstone was written. test_every_tombstone_cause_has_a_terminal_state is what caught #16020's new cause on this PR.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 6: each attempt starts result.txt afresh (rides along, inherited) span=src/kiro_crew/subagent_persistence.py:write_result_chunk — not changed (advisory CONCERNS)

Rationale: answered at 1541be1 (#16185 (comment)), 2a62ba0 and 5937a46. This head does not touch it. The review says gluing only affects interrupted runs. It does not: a respawn that completes with NO text writes nothing from memory, so without the fresh start, attempt 1's fragment stays behind a result_complete=True flag and the restart notice vouches for it as finished. That is the Goal's own reader. Both directions fail on base: test_a_textless_whole_answer_leaves_no_result_file and test_a_cut_off_respawn_delivers_the_partial_it_left_on_disk.

@bolichen97

Copy link
Copy Markdown
Contributor Author
  • Item 7: turn-limit, error and deadline endings now cap result.txt (rides along) span=src/kiro_crew/subagent_manager/run.py:_cap_unclaimed_result_impl — not changed (advisory CONCERNS)

Rationale: answered at 1541be1 (#16185 (comment)) and 2a62ba0 (#16185 (comment)). This is base's own tail cap, moved to _run's finally, the one place every ending passes. The ending claim replaced the tail's flag write, and a second writer of the flag is the ordering hazard this PR removes. The endings that never reached the tail are capped because of where the single call sits, with no extra code.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Oct 4, 2026
@chenmingwei23
chenmingwei23 merged commit 4da8c7b into main Oct 4, 2026
98 of 99 checks passed
@chenmingwei23
chenmingwei23 deleted the followup/15670-result-complete branch October 4, 2026 15:28
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

2 participants