Skip to content

fix(agents): hold the wave-close fallback open while a member's report is in flight - #16985

Merged
chenmingwei23 merged 1 commit into
mainfrom
replace/wave-close-fallback-9008
Oct 13, 2026
Merged

chenmingwei23 merged 1 commit into
mainfrom
replace/wave-close-fallback-9008

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Supersedes #9008 (original author @javenciu — commit authorship preserved; continued here as an in-repo PR so the same-repo review lanes can adjudicate the finding a fork PR fences off).

Problem / Motivation

Goal: A multi-member sub-agent wave emits its final digest exactly once, even when a member is done-but-unreported.

A multi-member wave can emit its final digest twice — or finalize prematurely with sibling results missing — when one member's SubagentInfo.done flips to True before that member's terminal report reaches the gateway completion consumer (_subagent_done). Reachable with two RUNNING members alone.

Why it matters

A duplicated wave digest double-delivers a completion to the parent (crash-/data-loss-class: the parent acts on the same wave twice), and a premature finalize strands every sibling's result until a gateway restart. Both are silent — no error surfaces — so a user sees a wave either repeated or quietly truncated.

Not a goal

  • Does not change the wave digest's count-trigger (SUBAGENT_DIGEST_CHUNK_SIZE) or the deliberate "small wave = one consolidated digest" behavior.
  • Does not alter the one-shot finalize claim or the shielded-report mechanism; only closes the done-but-unreported window around them.

What changed (motivation → approach → change)

  • Symptom: batch_members_pending() stops counting a member the moment info.done flips, but that member's contribution to the consumer's bp["done"] only lands when its (shielded, possibly slow) terminal report actually invokes the consumer. In that window a sibling completion observes done < total with no pending members, trips the last-member fallback (finalizes early), and the in-flight report then re-creates the batch-progress record via setdefault and finalizes the same wave again.
  • Root cause: the window between a terminal info.done = True flip and the consumer landing that member's done-count is unguarded; the earlier design that put a hold flag on the agent records was also vulnerable to an operator DELETE /api/spawn clear popping the member mid-window.
  • Change: a manager-level _reports_in_flight registry in subagent.py (immune to _agents mutation). It is armed in the same synchronous block as EVERY terminal info.done flip — the report machinery's, the run's failure except-bodies, the in-run limit bails, the reaper's, and the spawn-rejection flips (safe-announced synthetics arm immediately before their deferred announce, closing the exact late-arm gap this PR's review raised). It is released by the completion consumer in the same synchronous block as the bp["done"] increment, and structurally by the report path's try/finally plus the report task's done-callback, so a report that ends without reaching the consumer (injection timeout, announce failure, cancellation) never strands the wave. The wave-close decision (_last in _subagent_done) and the reaper's digest-hold sweep both consult batch_reports_in_flight; the sweep force-flushes a wave whose hold aged past the deadline with no report in flight (previously unsweepable). The cancelled-recovery limbo arm is deliberately report-free — its not info.done guard proves no hold was ever armed there.
  • Only the flip that makes the done transition arms. The run's tail after its reap reported (was_done), the claimed-success flip and a spawn approval denied after a Stop read done before flipping and skip the arm when another path already ended the record: that path armed at its own flip and its report releases the hold, while the late site spawns no report for the claim it lost, so a re-arm would pin the wave open until finalize_batch.
  • Consumer ordering (gateway.py): the completion consumer's one await on the wave path (the off-loop _subagent_batch_pending lookup) now runs BEFORE the member touches _batch_progress, and is skipped when the member's own count closes the wave. From reading the record through the hold-or-flush decision the consumer is synchronous, so a sibling can no longer close the wave between a member writing its result line and that member choosing how the line is delivered: the closing digest's settle set always carries every member it names, and no member is left held with no settler (which restart reconciliation would re-deliver).

Backwards compatibility

Removes nothing: this PR only adds a manager-internal registry and new predicate reads; no field, record, kind, validator, or public function is removed, renamed, or tightened, and no caller contract changes.

Tests

  • test/test_wave_digest_double_finalize.py (new): drives the REAL _subagent_done closure with a scripted manager so the race is deterministic — a sibling completion in the done-but-unreported window does not close the wave; an in-flight report cannot re-finalize after close; the consumer releases the hold with the count; the spawn-failure fallback still closes the wave; and the timeout/cancel/exception failure paths each arm in the same synchronous block as the flip.
  • test/test_wave_digest_double_finalize.py also drives a sibling closing the wave while a member is suspended in the membership lookup, and asserts every held member has a settler.
  • test/test_wave_digest_double_finalize.py TestOnlyTheTransitionArmsTheHold: the real spawn -> _run -> _run_inner path and the real _force_reap. The reap records the ending and its report reaches the consumer (hold released), then the run's stream ends and the run unwinds through its was_done tail; a Stop reaps a run parked at spawn approval, then the approval resolves denied; and the claimed-success flip with an ending recorded inside its result write. Each asserts no hold is left.
  • test/test_subagent_coverage.py: adds TestBatchReportsInFlight (arm/release/predicate semantics across every terminal path incl. the report-free cancelled-recovery arm) and test_stranded_hold_past_deadline_is_forced (reaper force-flushes an aged hold with no report in flight); updates test_closing_wave_not_forced to arm the hold. Both sweep tests run the sync entry and the awaited form the reaper loop calls. test_the_report_body_releases_its_hold_without_its_task pins the report body's own finally release apart from the task done-callback.
  • test/test_subagent_scale.py: retargets the subagent-done ordering guard at the extracted _report_terminal_guarded_impl and arms the hold in test_closing_wave_is_not_force_flushed.

Red before, green after (this round): with only the three arm guards reverted in place (run.py was_done tail and claimed-success finally, pump.py approval-denied flip), the three TestOnlyTheTransitionArmsTheHold tests fail at their final batch_reports_in_flight(...) is False assert, after their mid-test asserts passed (the reap's report released the hold; the tail ran). Restored, they pass. Mutations: removing the report body's own release (terminal.py _report_terminal_impl finally) reddens only test_the_report_body_releases_its_hold_without_its_task; making the async sweep always skip an unpending wave reddens test_stranded_hold_past_deadline_is_forced[async], and never skipping a held one reddens test_closing_wave_not_forced[async].

Determinism (testing-conventions D12) for the 8 new or changed node ids: 20/20 at -n0; 10/10 at -n 2 --dist loadgroup --max-worker-restart=2 beside test_subagent_reap_race.py and TestDigestHoldDeadline (-n 2, not 4: shared host); shuffled order, seed 19426, 8 passed. Every wait is on an event or a task (await_until), no sleeps. Scoped run: test_wave_digest_double_finalize.py, test_subagent_coverage.py, test_subagent_reap_race.py, test_subagent_stop_reason_consistency.py and the two test_subagent_scale.py wave classes, 447 passed at -n 2.

Manual verification

N/A — unit coverage sufficient: the race window is driven deterministically through the real completion closure, so no timing-dependent manual repro is needed.

Related Issues

Fixes #8554

Pattern harvest

Rule candidate: review-prompt
Pattern: a terminal state flag (done) that gates wave/batch accounting must be armed in the same synchronous block as the flip, never left to a later-scheduled task — a deferred arm leaves a window where a sibling observes the member as neither pending nor in flight. And only the flip that makes the transition arms: a late site re-arming an already-ended record outlives the release.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner October 5, 2026 09:28
@iamwhatever
iamwhatever requested a review from buluoray October 5, 2026 09:28
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of 99c95aef31b91fb82e48cf76ef54ecff18294627 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

SubagentManager._report_terminal_guarded is a new manager passthrough that nothing calls. The real body is reached directly through self._terminal._report_terminal_guarded_impl.

Not justified as shipped

  • Item 6 — zero consumers: grepping _report_terminal_guarded( across src/ and test/ finds 0 call sites for the new subagent.py wrapper async def _report_terminal_guarded(...). terminal.py calls self._report_terminal_guarded_impl directly, and the only other mention is a comment in test_subagent_scale.py:1760.
    Subtraction: delete SubagentManager._report_terminal_guarded from src/kiro_crew/subagent.py and keep only TerminalCoordinator._report_terminal_guarded_impl.
    Clears when: the wrapper is deleted, or a non-test caller of it is counted.
  • Item 3 — rides along: the stated fix is that a wave's digest is emitted once. Separately, the reaper sweep no longer skips an aged hold when no member is pending. The base comment kept that skip for the case where "the held flags are stale bookkeeping"; the PR now force-flushes there. It is declared and covered by test_stranded_hold_past_deadline_is_forced.

What this change ships

Inventory (6 items) — 4 justified

Intent: make a multi-member sub-agent wave deliver its final digest to the parent exactly once. This is a FIX.

  1. A sibling's completion no longer closes a wave while another member is done but its report has not landed (18 hold arms placed at the flip sites) — justified
  2. The completion consumer releases the hold in the same step that counts the member, and three structural releases make sure a failed report never strands the wave — justified
  3. The reaper now force-flushes an aged held digest when nothing is pending or in flight — rides along
  4. The consumer's membership lookup now runs before it touches the wave record, and is skipped when this member's count closes the wave — justified
  5. Three manager methods: batch_reports_in_flight, arm_report_in_flight, consume_report_hold — justified
  6. A manager passthrough _report_terminal_guarded that nothing calls — zero consumers

[FIRST-PRINCIPLES-REVIEWED] 99c95ae

Description read

[DESCRIPTION-READ] 5f6e41b180363a9b55d13824dcbb649c89058ed0075701c978a275a7127a0a7a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of 99c95aef31b91fb82e48cf76ef54ecff18294627 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The fix depends on every info.done flip site remembering to arm the hold, and an armed hold that never gets released has no deadline, so the wave stays open until restart.

Watch

  • Correctness rests on every flip site remembering to arm. The new docstring requires arming "in the same synchronous block as EVERY terminal info.done flip". That spans run.py, pump.py, gate.py, cancellation.py, terminal.py and waves.py. Some sites arm unconditionally and others must skip the arm (was_done, claimed-success, approval-denied). If a future flip site forgets to arm, the double-finalize race comes back, and no error or test catches it. Putting the flip and the arm in one SubagentManager method (say mark_terminal(info)) would make that mistake impossible. As written, the invariant is spread across six modules.
    Clears when: every terminal done = True goes through one helper that also arms, or a test fails when a .done = True assignment appears outside it.
  • A hold that is armed but never released has no backstop. In both sweep forms, if self._manager.batch_reports_in_flight(hold[0]): continue skips the force-flush however old the hold is. A leaked arm therefore keeps the wave open, and sibling results stay held until finalize_batch or a gateway restart. The description names this failure itself ("a re-arm would pin the wave open"). Today only the per-site discipline above prevents it.
    Clears when: the sweep force-flushes once an in-flight hold is past its own deadline (store the arm time), or the PR shows why a leak can't happen without relying on per-site discipline.

[DESIGN-REVIEWED] 99c95ae

Description read

[DESCRIPTION-READ] 5a2cb1cfd36fb1efc168dfe5ea765756bcea6f0f9164b2dad719a76937a8abdb

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 99c95aef31b91fb82e48cf76ef54ecff18294627 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 99c95ae

Verdict parsed from the review's SHA-scoped output markers for commit 99c95aef31b91fb82e48cf76ef54ecff18294627.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 99c95aef31b91fb82e48cf76ef54ecff18294627: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 1a3d907 to 64e6939 Compare October 5, 2026 09:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 99c95aef31b91fb82e48cf76ef54ecff18294627 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 99c95ae

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 99c95aef31b91fb82e48cf76ef54ecff18294627: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 64e6939 to deb49be Compare October 5, 2026 09:54
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from deb49be to 27606aa Compare October 5, 2026 10:32
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 27606aa to 6cc269d Compare October 5, 2026 11:06
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 6cc269d to 458c9e2 Compare October 5, 2026 11:39
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 5, 2026
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 458c9e2 to 1d7afe3 Compare October 5, 2026 16:01
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 5, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from f20125b to 2192e3c Compare October 12, 2026 09:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Pushed 2192e3c06: the two red lanes on f20125bb2 were in the new test file only. Semgrep test-sleep-as-barrier (two asyncio.sleep(30) hang stand-ins → un-set asyncio.Event().wait()) and the wait-discipline ratchet (wait_for(10s) → kiro_crew.testing.wait.await_until). No product code changed in this push. GPT 6.1 on f20125bb2: no findings.

Dispositions for the advisory CONCERNS (no code change):

  • First Principles item 5 (SubagentManager._report_terminal_guarded has no caller). It is required by the facade wiring, not by a caller: copy_component_docs(SubagentManager, _MANAGER_COMPONENTS) (src/kiro_crew/subagent.py) does getattr(facade_type, name.removesuffix("_impl")) for every *_impl on a component, so deleting the facade method makes the module fail at import. Kept, same as every other _impl facade forwarder.
  • First Principles item 3 (consumer ordering undeclared). Declared now in "What changed" and "Tests".
  • First Principles item 1 (fold the flip+arm pairs into one call). A fair refactor, but it rewrites every terminal arm of _run_impl, the reaper and the rejection paths in a PR whose job is the race fix. The source-scan test is the guard that a new flip cannot ship unarmed; it caught main's new session-not-found flip during this rebase. Left as a follow-up.
  • Design watch (a leaked hold strands the wave). Every arm has a structural releaser: the consumer's release with the count, the report path's try/finally, and the report/announce task done-callbacks (TestStrandGuards drives a task cancelled before its first run). An age-out on the in-flight hold would re-open the exact double-finalize this PR closes when a report is merely slow, so it is not added here. The "simpler shape" suggestion (count seen members in bp) is noted for the same follow-up.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 2192e3c. The design is right and well pinned: a wave no longer sends its digest twice or closes with sibling results missing in the done-but-unreported window. One narrow path can strand a wave, so requesting changes.

Blocking: a hold re-armed on an already-done record can outlive its only release. (Traced from source; not reproduced by running it.)

  • subagent_manager/run.py:3516-3518 does was_done = info.done; info.done = True; arm_report_in_flight(info), arming even when was_done is True. The comment calls this idempotent, but arm_report_in_flight_impl (waves.py:100) re-adds the id to _reports_in_flight unconditionally.
  • The claimed-success finally at run.py:3484-3493 has the same shape: it arms after await _write_finished_result_off_loop, and a reap can flip done inside that await.
  • Sequence: another path (typically _force_reap) records the ending first, takes the finalize claim, launches the report, and the report reaches the consumer and releases the hold. The run then unwinds to these lines and re-adds the id. _run's finally gets _claim_finalize(info) == False, so it spawns no report, and nothing releases the id until finalize_batch.
  • Effect: a wave that closes by count is fine. A wave that needs the last-member fallback (a member failed at spawn, or a report never reached the consumer) can never close, because _last requires no report in flight and the digest-hold sweep keeps skipping it (waves.py:353, :378). Held sibling results strand until a gateway restart, which is worse than base in that case.
  • Fix: arm only on the transition this code made (if not was_done: at :3518; capture not info.done before the flip in the :3484 finally). Optionally also bound the hold, letting the sweep force the flush once a hold is older than DIGEST_HOLD_SECS + _ON_DONE_TIMEOUT, with a test. test_no_terminal_done_flip_is_left_unarmed may need to accept a guarded arm.

Verified, with mutations: dropping and not batch_reports_in_flight(...) from the consumer's _last reddens test_sibling_completion_in_the_done_but_unreported_window_does_not_close_the_wave; never skipping a held wave in the sweep reddens test_closing_wave_not_forced; reverting the sweep to base reddens test_stranded_hold_past_deadline_is_forced; removing both report releases (terminal.py:311 and :642) reddens 4 tests including test_report_injection_timeout_clears_the_hold; removing the consumer's consume_report_hold reddens test_consumer_releases_the_hold_with_the_count.

Non-blocking:

  • The two report releases are only pinned together; removing terminal.py:311 alone stays green.
  • The async sweep's new branch may be unpinned; only sync-form tests reddened.
  • The body says test_subagent_reap_race.py was extended, but that file is not in the diff.
  • pump.py:1073 (if not approved: info.done = True) flips and arms without a not info.done guard; worth the same arm-on-transition treatment.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Picking up @buluoray's review on 2192e3c from the Crew page polish conductor. One fix push to follow.

…t is in flight

A multi-member wave could emit its final digest twice, or finalize
prematurely with results missing, when one member's done flag flipped
before its terminal report reached the completion consumer.

batch_members_pending() stops counting a member the moment info.done
flips, but that member's contribution to the consumer's done-count only
lands when its (shielded, possibly slow) terminal report actually
executes the consumer. In that window a sibling completion observes
done < total with no pending members, so the last-member fallback
finalized the wave early -- and the in-flight report then re-created the
batch-progress record and finalized the same wave again. Reachable with
two RUNNING members alone.

The hold lives in a manager-level `_reports_in_flight` registry (immune
to operator `DELETE /api/spawn` clears, which pop done members from
`_agents`): armed in the same synchronous block as EVERY terminal
`info.done` flip -- the report machinery's, the run's failure
except-bodies, the in-run limit bails, the reaper's, and spawn-rejection
flips (safe-announced synthetics arm immediately before their announce);
released by the completion consumer in the same synchronous block as the
done-count increment, and structurally by the report path's try/finally
(and the report task's done-callback) so a report that ends without
reaching the consumer -- injection timeout, announce failure,
cancellation -- never strands the wave. The reaper's digest-hold sweep
is in-flight-aware in both directions: it skips a wave whose close is
genuinely in flight, and force-flushes a wave whose hold aged past the
deadline with no report in flight (previously unsweepable).

Only the flip that makes the done transition arms. A flip site reached
after another path recorded the ending -- the run's tail unwinding after
its reap reported (`was_done`), the claimed-success flip, a spawn
approval denied after a Stop -- does not re-arm: that path armed at its
own flip and its report releases the hold, possibly already, while the
late site spawns no report for the claim it lost. A re-arm there would
pin `batch_reports_in_flight` until `finalize_batch`, so a wave needing
the last-member fallback could never close.

The cancelled-recovery limbo arm is deliberately report-free: holds are
armed only at a report's done-flip, which that arm's `not info.done`
guard proves never ran, so `batch_reports_in_flight` cannot strand on it.

Part of #8554.

Co-authored-by: iamwhatever <12429733+iamwhatever@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the replace/wave-close-fallback-9008 branch from 2192e3c to 99c95ae Compare October 12, 2026 20:44
@iamwhatever

Copy link
Copy Markdown
Contributor Author

@buluoray thanks, the blocking point was real. Fixed in 99c95ae (still one commit, rebased on main 65df088). Besides the rebase, the delta from 2192e3c is three guarded arms, two doc touches and the tests below.

Blocking: a re-armed hold outliving its only release. Fixed as you proposed: only the line that makes the done transition arms.

  • run.py was_done tail: the arm is now under if not was_done:.
  • run.py claimed-success finally: _flipped = not info.done is read before the flip, and it arms only if it flipped. One note: no reap can flip done inside that await today. _force_reap returns on _ending_claimed (terminal.py, top of _force_reap_impl), and the claim is set before the await. So this guard keeps the rule uniform; it is pinned by a test, but it is not a second live path.
  • pump.py approval-denied flip (your last non-blocking item): same guard, _was_done read before the flip.
  • The arm_report_in_flight_impl docstring (waves.py) and the wave-close paragraph in docs/system-specs/modules/subagent.md now say the arm belongs to the transition.

Red before, green after: TestOnlyTheTransitionArmsTheHold in test/test_wave_digest_double_finalize.py, waiting on events and tasks only, no sleeps.

  • Real spawn -> _run -> _run_inner plus the real _force_reap. The reap records the ending and claims, and its report reaches the consumer, which releases the hold (asserted mid-test). Then the stream answers the cancel with a cancelled completion, and the run unwinds through the was_done tail.
  • A Stop reaps a run parked at spawn approval, and its report releases the hold. Then the approval resolves denied.
  • The claimed-success flip, with another path's whole ending modeled inside its result write.

With only the three guards reverted in place, all three fail at their final batch_reports_in_flight(...) is False assert, after their mid-test asserts passed. Restored, all three pass. They also passed 20/20 at -n0, 10/10 at -n 2 beside co-tenants, and in a shuffled order (seed 19426).

Optional hold bound: not taken. After this change, every arm comes from the flip that ended the record. Each one is paired with a report or announce whose finally or done-callback releases it. The aged-hold sweep already force-flushes once no report is in flight. A per-hold arm timestamp would be a new mechanism guarding a strand path that no longer exists. Happy to add it as a follow-up if you still want it.

Non-blocking:

  • Releases pinned only together: test_the_report_body_releases_its_hold_without_its_task calls _report_terminal with no task around it, so only the body's own finally can release. Removing terminal.py:311 alone now reddens it, and only it.
  • Async sweep: test_closing_wave_not_forced and test_stranded_hold_past_deadline_is_forced now run both the sync entry and _sweep_digest_holds_async, the form the reaper loop calls. If the async form always skips a wave with nothing pending, [async] of the stranded test goes red. If it never skips a held wave, [async] of the closing test goes red.
  • Body: the test_subagent_reap_race.py line was stale (the guard retarget lives in test_subagent_scale.py). It is corrected.
  • pump.py:1073: done, see above.

@iamwhatever
iamwhatever requested a review from buluoray October 12, 2026 20:45
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Zero consumers: the SubagentManager._report_terminal_guarded passthrough span=0fdbae108d46

rebutted (disproportional on this head): correct, nothing calls the wrapper; TerminalCoordinator._report_terminal_impl reaches the body as self._report_terminal_guarded_impl.
It is inert: it adds no behaviour, and no caller can reach a different path through it.
Deleting it needs a new head and a full CI round for zero behaviour change, while this head is green and waits on a human re-review.
It is dropped in the next push this PR takes (for example, if the re-review asks for a change).

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Rides along: the sweep force-flushes an aged hold when nothing is pending span=04e7805621ed

rebutted (not riding along): this branch is the liveness half of the hold.
With the hold, a sibling completion defers the wave close to a member whose report is in flight. That deferral is the fix.
If that report then ends without reaching the consumer (injection timeout, announce failure), its structural release lands no count, and no event re-enters the consumer. Base closed that wave on the sibling's own completion. Without this branch, the fix would turn a double digest into a stranded wave.
A stale held flag is safe to flush: the consumer clears holds in the block that fires the chunk, and a flush-only record for a closed wave is a no-op.
Pinned by test_stranded_hold_past_deadline_is_forced (sync and async forms). @buluoray's mutation (sweep reverted to base) reddens it.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • Correctness rests on every flip site remembering to arm span=6dcfbe650af3

rebutted (the Clears-when test exists): test_no_terminal_done_flip_is_left_unarmed in test/test_subagent_coverage.py scans every src/kiro_crew/subagent_manager/**/*.py. It fails on any <x>.done = True that has no arm_report_in_flight(<x>) within the next 15 lines, unless the site carries the documented report-free marker.
Deferred announces have their own scan (test_every_deferred_safe_announce_is_armed_first). The facade's synthetic batch records go through _announce_rejection, which arms before it schedules the announce.
So a forgotten arm fails CI. The three transition-guarded arms keep the call inside that window, so the scan still sees them.
One mark_terminal(info) helper would move every flip site across six modules. That is a refactor beyond this fix, for the same guarantee the scan already gives.

@iamwhatever

Copy link
Copy Markdown
Contributor Author
  • A hold that is armed but never released has no backstop span=415f92c47689

rebutted (disproportional): every arm is paired with a release on the path that made it.
Report-path arms release in their own finally (_report_terminal_impl, _safe_announce). A flip-site arm precedes the one report its terminal path claims, and that report task's done-callback releases even when the task is cancelled before its first step. Lost-submission and rejection announces add a done-callback release.
The one leak found (a late flip re-arming after the report released) is fixed on this head: only the transition arms. TestOnlyTheTransitionArmsTheHold pins it and was red before the fix.
Each release is pinned and mutation-checked: in @buluoray's review, and by test_the_report_body_releases_its_hold_without_its_task for the body's own release.
A per-hold arm time is the optional bound the human review offered. The review reply declines it as a new mechanism for a path that no longer exists, and leaves it open as a follow-up.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed at head 99c95ae; this supersedes my request-changes on 2192e3c. The blocking point is fixed: every flip site now arms the report hold only on the done transition it made.

  • run.py:3558 arms under if not was_done:; run.py:3529 captures _flipped = not info.done before the claimed-success flip; admission/pump.py:1091 reads _was_done before the approval-rejection flip and arms under if not _was_done: (the non-blocking point from last round, addressed too).
  • subagent.md now states that only the line making the done transition arms, and why.

Verified with mutations (each guard replaced with an unconditional arm, then restored):

  • run.py:3558 -> TestOnlyTheTransitionArmsTheHold::test_a_run_unwinding_after_its_reap_reported_leaves_no_hold goes red.
  • run.py:3529 -> test_the_claimed_flip_does_not_rearm_an_ending_recorded_in_its_write goes red.
  • pump.py:1091 -> test_a_spawn_approval_denied_after_a_stop_leaves_no_hold goes red.
  • Clean: test_wave_digest_double_finalize.py + test_subagent_coverage.py, 309 passed. CI on this head: 88 pass, 0 fail.

My other non-blocking points are also covered: the sweep tests now run in sync and async form, and test_the_report_body_releases_its_hold_without_its_task pins the body release on its own. Declining the optional per-hold age bound is fine with me; it can stay a follow-up.

@chenmingwei23
chenmingwei23 merged commit 37c2973 into main Oct 13, 2026
102 checks passed
@chenmingwei23
chenmingwei23 deleted the replace/wave-close-fallback-9008 branch October 13, 2026 05:30
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Wave digest can finalize twice when a member is done-but-unreported

5 participants