Repository navigation
fix(agents): hold the wave-close fallback open while a member's report is in flight - #16985
Conversation
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (6 items) — 4 justifiedIntent: make a multi-member sub-agent wave deliver its final digest to the parent exactly once. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 99c95ae Description read[DESCRIPTION-READ] 5f6e41b180363a9b55d13824dcbb649c89058ed0075701c978a275a7127a0a7a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — 🟡 CONCERNSDesign-level review of Design-Verdict: CONCERNS The fix depends on every Watch
[DESIGN-REVIEWED] 99c95ae Description read[DESCRIPTION-READ] 5a2cb1cfd36fb1efc168dfe5ea765756bcea6f0f9164b2dad719a76937a8abdb That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
1a3d907 to
64e6939
Compare
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
64e6939 to
deb49be
Compare
deb49be to
27606aa
Compare
27606aa to
6cc269d
Compare
6cc269d to
458c9e2
Compare
458c9e2 to
1d7afe3
Compare
f20125b to
2192e3c
Compare
|
Pushed Dispositions for the advisory CONCERNS (no code change):
|
buluoray
left a comment
There was a problem hiding this comment.
Reviewed at head 2192e3c. The design is right and well pinned: a wave no longer sends its digest twice or closes with sibling results missing in the done-but-unreported window. One narrow path can strand a wave, so requesting changes.
Blocking: a hold re-armed on an already-done record can outlive its only release. (Traced from source; not reproduced by running it.)
subagent_manager/run.py:3516-3518doeswas_done = info.done; info.done = True; arm_report_in_flight(info), arming even whenwas_doneis True. The comment calls this idempotent, butarm_report_in_flight_impl(waves.py:100) re-adds the id to_reports_in_flightunconditionally.- The claimed-success
finallyatrun.py:3484-3493has the same shape: it arms afterawait _write_finished_result_off_loop, and a reap can flipdoneinside that await. - Sequence: another path (typically
_force_reap) records the ending first, takes the finalize claim, launches the report, and the report reaches the consumer and releases the hold. The run then unwinds to these lines and re-adds the id._run'sfinallygets_claim_finalize(info) == False, so it spawns no report, and nothing releases the id untilfinalize_batch. - Effect: a wave that closes by count is fine. A wave that needs the last-member fallback (a member failed at spawn, or a report never reached the consumer) can never close, because
_lastrequires no report in flight and the digest-hold sweep keeps skipping it (waves.py:353,:378). Held sibling results strand until a gateway restart, which is worse than base in that case. - Fix: arm only on the transition this code made (
if not was_done:at :3518; capturenot info.donebefore the flip in the :3484finally). Optionally also bound the hold, letting the sweep force the flush once a hold is older thanDIGEST_HOLD_SECS + _ON_DONE_TIMEOUT, with a test.test_no_terminal_done_flip_is_left_unarmedmay need to accept a guarded arm.
Verified, with mutations: dropping and not batch_reports_in_flight(...) from the consumer's _last reddens test_sibling_completion_in_the_done_but_unreported_window_does_not_close_the_wave; never skipping a held wave in the sweep reddens test_closing_wave_not_forced; reverting the sweep to base reddens test_stranded_hold_past_deadline_is_forced; removing both report releases (terminal.py:311 and :642) reddens 4 tests including test_report_injection_timeout_clears_the_hold; removing the consumer's consume_report_hold reddens test_consumer_releases_the_hold_with_the_count.
Non-blocking:
- The two report releases are only pinned together; removing
terminal.py:311alone stays green. - The async sweep's new branch may be unpinned; only sync-form tests reddened.
- The body says
test_subagent_reap_race.pywas extended, but that file is not in the diff. pump.py:1073(if not approved: info.done = True) flips and arms without anot info.doneguard; worth the same arm-on-transition treatment.
…t is in flight A multi-member wave could emit its final digest twice, or finalize prematurely with results missing, when one member's done flag flipped before its terminal report reached the completion consumer. batch_members_pending() stops counting a member the moment info.done flips, but that member's contribution to the consumer's done-count only lands when its (shielded, possibly slow) terminal report actually executes the consumer. In that window a sibling completion observes done < total with no pending members, so the last-member fallback finalized the wave early -- and the in-flight report then re-created the batch-progress record and finalized the same wave again. Reachable with two RUNNING members alone. The hold lives in a manager-level `_reports_in_flight` registry (immune to operator `DELETE /api/spawn` clears, which pop done members from `_agents`): armed in the same synchronous block as EVERY terminal `info.done` flip -- the report machinery's, the run's failure except-bodies, the in-run limit bails, the reaper's, and spawn-rejection flips (safe-announced synthetics arm immediately before their announce); released by the completion consumer in the same synchronous block as the done-count increment, and structurally by the report path's try/finally (and the report task's done-callback) so a report that ends without reaching the consumer -- injection timeout, announce failure, cancellation -- never strands the wave. The reaper's digest-hold sweep is in-flight-aware in both directions: it skips a wave whose close is genuinely in flight, and force-flushes a wave whose hold aged past the deadline with no report in flight (previously unsweepable). Only the flip that makes the done transition arms. A flip site reached after another path recorded the ending -- the run's tail unwinding after its reap reported (`was_done`), the claimed-success flip, a spawn approval denied after a Stop -- does not re-arm: that path armed at its own flip and its report releases the hold, possibly already, while the late site spawns no report for the claim it lost. A re-arm there would pin `batch_reports_in_flight` until `finalize_batch`, so a wave needing the last-member fallback could never close. The cancelled-recovery limbo arm is deliberately report-free: holds are armed only at a report's done-flip, which that arm's `not info.done` guard proves never ran, so `batch_reports_in_flight` cannot strand on it. Part of #8554. Co-authored-by: iamwhatever <12429733+iamwhatever@users.noreply.github.com>
2192e3c to
99c95ae
Compare
|
@buluoray thanks, the blocking point was real. Fixed in 99c95ae (still one commit, rebased on main 65df088). Besides the rebase, the delta from 2192e3c is three guarded arms, two doc touches and the tests below. Blocking: a re-armed hold outliving its only release. Fixed as you proposed: only the line that makes the done transition arms.
Red before, green after:
With only the three guards reverted in place, all three fail at their final Optional hold bound: not taken. After this change, every arm comes from the flip that ended the record. Each one is paired with a report or announce whose Non-blocking:
|
|
|
|
|
buluoray
left a comment
There was a problem hiding this comment.
Re-reviewed at head 99c95ae; this supersedes my request-changes on 2192e3c. The blocking point is fixed: every flip site now arms the report hold only on the done transition it made.
run.py:3558arms underif not was_done:;run.py:3529captures_flipped = not info.donebefore the claimed-success flip;admission/pump.py:1091reads_was_donebefore the approval-rejection flip and arms underif not _was_done:(the non-blocking point from last round, addressed too).subagent.mdnow states that only the line making the done transition arms, and why.
Verified with mutations (each guard replaced with an unconditional arm, then restored):
run.py:3558->TestOnlyTheTransitionArmsTheHold::test_a_run_unwinding_after_its_reap_reported_leaves_no_holdgoes red.run.py:3529->test_the_claimed_flip_does_not_rearm_an_ending_recorded_in_its_writegoes red.pump.py:1091->test_a_spawn_approval_denied_after_a_stop_leaves_no_holdgoes red.- Clean:
test_wave_digest_double_finalize.py+test_subagent_coverage.py, 309 passed. CI on this head: 88 pass, 0 fail.
My other non-blocking points are also covered: the sweep tests now run in sync and async form, and test_the_report_body_releases_its_hold_without_its_task pins the body release on its own. Declining the optional per-hold age bound is fine with me; it can stay a follow-up.
Supersedes #9008 (original author @javenciu — commit authorship preserved; continued here as an in-repo PR so the same-repo review lanes can adjudicate the finding a fork PR fences off).
Problem / Motivation
Goal: A multi-member sub-agent wave emits its final digest exactly once, even when a member is done-but-unreported.
A multi-member wave can emit its final digest twice — or finalize prematurely with sibling results missing — when one member's
SubagentInfo.doneflips toTruebefore that member's terminal report reaches the gateway completion consumer (_subagent_done). Reachable with two RUNNING members alone.Why it matters
A duplicated wave digest double-delivers a completion to the parent (crash-/data-loss-class: the parent acts on the same wave twice), and a premature finalize strands every sibling's result until a gateway restart. Both are silent — no error surfaces — so a user sees a wave either repeated or quietly truncated.
Not a goal
SUBAGENT_DIGEST_CHUNK_SIZE) or the deliberate "small wave = one consolidated digest" behavior.What changed (motivation → approach → change)
batch_members_pending()stops counting a member the momentinfo.doneflips, but that member's contribution to the consumer'sbp["done"]only lands when its (shielded, possibly slow) terminal report actually invokes the consumer. In that window a sibling completion observesdone < totalwith no pending members, trips the last-member fallback (finalizes early), and the in-flight report then re-creates the batch-progress record viasetdefaultand finalizes the same wave again.info.done = Trueflip and the consumer landing that member's done-count is unguarded; the earlier design that put a hold flag on the agent records was also vulnerable to an operatorDELETE /api/spawnclear popping the member mid-window._reports_in_flightregistry insubagent.py(immune to_agentsmutation). It is armed in the same synchronous block as EVERY terminalinfo.doneflip — the report machinery's, the run's failure except-bodies, the in-run limit bails, the reaper's, and the spawn-rejection flips (safe-announced synthetics arm immediately before their deferred announce, closing the exact late-arm gap this PR's review raised). It is released by the completion consumer in the same synchronous block as thebp["done"]increment, and structurally by the report path'stry/finallyplus the report task's done-callback, so a report that ends without reaching the consumer (injection timeout, announce failure, cancellation) never strands the wave. The wave-close decision (_lastin_subagent_done) and the reaper's digest-hold sweep both consultbatch_reports_in_flight; the sweep force-flushes a wave whose hold aged past the deadline with no report in flight (previously unsweepable). The cancelled-recovery limbo arm is deliberately report-free — itsnot info.doneguard proves no hold was ever armed there.was_done), the claimed-success flip and a spawn approval denied after a Stop readdonebefore flipping and skip the arm when another path already ended the record: that path armed at its own flip and its report releases the hold, while the late site spawns no report for the claim it lost, so a re-arm would pin the wave open untilfinalize_batch.gateway.py): the completion consumer's one await on the wave path (the off-loop_subagent_batch_pendinglookup) now runs BEFORE the member touches_batch_progress, and is skipped when the member's own count closes the wave. From reading the record through the hold-or-flush decision the consumer is synchronous, so a sibling can no longer close the wave between a member writing its result line and that member choosing how the line is delivered: the closing digest's settle set always carries every member it names, and no member is left held with no settler (which restart reconciliation would re-deliver).Backwards compatibility
Removes nothing: this PR only adds a manager-internal registry and new predicate reads; no field, record, kind, validator, or public function is removed, renamed, or tightened, and no caller contract changes.
Tests
test/test_wave_digest_double_finalize.py(new): drives the REAL_subagent_doneclosure with a scripted manager so the race is deterministic — a sibling completion in the done-but-unreported window does not close the wave; an in-flight report cannot re-finalize after close; the consumer releases the hold with the count; the spawn-failure fallback still closes the wave; and the timeout/cancel/exception failure paths each arm in the same synchronous block as the flip.test/test_wave_digest_double_finalize.pyalso drives a sibling closing the wave while a member is suspended in the membership lookup, and asserts every held member has a settler.test/test_wave_digest_double_finalize.pyTestOnlyTheTransitionArmsTheHold: the realspawn->_run->_run_innerpath and the real_force_reap. The reap records the ending and its report reaches the consumer (hold released), then the run's stream ends and the run unwinds through itswas_donetail; a Stop reaps a run parked at spawn approval, then the approval resolves denied; and the claimed-success flip with an ending recorded inside its result write. Each asserts no hold is left.test/test_subagent_coverage.py: addsTestBatchReportsInFlight(arm/release/predicate semantics across every terminal path incl. the report-free cancelled-recovery arm) andtest_stranded_hold_past_deadline_is_forced(reaper force-flushes an aged hold with no report in flight); updatestest_closing_wave_not_forcedto arm the hold. Both sweep tests run the sync entry and the awaited form the reaper loop calls.test_the_report_body_releases_its_hold_without_its_taskpins the report body's ownfinallyrelease apart from the task done-callback.test/test_subagent_scale.py: retargets the subagent-done ordering guard at the extracted_report_terminal_guarded_impland arms the hold intest_closing_wave_is_not_force_flushed.Red before, green after (this round): with only the three arm guards reverted in place (
run.pywas_donetail and claimed-successfinally,pump.pyapproval-denied flip), the threeTestOnlyTheTransitionArmsTheHoldtests fail at their finalbatch_reports_in_flight(...) is Falseassert, after their mid-test asserts passed (the reap's report released the hold; the tail ran). Restored, they pass. Mutations: removing the report body's own release (terminal.py_report_terminal_implfinally) reddens onlytest_the_report_body_releases_its_hold_without_its_task; making the async sweep always skip an unpending wave reddenstest_stranded_hold_past_deadline_is_forced[async], and never skipping a held one reddenstest_closing_wave_not_forced[async].Determinism (testing-conventions D12) for the 8 new or changed node ids: 20/20 at
-n0; 10/10 at-n 2 --dist loadgroup --max-worker-restart=2besidetest_subagent_reap_race.pyandTestDigestHoldDeadline(-n 2, not 4: shared host); shuffled order, seed 19426, 8 passed. Every wait is on an event or a task (await_until), no sleeps. Scoped run:test_wave_digest_double_finalize.py,test_subagent_coverage.py,test_subagent_reap_race.py,test_subagent_stop_reason_consistency.pyand the twotest_subagent_scale.pywave classes, 447 passed at-n 2.Manual verification
N/A — unit coverage sufficient: the race window is driven deterministically through the real completion closure, so no timing-dependent manual repro is needed.
Related Issues
Fixes #8554
Pattern harvest
Rule candidate: review-prompt
Pattern: a terminal state flag (
done) that gates wave/batch accounting must be armed in the same synchronous block as the flip, never left to a later-scheduled task — a deferred arm leaves a window where a sibling observes the member as neither pending nor in flight. And only the flip that makes the transition arms: a late site re-arming an already-ended record outlives the release.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)