Repository navigation
Conversation
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Before merge, confirm that nothing in What this change shipsInventory (6 items) — 6 justifiedIntent: a wall-clock step (NTP correction, VM restore) should no longer flush a fresh wave digest early, hold an expired one too long, or reconcile a wave at the wrong time. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] bb7267e Description read[DESCRIPTION-READ] 7b917f339e013c531038c12ea41b36d398ae69ff5ecf872fb6b8dfdecdf8772a That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] bb7267e Description read[DESCRIPTION-READ] 931427033212616d3bb48378d0139d2dc88612e31f8c6534798aab493a5fd1f5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
|
The new commit replaces the short fixed waits that the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects in test/test_subagent_wave_clock.py; it changes that file only. |
Two per-wave deadlines were stamped with time.time() and aged against the reaper's wall-clock now, so a wall-clock step moved them. The digest hold: the gateway stamps a held wave member with time.time() and the reaper compared that stamp with its own wall-clock now. A forward step of the hold deadline or more forced a partial digest out of a wave whose other members were still running, and a backward step kept finished results held past the deadline. The stuck-wave sweep: the three places that count a wave submission stamp its progress with time.time(), and the reaper compared that with its wall-clock now. A forward step of the window reconciled a submission that may still be in flight, which closes the wave early, and a backward step left a stuck wave unreconciled. Both stamps are now time.monotonic() readings, and the reaper passes its monotonic reading to both sweeps. Both are in-memory state of the running process, so no stored stamp is involved.
test/test_wait_discipline_ratchet.py counts each short fixed wait in a test file and lets a file's count only go down. test_subagent_wave_clock.py had two, against a baseline of zero: a 10 ms sleep poll that waits for two reaper ticks and a 5 s asyncio.wait_for on the reaper task. Wait for the ticks with kiro_crew.testing.wait.async_wait_until, cancel the reaper task in a finally so a failed wait leaves nothing running, and raise the wait_for ceiling to 30 s. The file counts zero again and the baseline is unchanged.
64579c9 to
bb7267e
Compare
iamwhatever
left a comment
There was a problem hiding this comment.
This PR moves the wave digest-hold stamp (_digest_held_at) and the wave-progress stamp (_batch_progress_ts) from time.time() to time.monotonic() and passes the reaper's existing mono_now to both sweeps, so a wall-clock step can no longer force a partial digest out early or keep a wave wedged; that fixes the root cause, which was two ends of one deadline reading different clocks. I grepped every reader and writer at the PR head: all three _batch_progress_ts writers (subagent.py:5544, admission/gate.py:325, waves.py:145) and the one _digest_held_at writer (slack/gateway.py:10098) are now monotonic, the only readers are the two sweeps (waves.py:222/242/313), and the other writes only reset to 0.0, so no mixed-clock comparison is left. Neither value is persisted across a restart, which matters because a monotonic reading means nothing in a new process. I checked the missing-key case too: it is safe because every _batch_submitted writer stamps progress in the same step. The new test_subagent_wave_clock.py drives the real hold closure, the spawn/lost-submission writers and the reaper loop with a stepped wall clock, in both directions, and its forward/backward cases fail on base. Checks are green and the Design and First Principles lanes PASS. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: A wall-clock step no longer moves a sub-agent wave's two in-memory deadlines: the digest hold and the stuck-wave window.
Digest hold. When one member of a wave finishes while its siblings still run, the gateway's completion consumer holds that member's result for the wave digest and stamps the hold (
_digest_held_atin_subagent_done,slack/gateway.py:10086). The reaper's sweep force-flushes a partial digest once the oldest hold isDIGEST_HOLD_SECS(120 s) old (subagent_manager/waves.py:330). The stamp wastime.time()and the sweep received the reaper's wall-clocknow, so a step between the two reads counted as elapsed time. A forward step of 120 s or more (an NTP correction, a manual change, a restored VM) flushed a hold seconds old while the rest of the wave still ran. A backward step made a hold older than the deadline read as younger, so its finished result stayed withheld for the length of the step on top of the deadline.Stuck wave. A wave whose submissions were counted but did not all arrive (a refused spawn, a lost submission) is force-reconciled once
_WAVE_STUCK_SECS(1800 s) pass with no progress. The three places that count a submission stamped progress withtime.time()(subagent.py:5420,subagent_manager/admission/gate.py:325,subagent_manager/waves.py:145), and the sweep got the wall-clocknow(subagent_manager/monitoring.py:1112; compared atsubagent_manager/waves.py:223). A forward step of 30 minutes reconciled a wave that had progressed seconds ago; a backward step left a wave past the window unreconciled.Why it matters
A partial digest the parent did not need splits one wave report in two and can prompt it to act before its other children finish; the backward case delays every finished sibling's result. An early reconcile counts a submission that has not arrived yet as a failed member, so the wave can close before that member lands, and the spec records what an early close costs: not one wrong message but a second digest for the same batch (
docs/system-specs/modules/subagent.md:2685). The backward case leaves a wave with a lost submission open, and its held results undelivered, for as long as the step.Not a goal
started,last_activity,_exec_started): run state, not wave state. fix: measure sub-agent durations on the monotonic clock #18992 covers the run timeout.What changed (motivation → approach → change)
now.time.monotonic()readings and the reaper passes its monotonic reading to both sweeps. Hold: the stamp in_subagent_done(slack/gateway.py:10098) and the sweep call in_reaper_loop_impl(subagent_manager/monitoring.py:1119). Wave: the three stamps above and the sweep call (subagent_manager/monitoring.py:1112). The reading is the loop's existingmono_now(subagent_manager/monitoring.py:1077), which the conversation sweep already takes (subagent_manager/monitoring.py:1123)._digest_held_atand_batch_progress_tsare the only wave-state timestamps; the rest is counts and sets (_batch_submitted,_seen_batches) and the gateway's per-wave counters and buffers._expired_digest_holdsreads the hold once (subagent_manager/waves.py:313,subagent_manager/waves.py:330); the two clears carry no clock (slack/gateway.py:10134,subagent_manager/terminal.py:360)._stuck_wave_candidatesreads the progress stamp (subagent_manager/waves.py:222),_reconcile_stuck_wavelogs its age (subagent_manager/waves.py:242) andfinalize_batch_implpops it on close (subagent_manager/waves.py:116). Every site that creates a_batch_submittedentry stamps in the same block, so the0.0default in_stuck_wave_candidates(subagent_manager/waves.py:222) never applies to a registered wave;0.0stays the "not held" value of the hold (subagent_manager/waves.py:313) and a live monotonic reading is positive.docs/system-specs/modules/subagent.mdnames the clock in the_digest_held_atrow, the hold-deadline bullet and the wave-liveness bullet. The source pintest_the_wave_reads_run_off_loop_for_every_coroutine_caller(test/test_overload_integration_glue.py:2541,test/test_overload_integration_glue.py:2542) namesmono_nowfor both sweeps.Backwards compatibility
Compatible: with no clock step both deadlines fire at the same age as before; nothing persisted changes.
Removes nothing: no field, key or function is removed or renamed; both sweeps keep their signatures, only the reaper's argument changes.
Tests
test/test_subagent_wave_clock.py(12 tests). Red onorigin/main(0a47021090, this file copied into a detached checkout),pytest test/test_subagent_wave_clock.py -p no:cacheprovider -q -n 0 --timeout=120, RC=1 in 3 of 3 runs:Green on this branch, the same command, RC=0 in 3 of 3 runs:
The 8 red tests: two digest-hold tests (a forward step must not flush a fresh hold; after a step back an expired hold must flush) and, for each of the three stamping sites, a forward step that must not reconcile a wave that just progressed and a step back after which a wave stuck past the window must still be reconciled. The 4 that pass on main are controls (no step, and a hold or wave really past its deadline). The hold is driven through the real gateway closure, each stamp through the real
spawn,spawn_asyncorrecord_lost_submission, and each sweep inside the real reaper loop; the flush, reconcile, kill, release, prune and store calls are recorders. The stepped clock replaces thetimebinding of the gateway module and of thekiro_crew.subagentfacade, through which the reaper and the rebound*_implmethods resolvetime.One mutation per rule, each in a throwaway checkout of this branch, fails only its own tests. Stamp in
record_lost_submissionback totime.time(): the[lost-submission]step-back test and the no-step control that drives it. Stamp inspawnback: only[refused-spawn]. Stamp inspawn_asyncback: only[unreadable-execution-record].nowpassed to the stuck-wave sweep: the three forward-step tests, the no-step control, and the source pintest_the_wave_reads_run_off_loop_for_every_coroutine_caller(test/test_overload_integration_glue.py:2541). Hold stamp back: the digest step-back test and the past-deadline control.nowpassed to the hold sweep: the digest forward-step test, its no-step control, and the pintest_the_wave_reads_run_off_loop_for_every_coroutine_calleragain (test/test_overload_integration_glue.py:2542).The 32 test files that name
_digest_held_at,_batch_progress_ts,_batch_submitted,record_lost_submission, the stuck-wave or hold sweeps or the reaper loop, plus the cron and Slack-gateway files that share the gateway test factory (test/test_cron_thread_routing.py),test/test_overload_integration_glue.py(whose source pins now readmono_now) andtest/test_flake_pattern_ratchet.py, run serially with-n 0on the final head: 3035 passed, exit 0. mypy on the five changed source files, flake8, isort, black on the changed files outside the baseline,scripts/check_black_formatting.py,scripts/check_harness_parity.py,scripts/docs-lint.sh,scripts/check_subprocess_encoding.py,scripts/check_sync_io_in_async.pyandCOMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.pyexit 0.Assertion lines of three red tests, from the first red run:
Manual verification
N/A: the clock step is driven through the
timemodule each side reads, against the real gateway closure, spawn path and reaper loop.Related Issues
No open issue covers either clock. Related: #18992 (merged), which moved the sub-agent durations to the monotonic clock and lists the digest-hold and stuck-wave sweeps under its Not a goal. Open #12821 adds a fourth writer of
_batch_progress_ts(register_externalin itssubagent.pyhunk) that stamps withtime.time(); if it lands after this change, that stamp needstime.monotonic()too.Pattern harvest
Rule candidate: review-prompt
Pattern: an in-memory deadline whose stamp and comparison both read
time.time().Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)