Skip to content

fix: age wave digest holds and stuck waves on the monotonic clock - #19367

Merged
javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/wave-digest-hold-uses-monotonic-clock
Oct 13, 2026
Merged

javenciu merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/wave-digest-hold-uses-monotonic-clock

Conversation

@javenciu

@javenciu javenciu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A wall-clock step no longer moves a sub-agent wave's two in-memory deadlines: the digest hold and the stuck-wave window.

Digest hold. When one member of a wave finishes while its siblings still run, the gateway's completion consumer holds that member's result for the wave digest and stamps the hold (_digest_held_at in _subagent_done, slack/gateway.py:10086). The reaper's sweep force-flushes a partial digest once the oldest hold is DIGEST_HOLD_SECS (120 s) old (subagent_manager/waves.py:330). The stamp was time.time() and the sweep received the reaper's wall-clock now, so a step between the two reads counted as elapsed time. A forward step of 120 s or more (an NTP correction, a manual change, a restored VM) flushed a hold seconds old while the rest of the wave still ran. A backward step made a hold older than the deadline read as younger, so its finished result stayed withheld for the length of the step on top of the deadline.

Stuck wave. A wave whose submissions were counted but did not all arrive (a refused spawn, a lost submission) is force-reconciled once _WAVE_STUCK_SECS (1800 s) pass with no progress. The three places that count a submission stamped progress with time.time() (subagent.py:5420, subagent_manager/admission/gate.py:325, subagent_manager/waves.py:145), and the sweep got the wall-clock now (subagent_manager/monitoring.py:1112; compared at subagent_manager/waves.py:223). A forward step of 30 minutes reconciled a wave that had progressed seconds ago; a backward step left a wave past the window unreconciled.

Why it matters

A partial digest the parent did not need splits one wave report in two and can prompt it to act before its other children finish; the backward case delays every finished sibling's result. An early reconcile counts a submission that has not arrived yet as a failed member, so the wave can close before that member lands, and the spec records what an early close costs: not one wrong message but a second digest for the same batch (docs/system-specs/modules/subagent.md:2685). The backward case leaves a wave with a lost submission open, and its held results undelivered, for as long as the step.

Not a goal

What changed (motivation → approach → change)

  • Symptom: after a forward wall-clock step a fresh hold was flushed, and a wave that had just progressed was reconciled; after a step back, a hold and a wave past their deadlines were left alone.
  • Root cause: each age was a wall-clock stamp measured against the reaper's wall-clock now.
  • Change: both stamps are time.monotonic() readings and the reaper passes its monotonic reading to both sweeps. Hold: the stamp in _subagent_done (slack/gateway.py:10098) and the sweep call in _reaper_loop_impl (subagent_manager/monitoring.py:1119). Wave: the three stamps above and the sweep call (subagent_manager/monitoring.py:1112). The reading is the loop's existing mono_now (subagent_manager/monitoring.py:1077), which the conversation sweep already takes (subagent_manager/monitoring.py:1123).
  • Same pattern elsewhere in the wave family: none. _digest_held_at and _batch_progress_ts are the only wave-state timestamps; the rest is counts and sets (_batch_submitted, _seen_batches) and the gateway's per-wave counters and buffers.
  • Readers: _expired_digest_holds reads the hold once (subagent_manager/waves.py:313, subagent_manager/waves.py:330); the two clears carry no clock (slack/gateway.py:10134, subagent_manager/terminal.py:360). _stuck_wave_candidates reads the progress stamp (subagent_manager/waves.py:222), _reconcile_stuck_wave logs its age (subagent_manager/waves.py:242) and finalize_batch_impl pops it on close (subagent_manager/waves.py:116). Every site that creates a _batch_submitted entry stamps in the same block, so the 0.0 default in _stuck_wave_candidates (subagent_manager/waves.py:222) never applies to a registered wave; 0.0 stays the "not held" value of the hold (subagent_manager/waves.py:313) and a live monotonic reading is positive.
  • Docs: docs/system-specs/modules/subagent.md names the clock in the _digest_held_at row, the hold-deadline bullet and the wave-liveness bullet. The source pin test_the_wave_reads_run_off_loop_for_every_coroutine_caller (test/test_overload_integration_glue.py:2541, test/test_overload_integration_glue.py:2542) names mono_now for both sweeps.

Backwards compatibility

Compatible: with no clock step both deadlines fire at the same age as before; nothing persisted changes.
Removes nothing: no field, key or function is removed or renamed; both sweeps keep their signatures, only the reaper's argument changes.

Tests

test/test_subagent_wave_clock.py (12 tests). Red on origin/main (0a47021090, this file copied into a detached checkout), pytest test/test_subagent_wave_clock.py -p no:cacheprovider -q -n 0 --timeout=120, RC=1 in 3 of 3 runs:

==================== 8 failed, 4 passed, 1 warning in 3.94s ====================

Green on this branch, the same command, RC=0 in 3 of 3 runs:

======================== 12 passed, 1 warning in 3.88s =========================

The 8 red tests: two digest-hold tests (a forward step must not flush a fresh hold; after a step back an expired hold must flush) and, for each of the three stamping sites, a forward step that must not reconcile a wave that just progressed and a step back after which a wave stuck past the window must still be reconciled. The 4 that pass on main are controls (no step, and a hold or wave really past its deadline). The hold is driven through the real gateway closure, each stamp through the real spawn, spawn_async or record_lost_submission, and each sweep inside the real reaper loop; the flush, reconcile, kill, release, prune and store calls are recorders. The stepped clock replaces the time binding of the gateway module and of the kiro_crew.subagent facade, through which the reaper and the rebound *_impl methods resolve time.

One mutation per rule, each in a throwaway checkout of this branch, fails only its own tests. Stamp in record_lost_submission back to time.time(): the [lost-submission] step-back test and the no-step control that drives it. Stamp in spawn back: only [refused-spawn]. Stamp in spawn_async back: only [unreadable-execution-record]. now passed to the stuck-wave sweep: the three forward-step tests, the no-step control, and the source pin test_the_wave_reads_run_off_loop_for_every_coroutine_caller (test/test_overload_integration_glue.py:2541). Hold stamp back: the digest step-back test and the past-deadline control. now passed to the hold sweep: the digest forward-step test, its no-step control, and the pin test_the_wave_reads_run_off_loop_for_every_coroutine_caller again (test/test_overload_integration_glue.py:2542).

The 32 test files that name _digest_held_at, _batch_progress_ts, _batch_submitted, record_lost_submission, the stuck-wave or hold sweeps or the reaper loop, plus the cron and Slack-gateway files that share the gateway test factory (test/test_cron_thread_routing.py), test/test_overload_integration_glue.py (whose source pins now read mono_now) and test/test_flake_pattern_ratchet.py, run serially with -n 0 on the final head: 3035 passed, exit 0. mypy on the five changed source files, flake8, isort, black on the changed files outside the baseline, scripts/check_black_formatting.py, scripts/check_harness_parity.py, scripts/docs-lint.sh, scripts/check_subprocess_encoding.py, scripts/check_sync_io_in_async.py and COMMENT_HISTORY_BASE_REF=origin/main scripts/check_comment_history.py exit 0.

Assertion lines of three red tests, from the first red run:

E       AssertionError: a fresh hold was force-flushed after a wall-clock step: call('wave-1', 'dashboard:chat-7', 3, 600.0108158588409)
E       AssertionError: a wave that just progressed was reconciled: call('wave-1', 3, 'submission never arrived (wave stuck > 1800s
E       AssertionError: a stuck wave was not reconciled (lost-submission)

Manual verification

N/A: the clock step is driven through the time module each side reads, against the real gateway closure, spawn path and reaper loop.

Related Issues

No open issue covers either clock. Related: #18992 (merged), which moved the sub-agent durations to the monotonic clock and lists the digest-hold and stuck-wave sweeps under its Not a goal. Open #12821 adds a fourth writer of _batch_progress_ts (register_external in its subagent.py hunk) that stamps with time.time(); if it lands after this change, that stamp needs time.monotonic() too.

Pattern harvest

Rule candidate: review-prompt
Pattern: an in-memory deadline whose stamp and comparison both read time.time().

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 09:40
@javenciu
javenciu requested a review from cruisercohen October 12, 2026 09:40
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of bb7267eb480e27e1b9692d61fe668bb0fbf48c00 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Before merge, confirm that nothing in src/ calls the sync _sweep_stuck_waves(now) or _sweep_digest_holds(now) with a wall-clock reading. Only the facade defines them; the reaper is the only caller.

What this change ships

Inventory (6 items) — 6 justified

Intent: a wall-clock step (NTP correction, VM restore) should no longer flush a fresh wave digest early, hold an expired one too long, or reconcile a wave at the wrong time. This is a FIX.

  1. A digest hold is stamped on the monotonic clock and the reaper's hold sweep ages it against mono_now — justified
  2. A stuck wave's progress is stamped on the monotonic clock at all three submission-counting sites (subagent.py, admission/gate.py, waves.py) — justified
  3. The reaper passes mono_now, which it already computes, to the stuck-wave sweep — justified
  4. Spec rows and bullets in subagent.md name the clock, as AGENTS.md requires in the same commit — justified
  5. The source pin in test_overload_integration_glue.py now reads mono_now — justified
  6. New test_subagent_wave_clock.py: 8 tests fail on base, 4 are controls — justified

[FIRST-PRINCIPLES-REVIEWED] bb7267e

Description read

[DESCRIPTION-READ] 7b917f339e013c531038c12ea41b36d398ae69ff5ecf872fb6b8dfdecdf8772a

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of bb7267eb480e27e1b9692d61fe668bb0fbf48c00 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] bb7267e

Description read

[DESCRIPTION-READ] 931427033212616d3bb48378d0139d2dc88612e31f8c6534798aab493a5fd1f5

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision bb7267eb480e27e1b9692d61fe668bb0fbf48c00 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed bb7267eb480e27e1b9692d61fe668bb0fbf48c00 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] bb7267e

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed bb7267eb480e27e1b9692d61fe668bb0fbf48c00 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] bb7267e

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu

Copy link
Copy Markdown
Contributor Author

The new commit replaces the short fixed waits that the repo's wait-discipline ratchet (test/test_wait_discipline_ratchet.py) rejects in test/test_subagent_wave_clock.py; it changes that file only.
The file counted 2 short fixed waits against a baseline of 0 and counts 0 again (the reaper poll moved to async_wait_until, the 5 s wait_for raised to 30 s); the ratchet passes on the head merged with main (c4c51aa), and the file passes 3 of 3 runs (12 tests each).

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
Two per-wave deadlines were stamped with time.time() and aged against
the reaper's wall-clock now, so a wall-clock step moved them.

The digest hold: the gateway stamps a held wave member with
time.time() and the reaper compared that stamp with its own
wall-clock now. A forward step of the hold deadline or more forced a
partial digest out of a wave whose other members were still running,
and a backward step kept finished results held past the deadline.

The stuck-wave sweep: the three places that count a wave submission
stamp its progress with time.time(), and the reaper compared that
with its wall-clock now. A forward step of the window reconciled a
submission that may still be in flight, which closes the wave early,
and a backward step left a stuck wave unreconciled.

Both stamps are now time.monotonic() readings, and the reaper passes
its monotonic reading to both sweeps. Both are in-memory state of the
running process, so no stored stamp is involved.
test/test_wait_discipline_ratchet.py counts each short fixed wait in a test
file and lets a file's count only go down. test_subagent_wave_clock.py had two,
against a baseline of zero: a 10 ms sleep poll that waits for two reaper ticks
and a 5 s asyncio.wait_for on the reaper task.

Wait for the ticks with kiro_crew.testing.wait.async_wait_until, cancel the
reaper task in a finally so a failed wait leaves nothing running, and raise the
wait_for ceiling to 30 s. The file counts zero again and the baseline is
unchanged.
@javenciu
javenciu force-pushed the fix/wave-digest-hold-uses-monotonic-clock branch from 64579c9 to bb7267e Compare October 12, 2026 20:03
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@javenciu
javenciu enabled auto-merge (squash) October 12, 2026 23:49

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR moves the wave digest-hold stamp (_digest_held_at) and the wave-progress stamp (_batch_progress_ts) from time.time() to time.monotonic() and passes the reaper's existing mono_now to both sweeps, so a wall-clock step can no longer force a partial digest out early or keep a wave wedged; that fixes the root cause, which was two ends of one deadline reading different clocks. I grepped every reader and writer at the PR head: all three _batch_progress_ts writers (subagent.py:5544, admission/gate.py:325, waves.py:145) and the one _digest_held_at writer (slack/gateway.py:10098) are now monotonic, the only readers are the two sweeps (waves.py:222/242/313), and the other writes only reset to 0.0, so no mixed-clock comparison is left. Neither value is persisted across a restart, which matters because a monotonic reading means nothing in a new process. I checked the missing-key case too: it is safe because every _batch_submitted writer stamps progress in the same step. The new test_subagent_wave_clock.py drives the real hold closure, the spawn/lost-submission writers and the reaper loop with a stepped wall clock, in both directions, and its forward/backward cases fail on base. Checks are green and the Design and First Principles lanes PASS. — crew-pr-reviewer (deep re-check)

@javenciu
javenciu merged commit 1c9ee3c into kirodotdev:main Oct 13, 2026
152 of 161 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants