Skip to content

fix: measure sub-agent durations on the monotonic clock - #18992

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/subagent-durations-use-monotonic-clock
Oct 11, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/subagent-durations-use-monotonic-clock

Conversation

@javenciu

@javenciu javenciu commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Every sub-agent age and deadline the reaper and its siblings act on is measured on the monotonic clock, so a wall-clock step neither ends a run early nor keeps a stuck one alive.

The sub-agent reaper (subagent_manager/monitoring.py, _reaper_loop_impl) measures every age it acts on against the wall clock, now = time.time():

  • Run timeout: elapsed = now - info.started, and a run past _default_timeout is force-reaped. started is a time.time() stamp.
  • Startup watchdogs:
    • _is_startup_stalled measures now - _exec_started - _start_queue_wait_ms;
    • _start_queue_saturated_secs measures now - _gate_wait_started;
    • _is_first_prompt_silent measures now - last_activity.
      All of those stamps are time.time() readings.
  • Conversation TTL: _sweep_conversations_async(now) releases a continuable conversation when now - last_used >= 6 h, and last_used is a time.time() stamp.

The recovery of a run cancelled from outside the manager has the same flaw (subagent_manager/cancellation.py, _schedule_cancel_recovery_impl). It waits up to _RECOVERY_SLOT_WAIT_SECS (60 s) for a free slot, with deadline = time.time() + _RECOVERY_SLOT_WAIT_SECS, and polls time.time() >= deadline every 0.25 s.

A step of the wall clock (an NTP correction, a manual clock change, a VM restored from a snapshot) moves all of these ages at once.

Why it matters

  • Forward step longer than the run timeout: every running sub-agent is force-reaped as timed out on the next reaper tick.
  • Forward step longer than the startup deadline (120 s): every run still starting is reaped as failed to start.
  • Backward step: a stuck run outlives its timeout by the size of the step.
  • Forward step of 6 h or more: every continuable conversation is released, which deletes its saved session. spawn_continue then answers conversation_gone.
  • After a restart: the TTL registry is rebuilt from the stored updated_at. A stamp in the future (the clock stepped back after it was written) kept a conversation past its TTL for the size of that step. The test uses 30 days.
  • Forward step of 60 s or more while a recovery waits for a slot: the wait ends on its next poll, and the run fails as "cancelled (recovery failed)" instead of being re-run once a slot frees. The test measures 0.16 s from the step to the failure.

Not a goal

  • No wall-clock field changes meaning. started, _exec_started, _gate_wait_started, _start_queue_wait_ms, last_activity and updated_at keep their wall-clock values for every reader. That includes the dashboard idle display (dashboard/ws.py), the stall flag, and state.json. The monotonic readings are separate fields written beside them.
  • Callers outside the reaper keep the wall-clock path. The three startup checks measure on the monotonic fields only when the reaper passes its monotonic reading and the run carries _exec_started_mono. A direct call keeps the wall-clock arithmetic, as before.
  • Not changed:
    • the stall flag (_maybe_flag_stall), a dashboard-only warning;
    • the digest-hold and stuck-wave sweeps.
      The reaper still passes its wall-clock now to these, and none of them reaps a run.
  • Suspend: on Linux and macOS, time.monotonic() does not advance while the host is suspended. Time a laptop spends asleep therefore no longer counts toward a run's timeout, a start's deadline or a conversation's TTL. After a gateway restart, a conversation's age comes from the stored wall-clock stamp, so downtime does count there.
  • Windows: gateway is killed 27-28 s after every resume from sleep (no suspend handling in the liveness monitor or the Windows watchdog path) #18173 (Windows: the gateway is killed 27-28 s after every resume from sleep) is about the liveness watchdog, a different path, and is not addressed here.

What changed (motivation → approach → change)

  • Motivation: the reaper's ages must not move when the wall clock is stepped.
  • Approach: give each duration the reaper acts on a time.monotonic() twin, written beside the existing wall-clock stamp, and measure on the twin. A stored wall-clock stamp that re-enters memory is turned into an age.
  • Change:
    • Run timeout: SubagentInfo._started_mono records the run's registration, and the reaper measures the run timeout from it.
    • Startup watchdogs:
      • _exec_started_mono, _gate_wait_started_mono, _start_queue_wait_mono_ms and _last_activity_mono are stamped, accumulated and cleared beside their wall-clock fields. That covers the execution start, the start-queue mark and grant, an adopted session, the activity writers, and the recovery-respawn reset.
      • The reaper passes mono_now to _is_startup_stalled, _start_queue_saturated_secs and _is_first_prompt_silent, and reports a start-up reap's elapsed time on the same clock.
      • The start span and the queue pause subtracted from it come from one clock on either path.
    • Conversation registry:
      • The registry holds time.monotonic() readings, written by the keep registration in run.py and both promote paths. The reaper passes its monotonic reading to the sweep.
      • Its readers only compare a value with the sweep's now, copy it back on a promote rollback, test membership or pop it. None reads it as a wall-clock time.
      • The rebuild after a restart seeds each conversation at time.monotonic() - age, with age = max(0, time.time() - updated_at), so a stamp in the future reads as age 0. This is the same ceiling idea as fix(members): cap the monotone last_active_ts so a forward clock jump cannot pin a crewmate #15477.
    • Recovery slot wait (second commit): the deadline and its poll read time.monotonic(). The deadline is a local variable, never stored or shown, so info.elapsed, the error text and the tombstone are unchanged.
    • Spec: the startup-watchdog bullet in docs/system-specs/modules/subagent.md names the monotonic twins the reaper reads.

Backwards compatibility

  • Nothing is persisted differently. state.json and every wall-clock field keep their values.
  • The registry exists only in memory. Its rebuild after a restart converts the stored stamps as described above.

Tests

  • test/test_subagent_coverage.py drives the real reaper loop with the clock stepped where it reads it, through the kiro_crew.subagent facade. Every kill, release, sweep, prune and store call in a tick is a recorder, so nothing is killed or deleted. Each test waits until the reaper has completed a tick, bounded at 30 s, rather than sleeping for a fixed time.
    • TestReaperClock: a forward step does not reap a healthy run, and a backward step still reaps a run past its timeout. Two no-step controls.
    • TestConversationTtlClock: a forward step keeps a just-used conversation. After a restart, a stored stamp in the future reads as age 0 and the conversation is released one TTL later. Control: a stored age past the TTL is still released.
    • TestStartupWatchdogClock: a forward step does not reap a run 5 s into its start. Controls: no step, and a start past its deadline is reaped.
  • test/test_subagent_turn_resilience.py::test_cancel_recovery_slot_wait_survives_a_forward_wall_clock_step drives the real recovery: a run is cancelled from outside, the pool is kept full, and the wall clock steps forward 120 s once the slot wait has polled, while the monotonic clock stays real. The test waits on the slot wait's own clock reads, never a fixed time: the clock fake counts them, and after the step the test waits for two more polls. The wait must not end, and the run respawns once a slot frees.
    • Red on the first commit, in each of 3 runs (1 failed, 1 passed, the control below): the slot wait ended on a wall-clock step: 'cancelled (recovery failed)'.
    • Green on the second. The neighbouring recovery tests pass unchanged, including test_cancel_recovery_failure_emits_done_and_delivers, whose patched 0.4 s give-up still fails fast.
    • Control: test_cancel_recovery_slot_wait_without_a_clock_step_gives_up_at_its_deadline. With no clock step and the pool kept full, the wait still ends at its deadline (patched to 0.4 s, and not before it), now read on the monotonic clock, and the run fails as "cancelled (recovery failed)", as before.
  • Existing tests changed:
    • test/test_continuable_followups.py: the rebuild test asserted the registry held the stored wall-clock stamp. With both clocks pinned, it now asserts that the newest record's age is carried across.
    • test/test_subagent_startup_pressure.py: its time doubles had no monotonic. They are completed with a real one through a _time_at helper. Their wall-clock assertions are unchanged.
    • On main (843abe3), the two changed files give 83 passed and 1 failed. The failure is the rebuild test, which asserts the monotonic registry and so fails there by design.
  • Red on main (843abe3, new tests copied in): 5 failed, 5 passed (the controls), in each of 3 runs. Failures:
    • a 5-second-old run was reaped as timed out after a 2 h forward wall-clock step
    • a run 60 s past its timeout was not reaped after a 1 h backward wall-clock step
    • a conversation used a moment ago was released after a 7 h forward wall-clock step
    • a conversation whose stored stamp is 30 days in the future was not released one TTL after the restart
    • a run 5 s into its start was reaped after a 2 h forward wall-clock step
  • Green, both commits, on 843abe3:
    • the four new and changed test files: 413 passed, in each of 3 runs;
    • the 100 test files that exercise the reaper, the continuation registry, run.py's start-up path and last_activity: 7441 passed, 3 skipped, 3 failed. The 3 failures are test/test_xdist_escaped_failure_guard.py cases, which fail the same way on 843abe3 alone (3 failed, 4 passed), so they predate this change.
    • test_session.py, one of the 100, checks that every field a routing producer writes is classified for delivery routing; _exec_started_mono is classified beside _exec_started as not delivery state.

Manual verification

Not run against a live gateway. The tests drive the real reaper loop and step only the clock it reads.

Screenshots / video

None; no UI change.

Related Issues

No issue found for these paths.

Pattern harvest

Rule candidate: an age or deadline that decides whether to kill, release or give up is measured on time.monotonic(). A wall-clock stamp is kept for display and persistence only, and a stored one becomes an age when it re-enters memory.

Other wall-clock durations, not changed here:

  • the stall flag;
  • the digest-hold and stuck-wave sweeps;
  • the autonudge loop and monitor deadlines (next_due_ts);
  • the task-runner watchdog;
  • the chat-turn heartbeat broadcast.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 11, 2026 00:44
@javenciu
javenciu requested a review from buluoray October 11, 2026 00:44
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention labels Oct 11, 2026
The sub-agent reaper measured a run's age, a start's progress and a
continuable conversation's idle time against time.time(), so a step of
the wall clock (an NTP correction, a manual change) reaped every healthy
run or every start in progress, spared a stuck run, or released every
continuable conversation at once.

Each run records its registration, its execution start, its start-queue
marks and waits, and its last activity on time.monotonic() beside the
wall-clock stamps, and the reaper measures the run timeout and the
startup watchdogs from the monotonic readings. The conversation registry
holds monotonic readings; its rebuild after a restart turns each stored
wall-clock stamp into an age, and a stamp in the future reads as age 0.
The wall-clock stamps, their readers and state.json are unchanged.
A run cancelled from outside the manager is recovered once, and the
recovery waits up to 60 s for a free slot. The wait measured its
deadline on time.time(), so a forward step of the wall clock (an NTP
step, a manual change, a resume from suspend) ended it on the next
poll and failed the run as "cancelled (recovery failed)".

The deadline and its poll now read time.monotonic(). The deadline is
local to the wait, so nothing stored or shown changes.
@javenciu
javenciu force-pushed the fix/subagent-durations-use-monotonic-clock branch from e00036f to 2761330 Compare October 11, 2026 10:47
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 276133088779619c6d390a11f59b81e4248b8425 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 276133088779619c6d390a11f59b81e4248b8425 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that every future write to last_activity also writes _last_activity_mono. If one is missed, the first-prompt-silent watchdog can reap a run that was active.

Suggestions

  • Put each wall-clock/monotonic pair behind one setter, such as _touch_activity(info) and _mark_exec_start(info). The twins are currently written by hand at six or more sites, and the setters would stop a future writer from updating only one clock.

[DESIGN-REVIEWED] 2761330

Description read

[DESCRIPTION-READ] 1808d59e6de798401d0548b6c413bc929d49e52e5afc81324a5667230966d673

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 276133088779619c6d390a11f59b81e4248b8425 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

The wall-clock fallback in the three startup checks and the two new helpers has no production caller. Every production caller now passes mono_now.

Not justified as shipped

  • Item 7 — one consumer, generalized: "A direct call keeps the wall-clock arithmetic." I grepped _is_startup_stalled(, _is_first_prompt_silent( and _start_queue_saturated_secs( in src/: 3 call sites, all in _reaper_loop_impl, and all pass mono_now after this diff. All 3 writers of _exec_started (run.py:2005, run.py:4434, cancellation.py:155) also set _exec_started_mono. That leaves the None branches with only test callers.
    Subtraction: remove the mono_now: float | None = None parameters and the wall-clock branches of _idle_since_activity and _startup_elapsed, and have the reaper pass one monotonic now.
    Clears when: a non-test caller that needs the wall-clock path is named, or the branches are removed.
  • Item 6 — undeclared in spec: the PR says suspend time "no longer counts toward a run's timeout", but subagent.md still says "The ordinary wall-clock deadline remains unchanged." The PR also states the clock behaviour only for Linux and macOS, not Windows.
    Clears when: a human confirms that leaving suspend time out of the run timeout is intended on all three platforms.

What this change ships

Inventory (7 items) — 5 justified

Intent: a wall-clock step should never reap a healthy sub-agent, spare a stuck one, or release a live conversation. This is a FIX.

  1. A forward wall-clock step no longer force-reaps a running sub-agent as timed out — justified
  2. A wall-clock step no longer reaps a start in progress through the three startup watchdogs — justified
  3. A forward step no longer releases continuable conversations, because the TTL registry is now monotonic — justified
  4. After a restart, a stored stamp in the future reads as age 0 — justified
  5. The cancel-recovery slot wait survives a wall-clock step (red test on base) — justified
  6. Laptop sleep no longer counts toward the run timeout, startup deadline or conversation TTL — undeclared in spec; Windows behaviour not stated
  7. The three checks keep a wall-clock path for direct callers — one consumer, generalized

[FIRST-PRINCIPLES-REVIEWED] 2761330

Description read

[DESCRIPTION-READ] 41f8d8b0bb0bc86a9c9d6d76ae766223ad4dc0cfed858950fc5aa15253c28bdf

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 276133088779619c6d390a11f59b81e4248b8425 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 2761330

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 276133088779619c6d390a11f59b81e4248b8425 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 2761330

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 11, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR moves every age the reaper acts on onto time.monotonic(): run timeout, the three startup watchdogs, the conversation TTL and the cancel-recovery slot wait. A wall-clock step then cannot reap a healthy run, keep a stuck one alive or expire every continuable conversation. That fixes the root cause. I read all production files. The two places this kind of change usually breaks both hold. First, I grepped the head for every writer of last_activity, _exec_started and _gate_wait_started, and each one also sets its monotonic twin. Second, every writer of the _conversations registry now stores a monotonic value (including the restart path, which turns a stored wall-clock stamp into an age), and the sweep's only caller passes mono_now. No SubagentInfo is built with an old started=, so the _started_mono default is sound. Leaving suspend time out of the timeouts and TTL is declared in the description and is consistent across the paired stamps. Display elapsed stays on the wall clock, which is in scope. The new tests step the wall clock forward and back, and checks are green. Advisory: the First Principles items (the wall-clock fallback branches now have only test callers, and the spec line "The ordinary wall-clock deadline remains unchanged" is stale) have no reply yet. A short disposition, or deleting the dead branches, would close them. — crew-pr-reviewer (deep re-check)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants