Repository navigation
fix: measure sub-agent durations on the monotonic clock - #18992
iamwhatever merged 2 commits into
Conversation
The sub-agent reaper measured a run's age, a start's progress and a continuable conversation's idle time against time.time(), so a step of the wall clock (an NTP correction, a manual change) reaped every healthy run or every start in progress, spared a stuck run, or released every continuable conversation at once. Each run records its registration, its execution start, its start-queue marks and waits, and its last activity on time.monotonic() beside the wall-clock stamps, and the reaper measures the run timeout and the startup watchdogs from the monotonic readings. The conversation registry holds monotonic readings; its rebuild after a restart turns each stored wall-clock stamp into an age, and a stamp in the future reads as age 0. The wall-clock stamps, their readers and state.json are unchanged.
A run cancelled from outside the manager is recovered once, and the recovery waits up to 60 s for a free slot. The wait measured its deadline on time.time(), so a forward step of the wall clock (an NTP step, a manual change, a resume from suspend) ended it on the next poll and failed the run as "cancelled (recovery failed)". The deadline and its poll now read time.monotonic(). The deadline is local to the wait, so nothing stored or shown changes.
e00036f to
2761330
Compare
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that every future write to Suggestions
[DESIGN-REVIEWED] 2761330 Description read[DESCRIPTION-READ] 1808d59e6de798401d0548b6c413bc929d49e52e5afc81324a5667230966d673 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The wall-clock fallback in the three startup checks and the two new helpers has no production caller. Every production caller now passes Not justified as shipped
What this change shipsInventory (7 items) — 5 justifiedIntent: a wall-clock step should never reap a healthy sub-agent, spare a stuck one, or release a live conversation. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 2761330 Description read[DESCRIPTION-READ] 41f8d8b0bb0bc86a9c9d6d76ae766223ad4dc0cfed858950fc5aa15253c28bdf That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
iamwhatever
left a comment
There was a problem hiding this comment.
This PR moves every age the reaper acts on onto time.monotonic(): run timeout, the three startup watchdogs, the conversation TTL and the cancel-recovery slot wait. A wall-clock step then cannot reap a healthy run, keep a stuck one alive or expire every continuable conversation. That fixes the root cause. I read all production files. The two places this kind of change usually breaks both hold. First, I grepped the head for every writer of last_activity, _exec_started and _gate_wait_started, and each one also sets its monotonic twin. Second, every writer of the _conversations registry now stores a monotonic value (including the restart path, which turns a stored wall-clock stamp into an age), and the sweep's only caller passes mono_now. No SubagentInfo is built with an old started=, so the _started_mono default is sound. Leaving suspend time out of the timeouts and TTL is declared in the description and is consistent across the paired stamps. Display elapsed stays on the wall clock, which is in scope. The new tests step the wall clock forward and back, and checks are green. Advisory: the First Principles items (the wall-clock fallback branches now have only test callers, and the spec line "The ordinary wall-clock deadline remains unchanged" is stale) have no reply yet. A short disposition, or deleting the dead branches, would close them. — crew-pr-reviewer (deep re-check)
Problem / Motivation
Goal: Every sub-agent age and deadline the reaper and its siblings act on is measured on the monotonic clock, so a wall-clock step neither ends a run early nor keeps a stuck one alive.
The sub-agent reaper (
subagent_manager/monitoring.py,_reaper_loop_impl) measures every age it acts on against the wall clock,now = time.time():elapsed = now - info.started, and a run past_default_timeoutis force-reaped.startedis atime.time()stamp._is_startup_stalledmeasuresnow - _exec_started - _start_queue_wait_ms;_start_queue_saturated_secsmeasuresnow - _gate_wait_started;_is_first_prompt_silentmeasuresnow - last_activity.All of those stamps are
time.time()readings._sweep_conversations_async(now)releases a continuable conversation whennow - last_used >= 6 h, andlast_usedis atime.time()stamp.The recovery of a run cancelled from outside the manager has the same flaw (
subagent_manager/cancellation.py,_schedule_cancel_recovery_impl). It waits up to_RECOVERY_SLOT_WAIT_SECS(60 s) for a free slot, withdeadline = time.time() + _RECOVERY_SLOT_WAIT_SECS, and pollstime.time() >= deadlineevery 0.25 s.A step of the wall clock (an NTP correction, a manual clock change, a VM restored from a snapshot) moves all of these ages at once.
Why it matters
spawn_continuethen answersconversation_gone.updated_at. A stamp in the future (the clock stepped back after it was written) kept a conversation past its TTL for the size of that step. The test uses 30 days.Not a goal
started,_exec_started,_gate_wait_started,_start_queue_wait_ms,last_activityandupdated_atkeep their wall-clock values for every reader. That includes the dashboard idle display (dashboard/ws.py), the stall flag, andstate.json. The monotonic readings are separate fields written beside them._exec_started_mono. A direct call keeps the wall-clock arithmetic, as before._maybe_flag_stall), a dashboard-only warning;The reaper still passes its wall-clock
nowto these, and none of them reaps a run.time.monotonic()does not advance while the host is suspended. Time a laptop spends asleep therefore no longer counts toward a run's timeout, a start's deadline or a conversation's TTL. After a gateway restart, a conversation's age comes from the stored wall-clock stamp, so downtime does count there.What changed (motivation → approach → change)
time.monotonic()twin, written beside the existing wall-clock stamp, and measure on the twin. A stored wall-clock stamp that re-enters memory is turned into an age.SubagentInfo._started_monorecords the run's registration, and the reaper measures the run timeout from it._exec_started_mono,_gate_wait_started_mono,_start_queue_wait_mono_msand_last_activity_monoare stamped, accumulated and cleared beside their wall-clock fields. That covers the execution start, the start-queue mark and grant, an adopted session, the activity writers, and the recovery-respawn reset.mono_nowto_is_startup_stalled,_start_queue_saturated_secsand_is_first_prompt_silent, and reports a start-up reap's elapsed time on the same clock.time.monotonic()readings, written by the keep registration inrun.pyand both promote paths. The reaper passes its monotonic reading to the sweep.now, copy it back on a promote rollback, test membership or pop it. None reads it as a wall-clock time.time.monotonic() - age, withage = max(0, time.time() - updated_at), so a stamp in the future reads as age 0. This is the same ceiling idea as fix(members): cap the monotone last_active_ts so a forward clock jump cannot pin a crewmate #15477.time.monotonic(). The deadline is a local variable, never stored or shown, soinfo.elapsed, the error text and the tombstone are unchanged.docs/system-specs/modules/subagent.mdnames the monotonic twins the reaper reads.Backwards compatibility
state.jsonand every wall-clock field keep their values.Tests
test/test_subagent_coverage.pydrives the real reaper loop with the clock stepped where it reads it, through thekiro_crew.subagentfacade. Every kill, release, sweep, prune and store call in a tick is a recorder, so nothing is killed or deleted. Each test waits until the reaper has completed a tick, bounded at 30 s, rather than sleeping for a fixed time.TestReaperClock: a forward step does not reap a healthy run, and a backward step still reaps a run past its timeout. Two no-step controls.TestConversationTtlClock: a forward step keeps a just-used conversation. After a restart, a stored stamp in the future reads as age 0 and the conversation is released one TTL later. Control: a stored age past the TTL is still released.TestStartupWatchdogClock: a forward step does not reap a run 5 s into its start. Controls: no step, and a start past its deadline is reaped.test/test_subagent_turn_resilience.py::test_cancel_recovery_slot_wait_survives_a_forward_wall_clock_stepdrives the real recovery: a run is cancelled from outside, the pool is kept full, and the wall clock steps forward 120 s once the slot wait has polled, while the monotonic clock stays real. The test waits on the slot wait's own clock reads, never a fixed time: the clock fake counts them, and after the step the test waits for two more polls. The wait must not end, and the run respawns once a slot frees.the slot wait ended on a wall-clock step: 'cancelled (recovery failed)'.test_cancel_recovery_failure_emits_done_and_delivers, whose patched 0.4 s give-up still fails fast.test_cancel_recovery_slot_wait_without_a_clock_step_gives_up_at_its_deadline. With no clock step and the pool kept full, the wait still ends at its deadline (patched to 0.4 s, and not before it), now read on the monotonic clock, and the run fails as "cancelled (recovery failed)", as before.test/test_continuable_followups.py: the rebuild test asserted the registry held the stored wall-clock stamp. With both clocks pinned, it now asserts that the newest record's age is carried across.test/test_subagent_startup_pressure.py: itstimedoubles had nomonotonic. They are completed with a real one through a_time_athelper. Their wall-clock assertions are unchanged.a 5-second-old run was reaped as timed out after a 2 h forward wall-clock stepa run 60 s past its timeout was not reaped after a 1 h backward wall-clock stepa conversation used a moment ago was released after a 7 h forward wall-clock stepa conversation whose stored stamp is 30 days in the future was not released one TTL after the restarta run 5 s into its start was reaped after a 2 h forward wall-clock steplast_activity: 7441 passed, 3 skipped, 3 failed. The 3 failures aretest/test_xdist_escaped_failure_guard.pycases, which fail the same way on 843abe3 alone (3 failed, 4 passed), so they predate this change.test_session.py, one of the 100, checks that every field a routing producer writes is classified for delivery routing;_exec_started_monois classified beside_exec_startedas not delivery state.Manual verification
Not run against a live gateway. The tests drive the real reaper loop and step only the clock it reads.
Screenshots / video
None; no UI change.
Related Issues
No issue found for these paths.
last_active_tsso a forward clock jump cannot pin a crewmate. The restart rule here uses the same ceiling.git merge-treereports one conflict this change adds, inmonitoring.py.elapsed = mono_now - info._exec_started_mono, againstinfo.timeout_secs or self._manager._default_timeout._exec_startedas a wall-clock time, and that field keeps its meaning here.test/test_subagent_coverage.py, so the conflict is end-of-file only.Pattern harvest
Rule candidate: an age or deadline that decides whether to kill, release or give up is measured on
time.monotonic(). A wall-clock stamp is kept for display and persistence only, and a stored one becomes an age when it re-enters memory.Other wall-clock durations, not changed here:
next_due_ts);Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement