Skip to content

test(subagents): pin the pre-first-turn reaper coverage gap - #10902

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:test/subagent-prefirstturn-reaper-gap
Sep 15, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
chenmingwei23:test/subagent-prefirstturn-reaper-gap

Conversation

@chenmingwei23

Copy link
Copy Markdown
Contributor

Problem / Motivation

A subagent can sit in the starting phase for over an hour with nothing flagging it, and a blocking orchestrator waiting on such children looks stuck too (issue #6303). "starting" means a run is registered but has not advanced past turn 0. Two states in that phase are reaped by no fast watchdog and end only at the wall-clock reaper, whose default _default_timeout is 3 hours:

  1. A queued wave member is invisible to every reaper. Admission that staggers or exceeds the concurrency cap appends the member to _queue and never inserts it into _agents. The reaper's per-agent loop iterates _agents only; _is_startup_stalled returns False without _exec_started; and _sweep_stuck_waves continues past any wave that still holds a queued member. The only exit is _drain_queue, gated on a free slot, so a fan-out wider than the cap parks its tail in starting behind runners that are themselves only reaped at the wall clock.
  2. A registered run parked before _run_inner (_exec_started is None, e.g. awaiting a spawn approval that, per the code's own comment, has "a wait with no deadline of its own") is invisible to the 120s startup watchdog, and its stall badge is suppressed while awaiting approval. It is backstopped only by the wall clock.

This PR records those two states as executable characterization tests. It does not fix them: whether a queued or pre-execution run should carry an independent deadline measured from registration, and what expiry should do (reap, re-queue, or notify), is a policy decision about what starting and queued mean, and a wrong value would kill legitimate long-queued fan-out. That decision is left for a maintainer and is posted as a question on issue #6303.

Why it matters

A run that shows starting for an hour-plus with no stalled indicator is indistinguishable, to a user, from a hang. The orchestrator that blocks on it inherits the same appearance. The startup watchdog added for the post-_exec_started window does not reach either state above, so the observable symptom in #6303 persists on current main. A related issue, #6484 ("runs stay queued without starting a child ACP process"), was closed by #7299, which fixed only the reporting of approval-parked runs and not the queued-member no-deadline gap; without a recorded test, a reader concludes the class is handled when half of it is not.

What changed

One test file, test/test_subagent_startup_prefirstturn_gap.py, five tests, no production change:

  • Each test is named for the gap, and every assertion that encodes the defect carries a present-tense docstring line stating it is currently-observed behaviour that a fix is expected to change -- so closing the gap must edit these tests deliberately rather than pass them silently.
  • The assertions pin the mechanism proved by reading the code: a _queue entry is absent from _agents; _sweep_stuck_waves leaves a queued-member wave untouched and batch_members_pending stays True; _is_startup_stalled returns False when _exec_started is None; and, for a pre-execution run, the reaper's per-agent decision does not terminate it at any instant short of _default_timeout. The wall-clock test pins the absence of a shorter bound, not the 3-hour value.
  • One contrast test records the covered half (a run wedged after _exec_started on turn 0 with no pid is reaped fast) so the boundary between covered and uncovered is legible.

Three mechanisms ruled out (recorded so they are not re-walked):

  • Slot-count leak. Every path that registers a run in _agents releases its slot on a terminal outcome through a one-shot release token that also drains the queue, so the queue does drain -- just as slowly as the wall-clock-reaped runners ahead of it free slots.
  • In-execution handshake hang. A run wedged in memory delegation or the ACP handshake has already set _exec_started (its first statement in _run_inner), so the 120s startup watchdog covers it.
  • Broadcast activity-refresh (the Subagent idle-stall clock is refreshed by a co-tenant's roster broadcast, so idle time measures the wrong thing #4841 class). A stream event that refreshes activity cannot suppress a stalled badge for a queued member, because a queued member is never in the _agents loop that flags or badges at all.

Tests

test/test_subagent_startup_prefirstturn_gap.py, 5 passed, run single-file with a fixed single worker (-n0) against a scratch KIROCREW_HOME. Comment History Gate passes on the added lines (diff-scoped against origin/main). No production code changed, so no behavioural test needed re-running.

Refs #6303

@chenmingwei23
chenmingwei23 requested review from a team and iamwhatever September 14, 2026 23:34
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 14, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 14, 2026
@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed a7fb865777ab100d811df3eaf474750394ba90f5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a7fb865

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Opus 4.8 Review (fork) — ✅ no blocking findings

Reviewed a7fb865777ab100d811df3eaf474750394ba90f5 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a7fb865

@github-actions

github-actions Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — 🟡 CONCERNS

Design-level review of a7fb865777ab100d811df3eaf474750394ba90f5 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

The "must edit these tests deliberately" guarantee fails for the wall-clock test: it asserts against a hand-copied mirror of the reaper, which a real fix bypasses silently.

Watch

  • The description's central mechanism claim — "a change closing the gap has to edit these tests deliberately rather than silently pass them" — holds for the tests that call real code (_is_startup_stalled, _sweep_stuck_waves, _queue/_agents membership) but not for test_gap_pre_execution_run_has_no_bound_shorter_than_the_wall_clock, whose subject is the test-local _reaper_would_terminate ("Mirrors _MonitoringMixin._reaper_loop"). A fix adding a pre-execution deadline check inside _reaper_loop_impl would not flow into the mirror, so that test stays green with a stale "expected to change" docstring — exactly the silent pass the PR exists to prevent, on the gap it calls the sole backstop.
    Clears when: that test exercises the reaper's real per-agent decision path (or the mirror is asserted equivalent to it), or its docstring and the description drop the deliberate-edit guarantee for the wall-clock bound.

Suggestions

[DESIGN-REVIEWED] a7fb865

Record two states a subagent can sit in while shown as `starting` that
neither the fast startup watchdog nor the stuck-wave sweep reaps: a wave
member stranded in `_queue` (never in `_agents`, so no per-agent reaper
sees it, and the sweep skips any wave holding one), and a registered run
parked before `_run_inner` with `_exec_started is None` (the startup
watchdog keys on `_exec_started`, so its predicate is False). In both,
the only backstop is the wall-clock reaper.

The tests are named for the gap and each defect-encoding assertion is
marked in its docstring as currently-observed behaviour a fix is expected
to change, so closing the gap must edit these tests deliberately rather
than pass them silently. The wall-clock assertion pins the absence of any
shorter bound, not the timeout value.

This change pins the gap; it does not fix it. Whether a queued or
pre-execution run should carry an independent deadline measured from
registration, and what expiry should do, is left for a maintainer.

## Pattern harvest
Not generalizable: a repo-specific characterization test for one manager's
reaper coverage; no reusable rule.

Refs kirodotdev#6303
@chenmingwei23
chenmingwei23 force-pushed the test/subagent-prefirstturn-reaper-gap branch from 1ba40ef to a7fb865 Compare September 15, 2026 02:02
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention labels Sep 15, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) September 15, 2026 13:06

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (1 file). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: adds one new characterization test file under test/ pinning two uncovered pre-first-turn subagent reaper states (queued wave member absent from _agents, registered run with _exec_started None); no production file touched. CodeQL is not applicable on this fork PR (default-setup emits no check-run); SAST coverage is Semgrep only, latest run success with 0 annotations.

@iamwhatever
iamwhatever merged commit 54d6c57 into kirodotdev:main Sep 15, 2026
112 of 119 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Sep 15, 2026
amadsalmon pushed a commit to amadsalmon/KiroCrew that referenced this pull request Sep 22, 2026
…ev#10902)

Record two states a subagent can sit in while shown as `starting` that
neither the fast startup watchdog nor the stuck-wave sweep reaps: a wave
member stranded in `_queue` (never in `_agents`, so no per-agent reaper
sees it, and the sweep skips any wave holding one), and a registered run
parked before `_run_inner` with `_exec_started is None` (the startup
watchdog keys on `_exec_started`, so its predicate is False). In both,
the only backstop is the wall-clock reaper.

The tests are named for the gap and each defect-encoding assertion is
marked in its docstring as currently-observed behaviour a fix is expected
to change, so closing the gap must edit these tests deliberately rather
than pass them silently. The wall-clock assertion pins the absence of any
shorter bound, not the timeout value.

This change pins the gap; it does not fix it. Whether a queued or
pre-execution run should carry an independent deadline measured from
registration, and what expiry should do, is left for a maintainer.

## Pattern harvest
Not generalizable: a repo-specific characterization test for one manager's
reaper coverage; no reusable rule.

Refs kirodotdev#6303
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants