Skip to content

fix(session): idle sweep keeps a parent whose sub-agents are still running - #11734

Merged
iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/idle-sweep-attached-subagents
Sep 25, 2026
Merged

iamwhatever merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/idle-sweep-attached-subagents

Conversation

@jeeshofone

@jeeshofone jeeshofone commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

An idle parent can look finished while its sub-agents are still running. The parent’s own semaphore is free after its turn, so the cleanup sweep could expire the runtime and lose the children’s work. A broken sub-agent probe also kept sessions fail-closed but only logged at debug level, which could hide a system-wide cleanup stall.

Why it matters

Background sub-agents are expected to keep working after the parent turn ends. Closing the tab or waiting past timeout_secs must not reset that shared runtime. Operators also need a bounded signal when the safety probe cannot answer and idle, orphan, and RSS cleanup start retaining candidates.

What changed (motivation → approach → change)

src/kiro_crew/session_cleanup.py asks the existing attached-sub-agent probe on both idle-sweep axes before any expiry side effect. After the await, it synchronously rechecks the injection counter, scanned incarnation, semaphore, and the branch-specific liveness condition. The final reset stays pinned to the scanned session. If a live slot reopens and installs a replacement during the probe, its freshly published ownership record survives the stale orphan verdict; a key that remains absent still drops the departed incarnation’s record.

Probe failures still fail closed. CleanupState.probe_failure_warned_at now limits the visible warning to one line per hour across all keys, while each traceback remains available at debug level. docs/system-specs/modules/session.md now describes both axes, the four post-await checks, the live-slot ownership exception, and this bounded visibility contract.

Case Before After
Child sub-agent is running 🟥 Parent could be expired 🟩 Parent is kept
Session changes during the probe 🟥 Old verdict could reach a new runtime 🟩 Incarnation and liveness are rechecked
Slot reopens with a replacement 🟥 Fresh ownership claim could be erased 🟩 Live slot keeps its claim
Probe raises 🟥 Cleanup retention is debug-only 🟨 Cleanup stays fail-closed and warns hourly
Childless idle session 🟦 Expires 🟦 Still expires

🟩 added · 🟨 changed · 🟥 removed · 🟦 unchanged

Backwards compatibility

Compatible: no accepted input or public API changes. Childless idle sessions still expire, closed slots still release stale claims, and probe failures keep the existing fail-closed behavior.

Tests

test/test_session_idle_attached_subagents.py covers attached children on idle and orphan axes, synchronous and async probes, probe exceptions, replacement incarnations, turns that begin or complete during the await, reopened-slot ownership followed by later orphan expiry, a permit taken inside the probe with a stale clock, normal childless expiry, one warning across keys and sweeps, hourly warning repetition, and quiet successful probes.

test/test_consolidate_cmd.py and test/test_session_coverage.py keep the pinned-reset call contract current. The post-rebase cleanup set passes with 95 tests and 17 platform skips. Full mypy passes 1,706 source files, and all formatting, history, brand, SDK, async, changelog, docs, diff, and single-commit gates pass. Two warning regressions fail at pytest call phase on old head d74a8be42; the quiet-probe control passes there. The reopened-slot regression fails at pytest call phase on reviewed head 5652a6a2d. On head 2458e85d9 (rebased onto origin/main c66f95b1c; since the patch-identical 34b106b70 it adds two wording fixes and no code change: the src/kiro_crew/session.py module map now says the sub-agent probe is asked on both idle-sweep axes and only the live-set re-assert is orphan-only, and the probe-failure contract in session_cleanup.py and session.md now names only a probe that raises, since an unreadable task store is absorbed as taskq_bridge.UNKNOWN_PENDING one layer down and never reaches the warning): test_a_held_permit_alone_stops_the_idle_expiry pins the idle-axis permit re-read directly (disabling that read fails it and its orphan twin); 251 passed across the session cleanup, idle, RSS, coverage, consolidate and injection test files; the black gate, isort, flake8, mypy (session.py, session_cleanup.py) and docs lint pass. On head a30e2f505 (same base; adds one test and no code change): test_an_injection_that_begins_inside_the_probe_stops_the_idle_expiry pins the injection-counter re-read on the idle axis, where an injection that commits during the probe leaves the permit free and last_used stale; narrowing that re-read to the orphan axis fails this test and no other in the idle and reap test files. 183 passed across the idle, reap, coverage and consolidate test files; the black gate, isort, flake8, comment-history, SDK-boundary and sync-IO gates pass. On head 4795670da (patch-identical to a30e2f505 per git range-diff, rebased onto origin/main 3f16157ba to pick up the Backend Lint black-step fix from #13698): 225 passed across the idle, busy-guard, cleanup, RSS, coverage and consolidate test files; the black gate, isort, flake8, mypy (session.py, session_cleanup.py), docs lint, comment-history, SDK-boundary and sync-IO gates pass.

Manual verification

N/A — unit coverage drives the real SessionManager._expire_idle cleanup wiring and directly exercises every changed branch.

Screenshots / video

N/A — no user-visible UI changes.

Related Issues

Closes #11709

Pattern harvest

Rule candidate: review-prompt
Pattern: a reset path that checks the parent semaphore must also check attached child work and re-judge liveness after an awaited probe.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@jeeshofone
jeeshofone requested review from a team and cixuuz September 18, 2026 08:09
@github-actions github-actions Bot added the fork Pull request from a fork (external contributor) label Sep 18, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 18, 2026
@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

GPT 5.6 Review (fork) — ✅ no blocking findings

Reviewed 4795670dac9b98140db0d84f784fd7ae691e6639 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 4795670

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Fable 5, fork) — ✅ PASS

Premise-level review of 4795670dac9b98140db0d84f784fd7ae691e6639 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

The bounded probe-failure WARNING is a second fix riding under the title's fix — confirm linked issue #11709 covers that stall-visibility half too.

Not justified as shipped

  • Item 5 — rides along: the title fixes idle expiry of parents with running children; "a broken sub-agent probe … only logged at debug level" is a separate declared fix. Harm named (silent system-wide reap stall), no premise risk, trivial internal surface (probe_failure_warned_at, one consumer: _note_probe_failure).

What this change ships

Inventory (6 items) — 5 justified

Intent: keep an idle parent's shared runtime alive while its background sub-agents still run — a FIX (defect readable on base: session_cleanup.py:1054-1181 asked the probe only on the orphan branch and reset the idle branch key-only; issue #11709 linked).

  1. An idle parent with running sub-agents is no longer expired — justified
  2. After the probe await, the sweep re-judges injection, incarnation, semaphore, and clock before consolidating, both axes — justified
  3. Idle expiry is pinned to the scanned session; a replacement keeps its runtime — justified
  4. A reopened slot's fresh ownership claim survives a stale orphan verdict — justified
  5. A failing probe warns hourly instead of debug-only — rides along (declared second fix; named harm)
  6. session.md and the session.py module map updated in the same commit — justified

The deleted pins ("The idle path is spelled separately … that axis is not what this change touches"; "keeps its long-standing key-only reset") recorded a scoping choice whose premise — no await before the idle reset — this fix itself removes; the description declares the reversal. The RSS recycle fires its callback only after a pinned reset succeeds (session_cleanup.py:490-508), so no unfixed sibling shares the consolidation-before-reset gap.

[FIRST-PRINCIPLES-REVIEWED] 4795670

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Fable 5, fork) — ✅ PASS

Design-level review of 4795670dac9b98140db0d84f784fd7ae691e6639 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Verify the trade that a persistently failing probe now holds ALL reaps — idle included, the former last-resort reaper — with only an hourly WARNING.

[DESIGN-REVIEWED] 4795670

@github-actions

github-actions Bot commented Sep 18, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5 Review (fork) — ✅ no blocking findings

Reviewed 4795670dac9b98140db0d84f784fd7ae691e6639 via the fork AI-review pipeline; updated in place on each push.

Review details

The new mid-sweep incarnation check reports a session that a concurrent sweep already reaped as "left running".

FINDING — src/kiro_crew/session_cleanup.py:1136 — self._owner._sessions.get(key) is not scanned is also true when the entry is gone (None), so an overlapping _expire_idle pass — the case test_session_terminated_reap.py:682 exercises — or an explicit reset landing inside the probe await makes the new branch log "Idle sweep: %s changed hands mid-sweep - left running" for a key whose session was in fact reset, telling an operator triaging a vanished runtime that it is still alive → Fix: bind current = self._owner._sessions.get(key) and log "already reaped mid-sweep" when current is None, keeping the "changed hands … left running" wording only when a different incarnation holds the key.

[OPUS-REVIEWED] 4795670

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Sep 18, 2026
cixuuz
cixuuz previously approved these changes Sep 19, 2026

@cixuuz cixuuz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Small, correct fix mirroring an existing, already-tested pattern: the RSS reap path already calls await self._has_attached_subagents(key) before resetting a session (verified at line 394 of the current file), and the idle sweep's orphan branch -- which ignores the clock entirely -- was the one reset path that never consulted it. Tests cover both branches, the coroutine-vs-bool probe answer, fail-closed on a raising probe, and confirm the guard doesn't leak (a childless idle session still expires).

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Sep 20, 2026
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from 1228f06 to c3387fc Compare September 20, 2026 01:36
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Sep 20, 2026
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Rebased onto current main: 1228f062e is now c3387fc14, same intent, one conflict in session_cleanup.py. Main's idle sweep meanwhile gained the completion-injection guard, the scanned incarnation pin, and a sub-agent probe on the orphan branch only. The resolution keeps all of that and hoists the probe out of the orphan branch so it runs on both axes, which is what this PR is for: the idle clock is the axis a long sub-agent run makes stale. The orphan branch's re-checks after the probe (injection counter, live slot set) are unchanged. 1071 tests across the idle-sweep and injection test files pass on the new head; the .github diff against main is empty.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from c3387fc to 77f6d34 Compare September 20, 2026 02:02
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the GPT 5.6 finding on c3387fc14 (the idle probe can reap a replacement session): upheld and fixed in 77f6d34c1.

Hoisting the sub-agent probe onto the idle axis put an await where main's idle path had none, and that path reset by key alone, so a session recreated under the same key during the probe would have been reset on its predecessor's verdict. The reset is now pinned to scanned on both paths (expect_session=scanned); reset revalidates identity under its own lock and declines on a mismatch. Tests: test_idle_reset_is_pinned_to_the_incarnation_the_probe_judged (probe yields, key changes hands, newcomer keeps its runtime; fails on c3387fc14), and test_an_idle_session_is_reset_not_removed in test_session_coverage.py now asserts the pinned call shape. 1071 tests across the idle-sweep and injection test files pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 20, 2026
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from 34b106b to b501786 Compare September 25, 2026 07:34
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the Opus 5 finding on 28a3a84b8 (src/kiro_crew/session.py:81, module map still says only the orphan axis refuses a session with attached sub-agent work): upheld and fixed in b50178606.

The module map now reads that BOTH axes ask the sub-agent probe and refuse a session with attached sub-agent work, and that only the orphan axis then re-asserts against the live set as the last read before the reset. That matches the code (await self._has_attached_subagents(key) at session_cleanup.py:1097, above the if is_orphan split) and docs/system-specs/modules/session.md:935-937. I grepped src/kiro_crew/session*.py and docs/system-specs for any other orphan-only description of the probe and found none. Prose-only change, so there is no behavioural regression to pin; the 251-test session cleanup set still passes on this head.

The Opus 5 check on 34b106b70 was "review incomplete" (no verdict posted, the review comment still named 28a3a84b8), not a new finding. 34b106b70 was a patch-identical rebase of 28a3a84b8 (range-diff =), so this head is the first to carry a change since that review.

Other reds on 34b106b70, for the record: Pod Boot Canary (Windows) failed test_pod_windows_boot.py::test_a_real_pod_boots_under_task_scheduler_and_leaves_nothing_behind with pod: no internal-API credential for pod 'KiroCrew', the known flake #13256 (this diff touches no pod, Task Scheduler or canary code; the same job passed on main runs 36105328552 and 36104536989). Backend Lint & Type Check (3.12) failed in Check formatting (black, baselined), upstream #13386. The cancelled lanes were superseded by the body edit.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

@iamwhatever follow-up to my earlier reply: the PR is now on head b50178606, rebased onto origin/main c66f95b1c (same single commit, plus the Opus module-map wording in session.py). Main added lines above the idle-sweep section of docs/system-specs/modules/session.md, so the spec references in that reply moved by 21 lines. The content is unchanged:

  • orphan-only live-set re-assert, probe asked on BOTH axes: session.md:935-937 (was 914-916)
  • "Every branch that resets suspends on the sub-agent probe" and the four post-await re-judges: session.md:962-979 (was 941-958)
  • the accepted fail-closed residual: session.md:1015-1024 (was 994-1003)

The session_cleanup.py references (:139-141, :227, :568-574, :583-585, :587-605, :1143) and the test references (:259, :340, :364, :387) still point at the same lines on this head.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from b501786 to 2458e85 Compare September 25, 2026 09:00
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the GPT 5.6 and Opus 5 findings on b50178606 (src/kiro_crew/session_cleanup.py:568-569, the probe-failure contract names "its task store unreadable" as the example cause, but that cause never reaches _note_probe_failure): upheld and fixed in 2458e85d9. Both lanes flagged the same span; this is the one disposition for it.

The finding is right. In dashboard/chat_utils.py, subagents_attached_async catches a queued-probe exception and sets queued = 1, and taskq_bridge answers UNKNOWN_PENDING = 1 for a store it cannot read instead of raising. _attached_verdict then returns True with no exception, so _has_attached_subagents never enters its except and the bounded WARNING never fires for that cause.

I took the narrowing fix, not the new-unknown-verdict one. The wrapper docstring (session_cleanup.py:568-578), the CleanupState.probe_failure_warned_at comment (:136-140) and the spec residual (docs/system-specs/modules/session.md:1015-1028) now say the WARNING covers only an exception that escapes the probe, and that an unreadable task store surfaces only as the per-key "still has sub-agent work - left running" INFO line. Having chat_utils / taskq_bridge return a distinct "unknown" verdict would change the shared probe contract that the RSS recycle, teardown and chat_done_payload also read. That is out of scope for this fix, and it is a fair follow-up if the maintainer wants an unknown-store signal above INFO. Wording-only change, so there is no behavioural regression to add. test_a_probe_that_answers_leaves_the_log_quiet already pins that a probe answering True, which is what an unreadable store looks like at this layer, logs no warning. 251 session cleanup tests, the black gate, isort, flake8, mypy and docs lint pass on 2458e85d9.

CI reds on b50178606 outside this diff: Backend Tests (Windows) (4) failed test_diag_procs.py::test_cpu_and_runq_percentages_are_deltas_between_two_scans (47.76 == 50.0 ± 1), the open flake #13134. Backend Lint & Type Check (3.12) exited 143 in Check formatting (black, baselined) with the runner shutdown signal, upstream #13386. macOS (1) was cancelled at the job time limit while still passing tests. Publish readiness signal hit API rate limit exceeded for installation on its status POST.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

@iamwhatever correction to my line-reference update above: the head is now 2458e85d9. It adds the probe-failure wording fix, which grew the _has_attached_subagents docstring by four lines, so the session_cleanup.py references below it moved. The code is unchanged. Current references:

  • CleanupState.probe_failure_warned_at: session_cleanup.py:136-141. PROBE_FAILURE_WARN_INTERVAL_SECS: :227 (unchanged)
  • the wrapper docstring on the fail-closed cost: :568-578 (was 568-574). The fail-closed call: :587-589 (was 583-585)
  • _note_probe_failure: :591-609 (was 587-605). The idle-axis permit re-read: :1147 (was 1143)
  • spec: session.md:935-937, :962-983 and the residual at :1015-1028. The residual now also says an unreadable task store surfaces only as the per-key INFO line, not the bounded WARNING
  • tests: test/test_session_idle_attached_subagents.py:259, :340, :364, :387 (unchanged)

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from 2458e85 to a30e2f5 Compare September 25, 2026 11:35
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Disposition for the GPT 5.6 advisory finding on 2458e85d9 (src/kiro_crew/session_cleanup.py:1122, narrowing the post-probe if self._injection_pending(key) re-read to the orphan axis would leave every test green): upheld and fixed in a30e2f505.

The finding is right about the coverage. The code already re-reads the counter on both axes after the probe (session_cleanup.py:1122, above the is_orphan split), but the only test of an injection starting inside the probe await, test_session_terminated_reap.py::test_an_injection_starting_inside_the_probe_await_spares_it, drives the orphan axis. On the idle axis the semaphore and clock re-reads could not stand in for it, because an injection commits its turn before acquiring the session: the permit stays free and last_used stays stale.

New test: test/test_session_idle_attached_subagents.py:285 test_an_injection_that_begins_inside_the_probe_stops_the_idle_expiry raises the injection counter inside the probe, leaves the permit and last_used untouched, and asserts on_session_expire never runs and the session survives. With the re-read narrowed to if is_orphan and self._injection_pending(key), it fails ("consolidated an idle session with an injection committed to it") and is the only failure across the idle and reap test files (1 failed, 53 passed); on a30e2f505 it passes. Test-only change; the commit is still the single PR commit, range-diff against 2458e85d9 shows only this test added. 183 passed across the idle, reap, coverage and consolidate test files; black gate, isort and flake8 pass.

@jeeshofone

Copy link
Copy Markdown
Contributor Author

@iamwhatever line references for your 23 Sep review, current as of head a30e2f505 (same base c66f95b1c). This head adds one test, so only the test references below it moved. session_cleanup.py and session.md are unchanged from 2458e85d9:

  • CleanupState.probe_failure_warned_at: session_cleanup.py:136-141. PROBE_FAILURE_WARN_INTERVAL_SECS: :227
  • the wrapper docstring on the fail-closed cost: :568-578. The fail-closed call: :587-589
  • _note_probe_failure: :591-609. The idle-axis permit re-read: :1147
  • spec: session.md:935-937, :962-983 and the residual at :1015-1028
  • tests: test/test_session_idle_attached_subagents.py:259 (unchanged), :371, :395, :418 (were 340, 364, 387)

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Sep 25, 2026
Probe attached work on both idle-sweep axes, re-judge the scanned incarnation after awaits, and expose fail-closed probe failures through a bounded warning. Update the session specification and regression coverage for the complete cleanup contract.
@jeeshofone
jeeshofone force-pushed the fix/idle-sweep-attached-subagents branch from a30e2f5 to 4795670 Compare September 25, 2026 21:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Idle sweep resets a session while its sub-agents are still running: the attached-child guard has one call site, on the RSS path only

3 participants