Skip to content

fix(session): keep a prefetch parent that still owes sub-agent work (#10987) - #14464

Merged
bolichen97 merged 2 commits into
mainfrom
drive/10987-prefetch-parent-subagents
Oct 12, 2026
Merged

bolichen97 merged 2 commits into
mainfrom
drive/10987-prefetch-parent-subagents

Conversation

@iamwhatever

@iamwhatever iamwhatever commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stop a speculative-prefetch reclaim from tearing down a parent session that still owes sub-agent work.

SessionLifecycleService.remove_if_unclaimed (the resume-prefetch TTL and the prewarm cap eviction) reclaims an idle speculative session when its first turn is unclaimed and its semaphore is free. A free semaphore proves only that the parent's own turn ended. Children spawned by that turn run on the parent's runtime, and a finished child still owes the parent its report. On current main the reclaim checks neither, so it cancels live children and drops their results.

Why it matters

The reclaim fires on a timer and on tab-focus pressure, so it silently kills in-progress sub-agent work under a race that is hard to see from the outside.

Not a goal

  • No change to the idle sweep or the RSS recycle; they keep their own probe.
  • The fix sits at the reclaim only. The identity sweep's spare-children decision has the same unfinished-report gap; it is deferred to Identity sweep can retire a parent while a child's report is still delivering #19435. has_unfinished_report_for stays a separate probe rather than a term inside has_pending_work_for, because that signal also drives the gateway reset-deferral sites and the folder-move guard, and widening it there is its own change.
  • No new bookkeeping in SubagentManager. The earlier revision of this PR added a per-parent attachment generation and an in-flight delivery count; both are gone.

What changed

The reclaim now reuses the two primitives main already has for the identity sweep's spare-children path, so it reads one owed-work signal instead of a union of per-interleaving guards:

  1. Under the registry lock: the existing claim checks, then open a tentative teardown fence (open_teardown_fence).
  2. Off the lock: has_pending_work_for_async (queued, running, a finished child whose terminal report is not registered yet, a live follow-up watcher) and _parent_has_sparable_child (live/queued, fenced admissions, store-only rows). A read that raises keeps the parent. has_unfinished_report_for (new, in-memory) is asked here and again under the lock: a registered terminal report still delivering after its run's task ended — a reaped run's report waits across the reap's awaits — which _has_live_parent_run_task does not see.
  3. Back under the lock: re-check the claim, a pending completion injection, has_live_or_queued_children and has_in_memory_pending_work_for, then commit only if commit_identity_sweep_retire promotes the fence. A commit selects children with snapshot_teardown_children_only, as the identity sweep does.
  4. Any keep releases the tentative fence (also on cancellation).

_ChildTeardownHandler gains the two pending-work methods SubagentManager already exposes, plus has_unfinished_report_for (a read over _report_owners).

Backwards compatibility

Compatible. No field, key or validator changes. The only behaviour change is that a reclaim now keeps a parent it would previously have reaped.

Tests

test/test_session.py::TestParentEndCancelsItsChildren:

  • test_remove_if_unclaimed_keeps_a_parent_with_a_live_child — the regression: fails on main (the reclaim returns True and cancels run-1).
  • test_remove_if_unclaimed_keeps_a_parent_that_owes_work[pending_work|unfinished_reports|store_children|fenced_children|store_read_raises] — owed work the live probe cannot see, and fail-closed on an unreadable store.
  • test_remove_if_unclaimed_keeps_a_parent_that_gains_work_during_the_read — work appearing during the off-loop read.
  • test_remove_if_unclaimed_reclaims_a_parent_that_owes_nothing — the reclaim still works and promotes the fence.

test/test_subagent_queued_visibility.py::test_an_unfinished_report_is_owed_after_its_run_task_ends — the real manager's _report_owners read.

The keep-tests fail with main's session_lifecycle.py and pass with this change. Local: test/test_session.py -k "TestParentEndCancelsItsChildren or unclaimed", test/test_eager_spawn.py, test/test_session_coverage.py, test/metrics/ green; black ratchet, isort, flake8, mypy and comment-history gates green.

Manual verification

N/A — no user-visible surface; the race is covered by the unit tests above.

Related Issues

Replacement for #10987 (fork PR by @Premshay, not pushable to this repository). Original author credited via Co-authored-by.

Pattern harvest

Rule candidate: review-prompt
Pattern: a teardown path that gates on a free semaphore alone ("idle") must also ask the sub-agent manager's pending-work signal before it pops, because children and owed reports outlive the parent's own turn.

Checklist

  • Two commits, Conventional Commits titles
  • Existing tests pass and new tests added
  • Self-review completed
  • Documentation updated — docs/system-specs/modules/session.md
  • No secrets, credentials, or internal references in the diff

@iamwhatever
iamwhatever requested a review from a team as a code owner September 27, 2026 11:42
@iamwhatever iamwhatever added the drive-to-green PR claimed by drive-to-green pipeline label Sep 27, 2026
@iamwhatever iamwhatever self-assigned this Sep 27, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Sep 27, 2026
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — 🟡 CONCERNS

Design-level review of f2096e9f1557e20259eb070be0df90994abd3ecf — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: CONCERNS

A kept parent now returns False, and the cap evictor reads False as "gone", so a live idle process drops out of the prewarm cap count.

Watch

  • The cap evictor misreads a kept session. chat_runner.py:_evict_prefetches_beyond treats a False from remove_if_unclaimed as "claimed or already gone — either way it no longer counts against the cap; dropping the registry entry suffices" and pops the key from _armed_prefetches. After this PR, False also means "kept because it owes sub-agent work", and that session is still live and unclaimed. The evictor stops counting it, _admit_prefetch decides room was made, and it spawns another process on top. The live population can then exceed the allowance, including the critical band where the allowance is zero and the idle population is supposed to drop to zero. The TTL path (_prefetch_ttl) has the same problem: its timer is one-shot, so a parent it keeps is never retried, and only the idle sweep can reclaim it later. The description's "only behaviour change is that a reclaim now keeps a parent" leaves out this caller's contract.
    Clears when: remove_if_unclaimed tells "kept, owes work" apart from "claimed/gone" (for example a tri-state result), and _evict_prefetches_beyond keeps counting a kept entry and returns False (refuses admission). A test should pin that an owed-work parent still holds a cap slot.

[DESIGN-REVIEWED] f2096e9

Description read

[DESCRIPTION-READ] ee1f2460f2993cd0b5062777433951432700e827a8149d89b662fcce07a7a7f0

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of f2096e9f1557e20259eb070be0df90994abd3ecf — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check chat_runner.py:5655: the cap eviction still reads False as "claimed or already gone", so a kept parent drops out of _armed_prefetches and stops counting against the cap.

Not justified as shipped

  • Item 5 — undeclared: the description never says that a False from the cap eviction now also means "kept alive". The caller's comment # False = claimed or already gone — either way it no longer counts against the cap now covers a live session too.

What this change ships

Inventory (6 items) — 5 justified

Intent: stop a prefetch or prewarm reclaim from killing sub-agents that a parent session still owes, or dropping their reports. This is a FIX.

  1. A prefetch or prewarm reclaim now keeps an idle parent that has a live or queued child — justified
  2. The reclaim keeps a parent with owed work that the live probe can't see (an unregistered report, a follow-up watcher, store or fenced rows) — justified
  3. The reclaim keeps a parent whose registered report is still delivering after its task ended (has_unfinished_report_for, one consumer). The identity sweep sibling is deferred to Identity sweep can retire a parent while a child's report is still delivering #19435 — justified
  4. If the sub-agent store can't be read, the reclaim keeps the parent instead of tearing it down — justified
  5. A kept parent now leaves the prefetch-cap registry, and the TTL path does not retry it. Only the idle sweep reclaims it later — undeclared
  6. The old pin test_remove_if_unclaimed_cancels_children is reversed. The description says so ("fails on main … cancels run-1"), and its origin fix(session): a parent that ends takes its sub-agents with it #11966 covered cancelling children on a removal that commits, which still holds — justified

[FIRST-PRINCIPLES-REVIEWED] f2096e9

Description read

[DESCRIPTION-READ] 47d0687eb3a1a8b4c514671d7d2c2fe4ce83c2997ae8c706ce08f75307daf77e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f2096e9f1557e20259eb070be0df90994abd3ecf and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f2096e9

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f2096e9f1557e20259eb070be0df90994abd3ecf: <one-sentence reason>

@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from 44d190e to c9748bf Compare September 27, 2026 11:58
@github-actions

github-actions Bot commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f2096e9f1557e20259eb070be0df90994abd3ecf — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f2096e9

Verdict parsed from the review's SHA-scoped output markers for commit f2096e9f1557e20259eb070be0df90994abd3ecf.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f2096e9f1557e20259eb070be0df90994abd3ecf: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from c9748bf to d17833b Compare September 27, 2026 17:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch 2 times, most recently from ed12833 to 12ab45f Compare September 27, 2026 18:05
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running merge conflict Branch has merge conflicts with its base — author must resolve before merge labels Sep 27, 2026
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from 12ab45f to 7e6ce9f Compare September 27, 2026 18:22
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Sep 27, 2026
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Sep 27, 2026
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from 7e6ce9f to 5286eb7 Compare September 27, 2026 21:58
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Sep 27, 2026
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

Following up on the design call above: main now carries the single owed-work signal this comment asked for, so the 7th-guard question is settled without a 7th guard.

The new head 8b6e330555 drops the attachment generation and the in-flight delivery count entirely and rebuilds remove_if_unclaimed on what main already uses for the identity sweep's spare-children path:

  • SubagentManager.has_pending_work_for_async / has_in_memory_pending_work_for — queued, running, a live follow-up watcher, and _has_live_parent_run_task, which spans exactly the done-flip-to-report-registered window GPT flagged at the old session_lifecycle.py:1967.
  • The tentative teardown fence (open_teardown_fence → _parent_has_sparable_child → commit_identity_sweep_retire) for children admitted while the off-loop read runs.

The diff is now one source file plus tests and a spec line. The PR body is rewritten to match.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f2096e9f1557e20259eb070be0df90994abd3ecf touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot removed the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 12, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor Author

GPT 6.1 BLOCK on 8b6e330555 (session_lifecycle.py:2788, unfinished registered reports): verified in code and fixed in cf5451925b.

The finding holds. A report task registered in _report_owners (subagent_manager/terminal.py:569) can still be waiting, for example on the reap gate across the reap's reset and kill awaits (terminal.py:1112-1125). _has_live_parent_run_task (subagent_manager/run.py:1428) reads only _tasks, so both pending-work reads came back false and the teardown could gate the report.

Fix: SubagentManager.has_unfinished_report_for(parent) reads _report_owners for a not-done task owned by that parent. remove_if_unclaimed asks it before the off-loop read and again in the locked recheck that comes right before the pop. Tests: test_remove_if_unclaimed_keeps_a_parent_that_owes_work[unfinished_reports] and test_an_unfinished_report_is_owed_after_its_run_task_ends on the real manager.

@iamwhatever

Copy link
Copy Markdown
Contributor Author

First Principles CONCERNS on cf5451925b (item 3, has_unfinished_report_for): the disposition is to take the second "clears when" and change no code.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I checked this at head cf54519.

What it fixes: an unclaimed resume-prefetch session that timed out, or was evicted by the prefetch cap, while sub-agents it owned were still running or still delivering their report, cancelled those sub-agents and lost their results. The reclaim now keeps that session.

Checked: the reclaim asks both has_unfinished_report_for and has_pending_work_for_async (session_lifecycle.py:2772-2773), re-checks under the lock, and treats a raising read as "keep" (fail-closed). The fence is released in finally.

Mutations: removing both pending-work probes reddens test_remove_if_unclaimed_keeps_a_parent_that_owes_work[pending_work] and test_remove_if_unclaimed_keeps_a_parent_that_gains_work_during_the_read; removing has_unfinished_report_for reddens ...owes_work[unfinished_reports]; falling through after a raising read reddens ...owes_work[store_read_raises]; dropping the fence release reddens 6 tests; dropping not task.done() reddens test_an_unfinished_report_is_owed_after_its_run_task_ends.

CI: the one red, macOS Tests / Backend Tests (macOS) (2), failed on both attempts only on test_provider_spawn_off_loop_18172.py::...::test_cancellation_during_the_spawn_still_reaps, a test this PR does not touch. #19310 and #19451 already track that flake. The other non-green lanes are cancelled runs.

Non-blocking follow-ups:

  • No test pins the async-only half of has_pending_work_for_async (store-side overflow): removing the off-lock call alone stays green because the test fake drives both probes from one dict.
  • chat_runner.py:5576-5577 still says False means "claimed or already gone"; it can now also mean kept because it owes work.
  • A kept prefetch session no longer counts against the prefetch cap while its process lives on. It is bounded by the idle sweep, but it quietly changes what the cap bounds.
  • subagent.md has no entry for the new public SubagentManager.has_unfinished_report_for, while its siblings are listed.

@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from cf54519 to 0a01ff7 Compare October 12, 2026 17:37
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 12, 2026
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from 0a01ff7 to f3ad874 Compare October 12, 2026 18:39
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
Joe Guo and others added 2 commits October 12, 2026 19:33
remove_if_unclaimed reclaimed an idle speculative session on the strength
of a free semaphore, which proves only that the parent's own turn ended.
Children spawned by that turn run on the parent's runtime, and a finished
child still owes the parent its report, so the reclaim cancelled live work.

The reclaim now takes the identity sweep's spare-children shape: open a
tentative teardown fence under the lock, ask the manager's pending-work
signal and the fence-aware store read off the lock, then re-ask the
in-memory half and promote the fence under the lock before the pop. Owed
work, a failed read or a pending injection keeps the parent.

Co-authored-by: Premshay <28099628+Premshay@users.noreply.github.com>
@iamwhatever
iamwhatever force-pushed the drive/10987-prefetch-parent-subagents branch from f3ad874 to f2096e9 Compare October 12, 2026 19:34
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 12, 2026
@bolichen97
bolichen97 merged commit d3ce3c6 into main Oct 12, 2026
104 checks passed
@bolichen97
bolichen97 deleted the drive/10987-prefetch-parent-subagents branch October 12, 2026 22:27
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-author-decision PR blocked on author input

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants