Skip to content

fix(subagent): free no slot on a second Stop of a queued run - #18470

Merged
iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/subagent-second-stop-frees-no-slot
Oct 9, 2026
Merged

iamwhatever merged 2 commits into
kirodotdev:mainfrom
javenciu:fix/subagent-second-stop-frees-no-slot

Conversation

@javenciu

@javenciu javenciu commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Stopping a queued sub-agent twice never lets more sub-agents run than the concurrency cap allows.

A user stops a sub-agent that is still waiting in the queue, then stops it again before its stop report has run.
That happens with a double-clicked Stop or with two clients, and the second Stop can land on the same tick or on the next one.
The second Stop frees a lane slot that a running agent still holds.
The pump then starts one more run than the cap allows, and each repeat lowers the running count by one more.
Both Stops answer True.

A stress run (200 spawns, a cap of 8, 100 queued spawns stopped, 50 of them twice) ran 54 agents at once and left running_count at -46.

Why it matters

The concurrency cap is what keeps a wide fan-out from starting dozens of runtime processes at once.
After a few double Stops it no longer holds, and the count stays wrong until the gateway restarts, so every later wave over-admits too.
Each bogus reap also resets a session key that never had a session.

Not a goal

  • Any change to how a first Stop of a queued run, a Stop all, or a parent-end teardown stops runs. Stop all's running sweep already skips queued records (not info.queued in cancel_for_parent_impl), and teardown and Stop all stop each run through cancel, so they reach the changed check.
  • The decrements that do not go through _release_slot, which do not share this cause:
    • The two "no approval mechanism configured" rejections in src/kiro_crew/subagent_manager/admission/gate.py undo the slot their own record took in the same synchronous step, after setting done.
    • release_reservation in admission/pump.py and _resume_unreserve in admission/waits.py give back a reservation (a ClaimPoint or a resume reserve), not a record's slot, each paired with its own increment.
  • The other _release_slot callers keep their behavior. They are the run's own finally (completion, failure, cancellation), the reap (Stop, deadline, stall), the spawn-approval failure paths in admission/pump.py, and the park yield in admission/waits.py. Each one reaches only a registered, unqueued record that took a slot. Sweep command: git grep -n "_release_slot(\|_running_count -= 1\|_running_count = max" -- src/kiro_crew/subagent_manager.

What changed (motivation → approach → change)

  • Symptom: after one Stop of a queued run, a second Stop of the same id reaps a record and lowers running_count by one.
  • Root cause: the first Stop registers the queued stop's terminal record (_report_queued_stop: queued=True, done=False until its report task runs). cancel_impl in src/kiro_crew/subagent_manager/cancellation.py sent only unregistered or done records down its queued path. The second Stop therefore treated the registered terminal record as a live run and called _force_reap. The reap's _release_slot (src/kiro_crew/subagent_manager/terminal.py) checks only the one-shot _slot_released token, which a record that never held a slot has not spent, so it returned True and the reap decremented running_count and drained the queue.
  • Change 1, in cancel_impl: a registered queued record takes the queued path, as an unregistered or done one does. That path joins a Stop all batch that owns the row, or finds the row already stopped and answers False. No reap, session reset or second report runs.
  • Change 2, in _release_slot_impl: return False for any queued record. Admission registers a fresh, unqueued record when it takes a slot, so a queued record never holds one. The guard sits in the one funnel every terminal path frees a slot through, so no path can free a slot for such a record, whichever one reaches it. The guard does not spend the token, because other readers treat _slot_released=True on a record that is not done as a parked run that yielded its slot.
  • Rejected: marking the terminal record's _slot_released at build time. Fairness, the stall monitor and the wait coordinator read a record that is not done with _slot_released=True as a parked run, so the record would look like one during its report window.
  • Rejected: a new "slot held" flag set at every acquisition site. It covers the same cases through five acquisition sites and their readers, for no added safety here.

Backwards compatibility

Compatible: only a Stop of an id whose queued stop is still reporting changes; it answers False ("not running"), which is the answer the same Stop gets once that report has run.
Removes nothing: no field, key, row or function is removed; _release_slot keeps its contract for every record that took a slot.

Tests

New class TestSecondStopOfAQueuedRun in test/test_subagent_scale.py. It uses the shared ManagerHarness from test/overload_fakes.py (a real manager whose runs end when the test says so) with a cap of 2: two runs hold both slots and a third waits.

  • test_second_stop_frees_no_slot_and_starts_nothing, [concurrent] (both Stops in one asyncio.gather) and [sequential] (the second Stop after the first returns). It asserts that running_count stays 2 and that the Stops answer True then False. It asserts that _release_slot refuses the terminal record and that a spawn made after the Stops waits instead of starting. After every run ends, it asserts that running_count is 0 and that the parent got exactly one report for the stopped id. It asserts that no session was reset, so no reap ran.
  • Helper _record_running_count records every value written to _running_count during the block. The test asserts every write stays between 0 and the cap.

Red on main (pytest test/test_subagent_scale.py::TestSecondStopOfAQueuedRun -n 0, with main's cancellation.py and terminal.py), RC=1, 2 failed:

E               AssertionError: two runs still hold both slots
E               assert 1 == 2
E               AssertionError: two runs still hold both slots

Each change is needed: with only change 1 the test fails on _release_slot (assert True is False); with only change 2 it fails on the answers (assert [True, True] == [True, False]), because the reap still runs.

Green on this branch: the same command gives RC=0, 2 passed.
Every test file that names _release_slot, cancel(, _report_queued_stop or ManagerHarness among the sub-agent, spawn, fairness and wait tests (47 files), run serially: RC=0, 2241 passed.
mypy and black --check on both changed source files, flake8, isort --check-only and black --check on the test file, scripts/check_black_formatting.py, scripts/check_comment_history.py (with COMMENT_HISTORY_BASE_REF=origin/main) and scripts/docs-lint.sh all exit 0.

Manual verification

At scale, with the same in-process stress driver that found the defect (200 spawns, a cap of 8, a stop storm over queued runs, each scenario asserting peak <= cap): on main it fails with after_running_count: -46; on this branch all 4 scenarios pass. The driver is not part of this PR; the unit tests above pin the mechanism.

Related Issues

None found. GitHub searches of this repository's PRs for _report_queued_stop, _running_count and _release_slot found related merged work (#16390, #16394, #13890, #337, #16191) and nothing about a second Stop; queued stop double, cancel queued subagent and over-admit subagent returned no results.

Overlap: open PRs touch the same files but not the changed lines. #11794 edits cancel_all_impl's docstring, #18055 adds a query next to _queued_stop_reported, and #18078, #16985 and #14464 change _reap_once_impl or _report_terminal_impl in terminal.py; none edits cancel_impl's queued check or _release_slot_impl.

Pattern harvest

Rule candidate: review-prompt
Pattern: a one-shot release token that defaults to "unspent" on records that never acquired the resource, so any path that reaches such a record releases a resource it does not hold.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable): not applicable. docs/system-specs/modules/subagent.md already states that "every running sweep skips a queued record"; this change brings the single Stop in line with that rule and changes no documented behavior
  • No secrets, credentials, or internal references in the diff

Stopping a sub-agent that is still waiting in the queue, then
stopping it again before its stop report has run (a double-clicked
Stop, a second client), freed a lane slot that a running agent
holds. The pump then started one more run than the cap allows, and
every repeat lowered the running count by one more. A stress run of
200 spawns with a cap of 8 and 50 double Stops ran 54 agents at once
and left the count at -46.

The first Stop registers the queued stop's terminal record
(queued=True, done=False until its report task runs). cancel only
sent unregistered or done records down its queued path, so the second
Stop treated that record as a live run and reaped it. The reap's
_release_slot spent the record's unspent token and decremented the
running count for a slot the record never held.

cancel now sends a registered queued record down the queued path,
which finds the row already stopped and answers False. _release_slot
returns False for any queued record: admission registers a fresh,
unqueued record when it takes a slot, so a queued one holds none.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 9, 2026 12:17
@javenciu
javenciu requested a review from bolichen97 October 9, 2026 12:17
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 9, 2026
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision 7b91bd73999c97f3603f865543164f6e08b19d28 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 7b91bd73999c97f3603f865543164f6e08b19d28 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

One thing to verify before merge: no future path puts a slot-holding record into _agents with queued=True. If one did, the new _release_slot guard would leak that slot quietly instead of failing loudly.

What this change ships

Inventory (4 items) — 4 justified

Intent: when a queued sub-agent is stopped twice, the concurrency cap must still hold. This is a FIX.

  1. A second Stop of a queued run whose stop report is still pending now answers "not running": no reap, no session reset, no slot freed — justified
  2. A slot can no longer be freed through a queued record, whichever terminal path reaches it (_release_slot_impl guard) — justified
  3. The second Stop answers from the registered record alone, with no store cancel on the event loop — justified
  4. Regression tests: a concurrent and a sequential double Stop that pin the slot count, the answers and one report; a test that pins no store call — justified

[FIRST-PRINCIPLES-REVIEWED] 7b91bd7

Description read

[DESCRIPTION-READ] 7c9d056aafcb8265786593244bea2cea20f6c5b3cd462937bad41b501a024b6e

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 7b91bd73999c97f3603f865543164f6e08b19d28 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that no code ever flips SubagentInfo.queued to False on a registered record; the new _release_slot guard relies on that holding.

[DESIGN-REVIEWED] 7b91bd7

Description read

[DESCRIPTION-READ] e134f1f06e3d6cf0bf413c7a248b23130852bec62922a7e1b8a3fb8111227e08

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 7b91bd73999c97f3603f865543164f6e08b19d28 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 7b91bd7

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 7b91bd73999c97f3603f865543164f6e08b19d28 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 7b91bd7

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 9, 2026
The first fix sent a second Stop of a queued run down cancel's queued
path, which calls the synchronous taskq_cancel_queued. That runs
store.get and store.cancel inside async def cancel_impl, on the gateway
event loop, for a row the first Stop already ended.

cancel_impl now answers a registered queued record "not running"
right after the batched-stop join, before any store call. Only a
queued stop's terminal record is registered queued, so a waiting row's
first Stop still takes the store path, and a Stop all in flight is
still joined.

test_second_stop_makes_no_store_call pins it: the second Stop makes no
taskq_cancel_queued call and leaves TaskStore.loop_thread_calls
unchanged.
@javenciu

javenciu commented Oct 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for catching this. The GPT 6.1 finding (no-sync-store-call-from-a-coroutine) is fixed in 7b91bd7, a second commit on top of the original.

  • cancel_impl now returns False for a registered queued record right after the batched-stop join, before taskq_cancel_queued. A second Stop of an already-stopped queued run therefore makes no store call on the event loop, as on main.
  • Only an already-ended record is registered with queued=True. A row still waiting in the queue is never registered, so a first Stop takes the same path as on main.
  • The _release_slot_impl guard is unchanged.

New test: TestSecondStopOfAQueuedRun::test_second_stop_makes_no_store_call. It asserts there is no taskq_cancel_queued call and that TaskStore.loop_thread_calls is unchanged across the second Stop. With the previous head's cancel_impl it fails (the second Stop asked the store to cancel the row again); on this head it passes. test/test_subagent_scale.py plus the 11 test files that name _release_slot, _report_queued_stop or cancel_impl, run serially: 1276 passed. mypy, flake8, isort, black, scripts/check_black_formatting.py, scripts/check_comment_history.py and docs-lint all pass.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 9, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This fixes over-admission when a queued sub-agent is stopped twice. The first Stop registers a queued=True, done=False terminal record. The second Stop treated that record as a live run, reaped it, and _release_slot lowered running_count for a slot the record never held. The root cause is fixed in two places: cancel_impl (cancellation.py:1330) now sends a registered queued record down the queued path and answers False, and _release_slot_impl (terminal.py:576) refuses any queued record. That guard sits in the one funnel every slot release goes through, so no other path can repeat the bug. I checked every queued=True constructor on main (admission/gate.py, _report_queued_stop, continuation.py). None of them is a record that holds a slot, and _report_queued_stop already declines to overlay a registered live run, so the new guard cannot strand a real slot. The tests cover the concurrent and sequential double Stop, and they check every write to _running_count stays within 0..cap, that a later spawn waits, that exactly one report goes out and that no reap happened. The description shows the tests red on main, with each change needed on its own. All lanes PASS, checks are green and the scope is small. — crew-pr-reviewer

@iamwhatever
iamwhatever enabled auto-merge (squash) October 9, 2026 19:34
@iamwhatever
iamwhatever merged commit 5644b2a into kirodotdev:main Oct 9, 2026
107 checks passed
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 9, 2026
@javenciu
javenciu deleted the fix/subagent-second-stop-frees-no-slot branch October 11, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants