Repository navigation
fix(subagent): free no slot on a second Stop of a queued run - #18470
iamwhatever merged 2 commits into
Conversation
Stopping a sub-agent that is still waiting in the queue, then stopping it again before its stop report has run (a double-clicked Stop, a second client), freed a lane slot that a running agent holds. The pump then started one more run than the cap allows, and every repeat lowered the running count by one more. A stress run of 200 spawns with a cap of 8 and 50 double Stops ran 54 agents at once and left the count at -46. The first Stop registers the queued stop's terminal record (queued=True, done=False until its report task runs). cancel only sent unregistered or done records down its queued path, so the second Stop treated that record as a live run and reaped it. The reap's _release_slot spent the record's unspent token and decremented the running count for a slot the record never held. cancel now sends a registered queued record down the queued path, which finds the row already stopped and answers False. _release_slot returns False for any queued record: admission registers a fresh, unqueued record when it takes a slot, so a queued one holds none.
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS One thing to verify before merge: no future path puts a slot-holding record into What this change shipsInventory (4 items) — 4 justifiedIntent: when a queued sub-agent is stopped twice, the concurrency cap must still hold. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 7b91bd7 Description read[DESCRIPTION-READ] 7c9d056aafcb8265786593244bea2cea20f6c5b3cd462937bad41b501a024b6e That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that no code ever flips [DESIGN-REVIEWED] 7b91bd7 Description read[DESCRIPTION-READ] e134f1f06e3d6cf0bf413c7a248b23130852bec62922a7e1b8a3fb8111227e08 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
The first fix sent a second Stop of a queued run down cancel's queued path, which calls the synchronous taskq_cancel_queued. That runs store.get and store.cancel inside async def cancel_impl, on the gateway event loop, for a row the first Stop already ended. cancel_impl now answers a registered queued record "not running" right after the batched-stop join, before any store call. Only a queued stop's terminal record is registered queued, so a waiting row's first Stop still takes the store path, and a Stop all in flight is still joined. test_second_stop_makes_no_store_call pins it: the second Stop makes no taskq_cancel_queued call and leaves TaskStore.loop_thread_calls unchanged.
|
Thanks for catching this. The GPT 6.1 finding (
New test: |
iamwhatever
left a comment
There was a problem hiding this comment.
This fixes over-admission when a queued sub-agent is stopped twice. The first Stop registers a queued=True, done=False terminal record. The second Stop treated that record as a live run, reaped it, and _release_slot lowered running_count for a slot the record never held. The root cause is fixed in two places: cancel_impl (cancellation.py:1330) now sends a registered queued record down the queued path and answers False, and _release_slot_impl (terminal.py:576) refuses any queued record. That guard sits in the one funnel every slot release goes through, so no other path can repeat the bug. I checked every queued=True constructor on main (admission/gate.py, _report_queued_stop, continuation.py). None of them is a record that holds a slot, and _report_queued_stop already declines to overlay a registered live run, so the new guard cannot strand a real slot. The tests cover the concurrent and sequential double Stop, and they check every write to _running_count stays within 0..cap, that a later spawn waits, that exactly one report goes out and that no reap happened. The description shows the tests red on main, with each change needed on its own. All lanes PASS, checks are green and the scope is small. — crew-pr-reviewer
Problem / Motivation
Goal: Stopping a queued sub-agent twice never lets more sub-agents run than the concurrency cap allows.
A user stops a sub-agent that is still waiting in the queue, then stops it again before its stop report has run.
That happens with a double-clicked Stop or with two clients, and the second Stop can land on the same tick or on the next one.
The second Stop frees a lane slot that a running agent still holds.
The pump then starts one more run than the cap allows, and each repeat lowers the running count by one more.
Both Stops answer True.
A stress run (200 spawns, a cap of 8, 100 queued spawns stopped, 50 of them twice) ran 54 agents at once and left
running_countat -46.Why it matters
The concurrency cap is what keeps a wide fan-out from starting dozens of runtime processes at once.
After a few double Stops it no longer holds, and the count stays wrong until the gateway restarts, so every later wave over-admits too.
Each bogus reap also resets a session key that never had a session.
Not a goal
not info.queuedincancel_for_parent_impl), and teardown and Stop all stop each run throughcancel, so they reach the changed check._release_slot, which do not share this cause:src/kiro_crew/subagent_manager/admission/gate.pyundo the slot their own record took in the same synchronous step, after settingdone.release_reservationinadmission/pump.pyand_resume_unreserveinadmission/waits.pygive back a reservation (aClaimPointor a resume reserve), not a record's slot, each paired with its own increment._release_slotcallers keep their behavior. They are the run's ownfinally(completion, failure, cancellation), the reap (Stop, deadline, stall), the spawn-approval failure paths inadmission/pump.py, and the park yield inadmission/waits.py. Each one reaches only a registered, unqueued record that took a slot. Sweep command:git grep -n "_release_slot(\|_running_count -= 1\|_running_count = max" -- src/kiro_crew/subagent_manager.What changed (motivation → approach → change)
running_countby one._report_queued_stop:queued=True,done=Falseuntil its report task runs).cancel_implinsrc/kiro_crew/subagent_manager/cancellation.pysent only unregistered ordonerecords down its queued path. The second Stop therefore treated the registered terminal record as a live run and called_force_reap. The reap's_release_slot(src/kiro_crew/subagent_manager/terminal.py) checks only the one-shot_slot_releasedtoken, which a record that never held a slot has not spent, so it returned True and the reap decrementedrunning_countand drained the queue.cancel_impl: a registeredqueuedrecord takes the queued path, as an unregistered ordoneone does. That path joins a Stop all batch that owns the row, or finds the row already stopped and answers False. No reap, session reset or second report runs._release_slot_impl: return False for anyqueuedrecord. Admission registers a fresh, unqueued record when it takes a slot, so a queued record never holds one. The guard sits in the one funnel every terminal path frees a slot through, so no path can free a slot for such a record, whichever one reaches it. The guard does not spend the token, because other readers treat_slot_released=Trueon a record that is not done as a parked run that yielded its slot._slot_releasedat build time. Fairness, the stall monitor and the wait coordinator read a record that is not done with_slot_released=Trueas a parked run, so the record would look like one during its report window.Backwards compatibility
Compatible: only a Stop of an id whose queued stop is still reporting changes; it answers False ("not running"), which is the answer the same Stop gets once that report has run.
Removes nothing: no field, key, row or function is removed;
_release_slotkeeps its contract for every record that took a slot.Tests
New class
TestSecondStopOfAQueuedRunintest/test_subagent_scale.py. It uses the sharedManagerHarnessfromtest/overload_fakes.py(a real manager whose runs end when the test says so) with a cap of 2: two runs hold both slots and a third waits.test_second_stop_frees_no_slot_and_starts_nothing,[concurrent](both Stops in oneasyncio.gather) and[sequential](the second Stop after the first returns). It asserts thatrunning_countstays 2 and that the Stops answer True then False. It asserts that_release_slotrefuses the terminal record and that a spawn made after the Stops waits instead of starting. After every run ends, it asserts thatrunning_countis 0 and that the parent got exactly one report for the stopped id. It asserts that no session was reset, so no reap ran._record_running_countrecords every value written to_running_countduring the block. The test asserts every write stays between 0 and the cap.Red on main (
pytest test/test_subagent_scale.py::TestSecondStopOfAQueuedRun -n 0, with main'scancellation.pyandterminal.py), RC=1, 2 failed:Each change is needed: with only change 1 the test fails on
_release_slot(assert True is False); with only change 2 it fails on the answers (assert [True, True] == [True, False]), because the reap still runs.Green on this branch: the same command gives RC=0, 2 passed.
Every test file that names
_release_slot,cancel(,_report_queued_stoporManagerHarnessamong the sub-agent, spawn, fairness and wait tests (47 files), run serially: RC=0, 2241 passed.mypyandblack --checkon both changed source files,flake8,isort --check-onlyandblack --checkon the test file,scripts/check_black_formatting.py,scripts/check_comment_history.py(withCOMMENT_HISTORY_BASE_REF=origin/main) andscripts/docs-lint.shall exit 0.Manual verification
At scale, with the same in-process stress driver that found the defect (200 spawns, a cap of 8, a stop storm over queued runs, each scenario asserting
peak <= cap): on main it fails withafter_running_count: -46; on this branch all 4 scenarios pass. The driver is not part of this PR; the unit tests above pin the mechanism.Related Issues
None found. GitHub searches of this repository's PRs for
_report_queued_stop,_running_countand_release_slotfound related merged work (#16390, #16394, #13890, #337, #16191) and nothing about a second Stop;queued stop double,cancel queued subagentandover-admit subagentreturned no results.Overlap: open PRs touch the same files but not the changed lines. #11794 edits
cancel_all_impl's docstring, #18055 adds a query next to_queued_stop_reported, and #18078, #16985 and #14464 change_reap_once_implor_report_terminal_implinterminal.py; none editscancel_impl's queued check or_release_slot_impl.Pattern harvest
Rule candidate: review-prompt
Pattern: a one-shot release token that defaults to "unspent" on records that never acquired the resource, so any path that reaches such a record releases a resource it does not hold.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)docs/system-specs/modules/subagent.mdalready states that "every running sweep skips aqueuedrecord"; this change brings the single Stop in line with that rule and changes no documented behavior