Repository navigation
fix(subagent): hold the serial guard for a queued spawn's done-probe - #15869
bolichen97 merged 2 commits into
Conversation
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check before merge: any future state write in [DESIGN-REVIEWED] 94e0689 Description read[DESCRIPTION-READ] a62e8b609256f1e99213ae24a82cb7388548b54fae6732b0f9e2a46c2f05224f That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5, fork) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS
Not justified as shipped
What this change shipsInventory (5 items) — 4 justifiedIntent: an app's serial lock keeps holding while its spawn is still queued, so the same work can't be queued twice. FIX (#15668).
[FIRST-PRINCIPLES-REVIEWED] 94e0689 Description read[DESCRIPTION-READ] 8122220bc764c7dc2e40934042a0805ffea5b1b36cdf38db739bd06ca22aaf81 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
d6709ae to
144effe
Compare
144effe to
3e4e917
Compare
b578439 to
c3b97dd
Compare
|
/ai-review opus |
c3b97dd to
f30c6cf
Compare
|
/ai-review opus |
f5611b0 to
c6edfc8
Compare
|
Maintainer follow-up pushed as What changed and why
Tests. CI state on the previous heads
Out of scope, for a possible follow-up. A low-memory deferral ( |
span=f6e70c057000
|
span=549f6a24a0f7
|
|
/ai-review opus |
|
/ai-review scope |
|
/ai-review gpt |
9645514 to
4862fcf
Compare
span=10162e9148b0
|
|
Maintainer follow-up: I replaced my own commit 9645514 with a reworked second commit on top of the author's Why the rework
These are the same defect in two directions. A manager-side set has to be updated by every path that moves a row, and both directions had missed paths. The store already commits every one of those transitions, so it now keeps the record itself. What changed
Tests (each fails on 9645514, verified on a detached worktree at that sha)
The targeted suites pass locally (taskq store, admission integration, done-probe, subagent, fairness lanes, gate, dependency mark: 408 passed). black, isort, flake8, mypy, docs-lint and the harness-parity gate are clean. CI on 9645514. Opus 5 ("could not run") and Security Scope ("no leg reported") are infrastructure failures, not findings. GPT 5.6's one blocking finding is fixed above and has its own disposition. |
span=c5030d440b82
|
span=c5bba5462e2f
|
span=2084d663d40e
|
span=10162e9148b0
|
buluoray
left a comment
There was a problem hiding this comment.
Summary
Reviewed head 4862fcf8f39d8149a1bae1c0bbf5ff988b4290f4, which is 2 commits: c6edfc8 from the author and 4862fcf from the maintainer. The fix holds up. Before it, build_done_probe treated any id missing from _agents as done. Now it asks SubagentManager.is_queued first, and that reads three in-memory sources: _queue, _dispatch_window_ids, and the new TaskStore.is_unstarted index. A queued spawn therefore keeps the caller's serial guard, which closes #15668. Every new error path fails closed. When the code is unsure, the guard stays held until SPAWN_TIMEOUT_MS. It never lets a duplicate through. The one exception is the explicit fallback for a manager with no is_queued. That fallback keeps the old "untracked = done" contract, and the maintainer's disposition says it is there on purpose.
CI on the head SHA: 99 success, 10 skipped, 2 neutral, 0 failed. Auto-merge is off. No GitHub review has been posted yet.
The PR conflicts with current main. git merge-tree --write-tree --name-only origin/main <head> shows conflicts in exactly two files:
src/kiro_crew/subagent_manager/admission/pump.py, at the pop site in_drain_queue_sync_impl. Main rewrote the comment line just underself._manager._dispatching_ids.add(queued_id)("re-emit its" became "ask for its"). This PR addsself._manager._dispatch_window_ids.add(queued_id)on the line next to it.src/kiro_crew/taskq/store.py, next to_SQL_UNSTARTED. Main added_SQL_CHILD_ONLYright after that line. This PR rewrites that line to come from the new_UNSTARTEDfrozenset.
Both conflicts are adjacent-line conflicts. They come from #16191 ("keep the queued count exact and cheap"), not from #16020. #16020 (chat-start priority queue) changes neither pump.py nor store.py. It also does not interact with the serial-guard logic. A subagent waiting at the new PrioritySemaphore already has an _agents row, so the probe reads info.done == False and holds the guard without consulting is_queued. The correct resolution keeps both sides in both files:
- In
pump.py, keep the_dispatch_window_ids.add(queued_id)line inside theif queued_id:block, and take main's comment text. The trap is mis-indenting the add outside theif. That would add""to the set. It would not hold the guard wrongly, becauseis_queued("")returns False early, but it is still wrong. - In
store.py, keep_UNSTARTED, the derived_SQL_UNSTARTED, and main's_SQL_CHILD_ONLY.
I checked for semantic interaction with #16191 in the auto-merged result:
- Every
UPDATE tasks SET state=site in the mergedstore.pyis still covered by_note_state:claim(exempt by design),transition,cancel, and bothwake_waitbranches. - #16191's new
TaskStore.post()only changes when a store method runs, not which method runs. It does not bypass the write-through. _dispatching_idshas the same discard sites on main as on the PR's base (gate.py failed-claim, pump_unmark_dispatching, the not-a-claim branch, the pop add). The PR's_dispatch_window_idspairing is therefore still complete after the rebase.
Verified
- Probe logic.
src/kiro_crew/apps/spawn_sdk.py:240-248: a tracked id answersinfo.done; an untracked but queued id answers False; otherwise True. is_queued_impl.src/kiro_crew/subagent_manager/run.py:381-420: returns False for a live_agentsid, skips_resume_identries, then checks the window set, thenstore.is_unstarted. It does no SQLite read on the loop (is_unstartedtakes only_unstarted_lock).- Store write-through.
src/kiro_crew/taskq/store.py:_note_stateruns after the COMMIT inaccept,insert_if_absent,transition,cancelandwake_wait. It is skipped on rollback and onrowcount != 1early returns, because those paths return before reaching it.claimstays inside the set, so it needs no write. I grepped everyUPDATE tasksstatement. The only ones that do not write state are lease, progress, wait_json, next_run_at and params updates, and none of them change set membership. - "Exactly one TaskStore writes tasks.db" (maintainer's GPT disposition). Confirmed:
taskq/__init__.py:195is the only writer, anddoctor_checks/workload.py:103opens withdiagnostic=Truefor reads only. - Non-durable pressure deferral.
gate.py_deferredrefuses when no durable row exists. A non-durable row therefore never sits in_parked_deferswhile unnamed, so there is no fail-open gap there. - Mutation testing on the targeted files only (
test_spawn_done_probe_queued.py,test_taskq_store.py,test_taskq_admission_integration.py,test_subagent.py,-kscoped). The baseline passed. Each mutation was restored afterwards andgit statuswas clean:- Probe ignores
is_queued→ 12 red, includingtest_a_queued_run_holds_the_guard_despite_no_agents_rowandtest_is_queued_names_a_store_only_overflow_row. is_queuedskips the store → 10 red (test_is_queued_holds_for_a_spawn_deferred_at_accept[*],test_is_queued_holds_for_a_row_deferred_at_drain_time, …).is_queuedskips the window set →test_unmark_dispatching_keeps_the_window_for_a_retained_claimred._note_stateremoved fromtransition/wake_wait→test_unstarted_index_follows_every_state_writered._note_stateremoved fromcancel→ 5 red, includingtest_is_queued_releases_a_store_only_row_ended_without_running[boundary_cancel|cancel_tree]._note_stateremoved fromaccept→ 13 red.- Seed at
openremoved →test_unstarted_index_is_seeded_from_disk_at_openred. _unmark_dispatchingignoresretained→test_unmark_dispatching_keeps_the_window_for_a_retained_claimred.retry_retained_claimsnever discards the window →test_claim_revalidation_outage_retains_generation_and_slot_until_retryred.- Pop-time
_dispatch_window_ids.add(queued_id)removed → nothing reddens (see follow-ups).
- Probe ignores
- I ran no tests locally beyond these targeted files and mutations. I did not run the full suite.
- Specs.
docs/system-specs/modules/subagent.md(new "An id stays pending wherever the row waits" bullet) andtaskq.md(bounded dispatch window) are updated in the same change, as AGENTS.md requires.
Bot claims disposition
- GPT FINDING
run.py:400("every path that ends one … unnames it" contradicts the stale cross-connection view). Not real as a defect. Only one writingTaskStoreexists, and I verified that above. The wording could be tighter, but it does not gate. - Design CONCERNS: description omits
TaskStore.is_unstarted/ names_dispatching_params. Was real. It is now resolved: the current PR body namesis_unstarted,_unstarted_idsand_dispatch_window_ids, and_dispatching_paramsno longer appears in the code or the body. - Design suggestion: drop
retained=because the store index already covers retained claims. Correct that it is redundant today. Retention only happens after a committed claim, which leaves the rowadmittedand therefore in the index. The maintainer's reason for keeping it (a failed seed read, or a future non-durable retain) is reasonable. Advisory, does not gate. - First-Principles CONCERNS item 3 (undeclared store index). Was real. It is now resolved by the description update. Item 7 (the
is_queuedfallback has no consumers) is kept with a sound reason: without it,SpawnSDK.is_donewould swallow anAttributeErrorand hold every guard until timeout. - Opus: no findings. I agree. Nothing blocking survived verification.
Follow-ups (non-blocking)
- No test pins the pop-time window add. Deleting
self._manager._dispatch_window_ids.add(queued_id)(pump.py:664) reddens nothing in the targeted files. The retained-claim outage test seeds the set by hand. For a durable row the store index masks the gap. For a popped non-durable row (incognito / temporary) that add is the only record between pop and registration, and nothing exercises that path. A test should pop an incognito queued row, stop before registration, and assertis_queuedis True. - Possible stale window id. In
retry_retained_claims(pump.py:191-213), aclaim_and_startthat raises skips the_dispatch_window_ids.discard. The id then stays named forever, even after the store row goes terminal. This fails closed: the probe holds the caller's guard until timeout. Wrap the discard intry/finally. - Rebase onto current main. It is conflicting. The resolution notes above cover
pump.pyandstore.py. #16191 is the cause, not #16020. - Commit count. The PR carries 2 commits (author's plus maintainer's). If PR Hygiene's single-commit rule applies at merge, squash on merge.
- Optional wording. GPT's
run.py:400docstring could say "every path through this store instance". Doc-only.
|
@buluoray rebased onto current main as 625fb2e: the same two commits (mine, then @bolichen97 maintainer follow-up), authorship unchanged, no content change versus the head you approved (4862fcf). Could you re-approve once CI is green? Thanks. |
A spawn admitted behind the concurrency / adaptive cap returns its real agent_id to the caller but has no _agents entry until it drains. The app serial-lock done-probe reads an untracked id as finished, so it released the guard for a queued run that had not started and let the caller queue a duplicate of that same work. SubagentManager grows is_queued(agent_id): true while a fresh spawn waits in _queue, and across the pump's pop-to-claim / retained-claim window, which it now tracks in _dispatching_params (kept in lockstep with _dispatching_ids and retained across a retained claim until the retry registers or refuses it). The done-probe consults it: an untracked id is done unless it is queued. The existing queued read surfaces (?queued=1, queued_run_async, and _queued_run_payload) already answer a queued id to spawn_status / spawn_list, so no new observability projection is added. Fixes kirodotdev#15668
The done-probe released the serial guard for a durable row that was waiting on disk with no _queue entry and no _agents row, because the manager-side id set was filled at only two of the places a row can end up waiting there. A pressure deferral at accept (posture-critical or low-memory, from spawn or spawn_async) and a deferral at drain time (park_defer) added nothing, so is_queued read the row as finished. The set also kept ids that the store ended without running them, such as a boundary cancel or a cancel-tree. For those ids the probe held the guard until its timeout. There were two problems: some paths that leave a row waiting never added it, and some paths that end a row never removed it. A manager-side set cannot track either reliably, because every path that moves a row would have to remember to update it. The store already commits each of those transitions, so the store now keeps the record. TaskStore keeps an in-memory index of the ids in a claimable or admitted state. It loads the index at open and updates it after the commit of every write that moves a row into or out of that set: accept, insert_if_absent, transition, cancel and wake_wait. is_queued reads that index for durable rows, with a lock and no SQLite access, because the probe runs on the gateway loop. _accepted_unstarted_ids and its add and drop sites in the gate, the bridge and the eviction are gone. Shrink the dispatch window to an id set. Nothing read the params values, and the comment justifying them cited a reader that does not exist.
bolichen97
left a comment
There was a problem hiding this comment.
LGTM. Checked the main P0 risk, a guard stuck forever: every TaskStore write that moves a row out of claimable/admitted (transition, cancel, wake_wait) calls _note_state after COMMIT, there is no DELETE path, and open() re-seeds the index. The window ids drop when a retained claim registers or is refused. I see no leak that would hold a serial lock and block spawns.
Problem / Motivation
Goal: A spawn still waiting behind the concurrency cap is treated as
pending work, so a caller's serial lock is never released for it and no
duplicate of that not-yet-run work can be queued.
A spawn admitted behind the concurrency / adaptive cap returns its real
agent_idto the caller but has no_agentsentry until it drains. The appserial-lock done-probe (
build_done_probe) reads an id the manager does nottrack as finished, so for a queued spawn — pending work that has not
started — it released the caller's serial guard. The guard's whole purpose is
to stop a duplicate being queued, so clearing it let the caller enqueue a
second copy of that same not-yet-run work (
#15668).Why it matters
A duplicate unattended run is silent and not self-correcting: the serial lane
re-reads the still-undone task and spawns it again. The done-probe runs on a
short interval, so the pop-to-claim window is sampled repeatedly.
Not a goal
Adding a new queued-observability surface.
spawn_status/spawn_listalready answer a queued id (
?queued=1,queued_run_async,_queued_run_payload); this change does not duplicate that. It fixes only theserial-guard done-probe.
What changed (motivation → approach → change)
The gap is that the done-probe cannot tell "untracked because finished" from
"untracked because still queued".
SubagentManagergrowsis_queued(agent_id),and the done-probe consults it: an untracked id is done unless it is
queued, so a queued run holds the guard.
is_queuedanswers from threeplaces, all in memory, because the probe is a sync call on the gateway loop
where a SQLite read is forbidden:
_queue: a fresh spawn waiting in the in-memory dispatch window._dispatch_window_ids(new, insubagent.py): the pump'spop-to-claim / retained-claim window. It is marked with
_dispatching_idsand kept across a retained claim, through both the per-row and the
drain-pass cleanup, until the retry registers or refuses the row. It is the
only record of a popped non-durable row (
incognito/temporary, or nostore at all).
TaskStore.is_unstarted(new, intaskq/store.py): an in-memoryindex (
_unstarted_ids) of the rows in a claimable oradmittedstate. Itis seeded from disk at
openand written through by_note_stateafterthe
COMMITof every state write (accept,accept_one,transition,cancel,wake_wait).claimskips it because claimable toadmittedstays inside the set. This covers the rows that wait on disk with no
_queueor_agentsentry: the cap's store-only overflow, a pressuredeferral at accept or at drain time, and a window eviction. Every path that
ends a row (settle, cancel, boundary cancel, cancel-tree, wait expiry)
removes it, so the manager keeps no per-path bookkeeping that could miss a
path. Every future store state write must keep calling
_note_state. Afailed seed read logs a warning and leaves the index empty instead of
refusing the open.
subagent.md,taskq.mdand the store module docstring document the indexand the window in the same commit.
Backwards compatibility
No wire or API change.
is_queuedis additive; the done-probe degrades to theprior contract (untracked = done) for a manager that does not expose it.
Tests
test/test_spawn_done_probe_queued.py: the probe contract. A tracked runreports its done flag. An untracked id is done unless
is_queued. Thequeued and dispatching-window cases hold the guard. A manager without
is_queuedis tolerated.test/test_subagent.py:is_queuednames_queueand the_dispatch_window_idswindow and skips resume / id-less rows._unmark_dispatchingkeeps the window for a retained claim and drops it oncethe claim settles.
test/test_taskq_store.py::test_unstarted_index_*: the index follows everystate write, ignores a write that did not commit, is seeded from disk at
open, and is read on the loop without taking a connection.
test/test_taskq_admission_integration.py: the retained-claim outage path(
is_queuedholds through the retention window and clears after the retryregisters the run), plus
test_is_queued_*for the store-only overflow row,every store-only transition (sync and async spawn), a deferral at accept, a
deferral at drain time, and a store-only row that ends without running.
Manual verification
Ran the affected suites with
-n0plus black (py312) / isort / flake8 / mypy.Screenshots / video
N/A — no user-visible surface.
Related Issues
Fixes #15668
Pattern harvest
Rule candidate: a serial-lock / idempotency done-probe must distinguish
"untracked because finished" from "untracked because not yet started" — a bare
registry miss conflates them and releases the guard for pending work.
Checklist
Contribution License Agreement
By submitting this pull request, I confirm that my contribution is made under
the terms of the project's license.