Skip to content

fix(subagent): hold the serial guard for a queued spawn's done-probe - #15869

Merged
bolichen97 merged 2 commits into
kirodotdev:mainfrom
chenmingwei23:fix/spawn-queued-run-invisible-15668
Oct 3, 2026
Merged

bolichen97 merged 2 commits into
kirodotdev:mainfrom
chenmingwei23:fix/spawn-queued-run-invisible-15668

Conversation

@chenmingwei23

@chenmingwei23 chenmingwei23 commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A spawn still waiting behind the concurrency cap is treated as
pending work, so a caller's serial lock is never released for it and no
duplicate of that not-yet-run work can be queued.

A spawn admitted behind the concurrency / adaptive cap returns its real
agent_id to the caller but has no _agents entry until it drains. The app
serial-lock done-probe (build_done_probe) reads an id the manager does not
track as finished, so for a queued spawn — pending work that has not
started — it released the caller's serial guard. The guard's whole purpose is
to stop a duplicate being queued, so clearing it let the caller enqueue a
second copy of that same not-yet-run work (#15668).

Why it matters

A duplicate unattended run is silent and not self-correcting: the serial lane
re-reads the still-undone task and spawns it again. The done-probe runs on a
short interval, so the pop-to-claim window is sampled repeatedly.

Not a goal

Adding a new queued-observability surface. spawn_status / spawn_list
already answer a queued id (?queued=1, queued_run_async,
_queued_run_payload); this change does not duplicate that. It fixes only the
serial-guard done-probe.

What changed (motivation → approach → change)

The gap is that the done-probe cannot tell "untracked because finished" from
"untracked because still queued". SubagentManager grows is_queued(agent_id),
and the done-probe consults it: an untracked id is done unless it is
queued, so a queued run holds the guard. is_queued answers from three
places, all in memory, because the probe is a sync call on the gateway loop
where a SQLite read is forbidden:

  1. _queue: a fresh spawn waiting in the in-memory dispatch window.
  2. _dispatch_window_ids (new, in subagent.py): the pump's
    pop-to-claim / retained-claim window. It is marked with _dispatching_ids
    and kept across a retained claim, through both the per-row and the
    drain-pass cleanup, until the retry registers or refuses the row. It is the
    only record of a popped non-durable row (incognito / temporary, or no
    store at all).
  3. TaskStore.is_unstarted (new, in taskq/store.py): an in-memory
    index (_unstarted_ids) of the rows in a claimable or admitted state. It
    is seeded from disk at open and written through by _note_state after
    the COMMIT of every state write (accept, accept_one, transition,
    cancel, wake_wait). claim skips it because claimable to admitted
    stays inside the set. This covers the rows that wait on disk with no
    _queue or _agents entry: the cap's store-only overflow, a pressure
    deferral at accept or at drain time, and a window eviction. Every path that
    ends a row (settle, cancel, boundary cancel, cancel-tree, wait expiry)
    removes it, so the manager keeps no per-path bookkeeping that could miss a
    path. Every future store state write must keep calling _note_state. A
    failed seed read logs a warning and leaves the index empty instead of
    refusing the open.

subagent.md, taskq.md and the store module docstring document the index
and the window in the same commit.

Backwards compatibility

No wire or API change. is_queued is additive; the done-probe degrades to the
prior contract (untracked = done) for a manager that does not expose it.

Tests

  • test/test_spawn_done_probe_queued.py: the probe contract. A tracked run
    reports its done flag. An untracked id is done unless is_queued. The
    queued and dispatching-window cases hold the guard. A manager without
    is_queued is tolerated.
  • test/test_subagent.py: is_queued names _queue and the
    _dispatch_window_ids window and skips resume / id-less rows.
    _unmark_dispatching keeps the window for a retained claim and drops it once
    the claim settles.
  • test/test_taskq_store.py::test_unstarted_index_*: the index follows every
    state write, ignores a write that did not commit, is seeded from disk at
    open, and is read on the loop without taking a connection.
  • test/test_taskq_admission_integration.py: the retained-claim outage path
    (is_queued holds through the retention window and clears after the retry
    registers the run), plus test_is_queued_* for the store-only overflow row,
    every store-only transition (sync and async spawn), a deferral at accept, a
    deferral at drain time, and a store-only row that ends without running.

Manual verification

Ran the affected suites with -n0 plus black (py312) / isort / flake8 / mypy.

Screenshots / video

N/A — no user-visible surface.

Related Issues

Fixes #15668

Pattern harvest

Rule candidate: a serial-lock / idempotency done-probe must distinguish
"untracked because finished" from "untracked because not yet started" — a bare
registry miss conflates them and releases the guard for pending work.

Checklist

  • Tests cover the change
  • Lint / type checks pass
  • No new user-visible surface

Contribution License Agreement

By submitting this pull request, I confirm that my contribution is made under
the terms of the project's license.

@chenmingwei23
chenmingwei23 requested a review from a team as a code owner October 1, 2026 13:00
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 94e06897c6f1fa61dbca620bc833d21cf829273f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 94e0689

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 94e06897c6f1fa61dbca620bc833d21cf829273f via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check before merge: any future state write in TaskStore that skips _note_state makes the done-probe read the wrong answer, and no test fails when that happens.

[DESIGN-REVIEWED] 94e0689

Description read

[DESCRIPTION-READ] a62e8b609256f1e99213ae24a82cb7388548b54fae6732b0f9e2a46c2f05224f

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — 🟡 CONCERNS

Premise-level review of 94e06897c6f1fa61dbca620bc833d21cf829273f via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

_dispatch_window_ids repeats _dispatching_ids. Its one difference, keeping retained claims, is already covered by TaskStore.is_unstarted.

Not justified as shipped

  • Item 3 — duplicate of _dispatching_ids (src/kiro_crew/subagent.py:3645): the description says the window "outlives it across a retained claim". A retained claim is a row "already ADMITTED under this process" (pump.py:1175). ADMITTED is in _UNSTARTED, and claim keeps the row in the index. So item 4 already names every retained id. Non-durable rows get (0, True, "") from taskq_claim and are never retained. That leaves the new set equal to _dispatching_ids on every path the store index does not cover. I grepped _dispatch_window_ids in the diff: 1 add, 4 discards, 1 reader, plus the retained= plumbing in 3 pump sites.
    Subtraction: delete _dispatch_window_ids, the retained parameter of _unmark_dispatching, and the retry-path discard. is_queued_impl reads _dispatching_ids instead.
    Clears when: the set is removed, or a path is shown where an id is in _dispatch_window_ids but in neither _dispatching_ids nor TaskStore.is_unstarted.

What this change ships

Inventory (5 items) — 4 justified

Intent: an app's serial lock keeps holding while its spawn is still queued, so the same work can't be queued twice. FIX (#15668).

  1. The serial-lock done-probe treats an untracked but queued spawn as not done — justified
  2. SubagentManager.is_queued(id), a new public method that answers from memory — justified
  3. A pump-side window set kept across retained claims — duplicate of _dispatching_ids plus the store index
  4. TaskStore.is_unstarted, an in-memory index of unstarted ids that every state write updates — justified
  5. The spec and docstrings for subagent.md and taskq.md are updated — justified

[FIRST-PRINCIPLES-REVIEWED] 94e0689

Description read

[DESCRIPTION-READ] 8122220bc764c7dc2e40934042a0805ffea5b1b36cdf38db739bd06ca22aaf81

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 94e06897c6f1fa61dbca620bc833d21cf829273f via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 94e0689

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/spawn-queued-run-invisible-15668 branch from d6709ae to 144effe Compare October 1, 2026 14:08
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 1, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/spawn-queued-run-invisible-15668 branch from 144effe to 3e4e917 Compare October 1, 2026 14:56
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 1, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/spawn-queued-run-invisible-15668 branch from b578439 to c3b97dd Compare October 2, 2026 08:09
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review opus

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 2, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/spawn-queued-run-invisible-15668 branch from c3b97dd to f30c6cf Compare October 2, 2026 16:26
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review opus

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: checking Automated validation is still running labels Oct 2, 2026
@github-actions github-actions Bot added the readiness: action required A blocking check or review needs attention label Oct 2, 2026
@chenmingwei23
chenmingwei23 force-pushed the fix/spawn-queued-run-invisible-15668 branch from f5611b0 to c6edfc8 Compare October 2, 2026 23:20
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 2, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

Maintainer follow-up pushed as 9645514600 on top of the author's c6edfc8ea0. The author's commit and authorship are unchanged, so the PR now has 2 commits and is not rebased (no conflict with main).

What changed and why

  1. Eviction gap (a real defect inside this PR's own contract). _evict_for_lanes moves a windowed, cap-queued row back to store-only. Before this change, nothing recorded the row there: it was in neither _queue, _dispatch_window_ids nor _accepted_unstarted_ids. So is_queued returned False, the done-probe read the id as finished, and the caller's serial guard was released. This is the duplicate-queue failure spawn_run results never reach the session: no completion event, spawn_status "not found", spawn_list empty #15668 describes, and it is reachable under ordinary multi-lane load. The eviction now adds the id to the set, and the refill removes it again when the row rejoins the window.
  2. Where store-only is recorded. The add moved from spawn_async (if not window_hint) to the gate's queue branch, the one place a durable row actually lands store-only. That branch covers sync spawn as well. It also stops the old add from firing for a row that then started or was refused, which left the id in the set.
  3. First-principles item 6. _dispatching_params: dict became _dispatch_window_ids: set[str]. Nothing read the values, and the comment justifying them cited _queued_info_for, which does not exist.

Tests. test_is_queued_follows_a_row_across_every_store_only_transition (parametrized over spawn and spawn_async) runs the real gate, the eviction and the refill. I checked that it fails without each fix: the gate assertion fails if gate.py is reverted, and the eviction assertion fails if taskq_bridge.py is reverted. The targeted suites pass locally (91 files, 3398 passed). Black, isort, flake8 and mypy are clean.

CI state on the previous heads

  • Backend Tests (3.12, 8) / (Windows) (8) / Coverage Gate on f5611b0: a real defect. taskq_settle reached self._manager on the bridge doubles in test_subagent_dependency_mark.py. The author's c6edfc8 fixes it with _drop_accepted_unstarted, which I kept.
  • GPT / Opus / Design / First Principles / UX / Security Scope / Internal Content Scan on f5611b0 all reported "could not run". The job logs show the resolver's PR-list query failing on all 3 attempts, which is GitHub API rate-limit exhaustion, not this diff. Those lanes re-run on the new head.
  • The 4309bb blocking findings (GPT F1 / Opus F2 / the Design watch: a sync state_of on the loop; Opus F1: the unbound logger) were already fixed by the author at f5611b0, because the store branch is gone. test_is_queued_names_a_store_only_overflow_row patches state_of to raise.

Out of scope, for a possible follow-up. A low-memory deferral (_deferred, QUEUED_REASON_LOW_MEMORY) also leaves an accepted row in neither _queue nor _agents. This PR's goal names the concurrency/adaptive cap only, so I did not widen it.

@bolichen97

Copy link
Copy Markdown
Contributor
span=f6e70c057000
  • Item 6: _dispatching_params stores params dicts nobody reads

Fixed in 9645514. The dict is now _dispatch_window_ids: set[str], and every access is add, discard or membership. The comment no longer cites the nonexistent _queued_info_for. It now says the set outlives _dispatching_ids only across a retained claim.

@bolichen97

Copy link
Copy Markdown
Contributor
span=549f6a24a0f7
  • Item 7: the probe's fallback for a manager without is_queued has no consumers

Kept on purpose. Without the fallback, a manager lacking is_queued makes _probe raise AttributeError, and SpawnSDK.is_done swallows that and answers "not done". So every untracked id would hold the caller's serial lock until SPAWN_TIMEOUT_MS, which is the stall this PR removes. The fallback keeps the probe total and fails toward the existing contract (untracked = done). It also matches the idiom of the function it lives beside: build_spawn_impl takes the same duck-typed subagents: object and already branches on getattr(subagents, "spawn_async", None), with the comment "a manager double without it spawns inline". One getattr costs nothing. Removing it makes the probe's correctness depend on every manager-shaped object the SDK is handed.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 2, 2026
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review opus

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review scope

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

/ai-review gpt

@bolichen97
bolichen97 force-pushed the fix/spawn-queued-run-invisible-15668 branch from 9645514 to 4862fcf Compare October 3, 2026 00:59
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor
span=10162e9148b0
  • BLOCKING run.py:414: a store-only row the store ends without running it stays in is_queued

Fixed in the new head. The defect was real. _accepted_unstarted_ids was dropped at only three sites (register, refill, settle and queued cancel). An exact boundary cancel (taskq_cancel_boundary_store), a parent's cancel-tree (WaitLedger.cancel_tree) and wait expiry each end a row without touching it, so the probe held the caller's serial guard until SPAWN_TIMEOUT_MS.

I did not add more drop sites. The manager set is gone. TaskStore now keeps an in-memory index of the ids in a claimable or admitted state (is_unstarted). It is updated after the commit of every write that moves a row into or out of that set: accept, insert_if_absent, transition (and so finish), cancel and wake_wait. It is reloaded at open. Every terminal write goes through transition or cancel, so a terminal row always leaves the index. The same index also names the rows that the set never added: a pressure deferral at accept, from spawn or spawn_async, and a deferral at drain time. is_queued reads it under its own lock and never takes SQLite or _lock.

Tests that fail on 9645514 and pass now:

  • test_taskq_admission_integration.py::test_is_queued_releases_a_store_only_row_ended_without_running[boundary_cancel|cancel_tree]
  • ::test_is_queued_holds_for_a_spawn_deferred_at_accept[sync|async × posture|low_memory]
  • ::test_is_queued_holds_for_a_row_deferred_at_drain_time
  • test_taskq_store.py::test_unstarted_index_* (4)

I checked the store pins by mutation. Each test goes red when the index write in accept, transition, cancel or wake_wait is deleted, or when the set is widened to "not terminal".

@bolichen97

Copy link
Copy Markdown
Contributor

Maintainer follow-up: I replaced my own commit 9645514 with a reworked second commit on top of the author's c6edfc8ea0. The author's commit and authorship are unchanged, the PR still has 2 commits, and it was not rebased because there is no conflict with main.

Why the rework

  1. A blocking regression on 9645514 (post-push review). A durable row can be left QUEUED on disk with no _queue entry and no _agents row by a pressure deferral. That happens at accept (_deferred: posture-critical or low-memory, from sync spawn and from spawn_async) and at drain time (park_defer, after _unmark_dispatching). None of those paths added the row to _accepted_unstarted_ids, so is_queued returned False and the done-probe released the serial guard. That is the spawn_run results never reach the session: no completion event, spawn_status "not found", spawn_list empty #15668 bug class again.
  2. GPT's blocking finding on the same head. The set was dropped at too few places. An exact boundary cancel, a parent's cancel-tree and wait expiry end a store-only row without dropping it, so the probe held the guard until its timeout.

These are the same defect in two directions. A manager-side set has to be updated by every path that moves a row, and both directions had missed paths. The store already commits every one of those transitions, so it now keeps the record itself.

What changed

  • TaskStore keeps an in-memory index of the ids in a claimable or admitted state (is_unstarted, the same set _SQL_UNSTARTED / count_pending(include_admitted=True) already define). It is loaded at open and updated after the commit of every write that crosses that set: accept, insert_if_absent, transition (and so finish / advance / enter_wait), cancel and wake_wait. A rolled-back write never reaches it. The read takes its own lock only, never _lock or _c(), so the done-probe on the gateway loop does no SQLite I/O and does not trip the strict on-loop guard.
  • is_queued is False for an id with a live _agents row. Otherwise it is True when _queue names the id, when _dispatch_window_ids names it (the only record of a popped non-durable row), or when store.is_unstarted(id) is true.
  • _accepted_unstarted_ids, _drop_accepted_unstarted and their add and drop sites in gate.py, taskq_bridge.py and the eviction are removed. Those two files are now byte-identical to main.
  • Specs updated in the same commit. subagent.md § Fairness lanes gains a new bullet, "An id stays pending wherever the row waits (is_queued)": which transitions add to or drop from _queue, _dispatch_window_ids and the store index. taskq.md § Bounded dispatch window now names the index as the store's one in-memory fact.

Tests (each fails on 9645514, verified on a detached worktree at that sha)

  • test_is_queued_holds_for_a_spawn_deferred_at_accept[sync|async × posture|low_memory]: the posture-critical spawn_async probe and the sync spawn deferral.
  • test_is_queued_holds_for_a_row_deferred_at_drain_time: the drain-time deferral through the real pump.
  • test_is_queued_releases_a_store_only_row_ended_without_running[boundary_cancel|cancel_tree]: GPT's finding.
  • test_is_queued_names_a_store_only_overflow_row and test_is_queued_follows_a_row_across_every_store_only_transition (cap store-only branch and eviction): now with no hand-replicated bookkeeping, and with _c patched to fail, so any SQLite read fails the test.
  • test_taskq_store.py::test_unstarted_index_* (4). By mutation, each goes red when the index write in accept, transition, cancel or wake_wait is deleted, or when the set is widened to "not terminal". claim writes nothing to the index, because claimable to admitted stays inside it.

The targeted suites pass locally (taskq store, admission integration, done-probe, subagent, fairness lanes, gate, dependency mark: 408 passed). black, isort, flake8, mypy, docs-lint and the harness-parity gate are clean.

CI on 9645514. Opus 5 ("could not run") and Security Scope ("no leg reported") are infrastructure failures, not findings. GPT 5.6's one blocking finding is fixed above and has its own disposition.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 3, 2026
@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor
span=c5030d440b82
  • Watch: the description omits TaskStore.is_unstarted and names the stale _dispatching_params

Fixed in the PR description (no code change). "What changed" now lists all three sources is_queued reads: _queue, _dispatch_window_ids, and TaskStore.is_unstarted with its _unstarted_ids index. For the index it covers the seed at open, the _note_state write-through after each committed write, why claim skips that write, which store-only rows it covers, and the rule that every future store write keeps calling _note_state. "Tests" now names test_taskq_store.py::test_unstarted_index_* and the test_is_queued_* integration tests. The string _dispatching_params no longer appears.

@bolichen97

bolichen97 commented Oct 3, 2026 •

Copy link
Copy Markdown
Contributor
span=c5bba5462e2f
  • Item 3 (undeclared): the description does not mention the TaskStore.is_unstarted index and still names _dispatching_params

Fixed in the PR description (no code change). "What changed" now declares the TaskStore unstarted-row index as item 3 of is_queued's sources. It covers _unstarted_ids, the seed at open, the _note_state write-through after every committed state write (accept, accept_one, transition, cancel, wake_wait), why claim skips it, why it must be in memory (the sync done-probe runs on the gateway loop), and the permanent rule that every future store state write keeps calling _note_state. "Tests" now names test_taskq_store.py::test_unstarted_index_*. The stale name _dispatching_params is gone. The description now uses _dispatch_window_ids throughout.

@bolichen97

Copy link
Copy Markdown
Contributor
span=2084d663d40e
  • Suggestion: drop retained= from _unmark_dispatching and let the store index cover the retained window

Kept, by choice. Today a retained claim is always durable and already committed as admitted (claim_and_start asserts a store and claims first), so the index names it in the normal case. The flag covers two cases the index does not:

  1. _load_unstarted_ids is best effort by design. A failed seed read logs a warning and leaves the index empty so that open does not refuse every spawn. If a row from an earlier run is popped, claimed and then retained during that degraded state, _dispatch_window_ids is its only in-memory record. claim never writes the index, so the claim does not fill that gap.
  2. With the flag, is_queued stays correct for every row the pump popped without relying on an invariant in another component (that the pump never retains a non-durable row). Without it, a later change that retains a non-durable row would silently put back spawn_run results never reach the session: no completion event, spawn_status "not found", spawn_list empty #15668.

The cost is one keyword argument and one membership test at each of the two cleanup sites. test_unmark_dispatching_keeps_the_window_for_a_retained_claim and the retained-claim outage test in test_taskq_admission_integration.py pin the behavior. The change would also need another full CI and review round for an advisory simplification, and the Actions queue is saturated right now.

@bolichen97

Copy link
Copy Markdown
Contributor
span=10162e9148b0
  • FINDING run.py:400: "every path that ends one ... unnames it" contradicts the store's stale cross-connection view

Not changed. This is a docstring-wording finding, and the sentence is accurate for every connection that writes this file. Exactly one TaskStore writes tasks.db: the gateway's own, built at taskq/__init__.py:195. That is the instance is_queued_impl reads as self._manager._taskq. The only other constructor, doctor_checks/workload.py:103, is TaskStore(path, diagnostic=True) and only reads (doctor_lines, count_by_state). Every path the docstring lists (settle, cancel, boundary cancel, cancel-tree, wait expiry) is a method on that one instance, and each calls _note_state after its own COMMIT. The same docstring also says the index is "written through by the same commit that moves the row", which already scopes it to this connection. The is_unstarted docstring gives the cross-connection caveat for a writer that does not exist today. Changing only this wording would restart every CI job and AI-review lane while the Actions queue is saturated, for no change in behavior.

buluoray
buluoray previously approved these changes Oct 3, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary

Reviewed head 4862fcf8f39d8149a1bae1c0bbf5ff988b4290f4, which is 2 commits: c6edfc8 from the author and 4862fcf from the maintainer. The fix holds up. Before it, build_done_probe treated any id missing from _agents as done. Now it asks SubagentManager.is_queued first, and that reads three in-memory sources: _queue, _dispatch_window_ids, and the new TaskStore.is_unstarted index. A queued spawn therefore keeps the caller's serial guard, which closes #15668. Every new error path fails closed. When the code is unsure, the guard stays held until SPAWN_TIMEOUT_MS. It never lets a duplicate through. The one exception is the explicit fallback for a manager with no is_queued. That fallback keeps the old "untracked = done" contract, and the maintainer's disposition says it is there on purpose.

CI on the head SHA: 99 success, 10 skipped, 2 neutral, 0 failed. Auto-merge is off. No GitHub review has been posted yet.

The PR conflicts with current main. git merge-tree --write-tree --name-only origin/main <head> shows conflicts in exactly two files:

  • src/kiro_crew/subagent_manager/admission/pump.py, at the pop site in _drain_queue_sync_impl. Main rewrote the comment line just under self._manager._dispatching_ids.add(queued_id) ("re-emit its" became "ask for its"). This PR adds self._manager._dispatch_window_ids.add(queued_id) on the line next to it.
  • src/kiro_crew/taskq/store.py, next to _SQL_UNSTARTED. Main added _SQL_CHILD_ONLY right after that line. This PR rewrites that line to come from the new _UNSTARTED frozenset.

Both conflicts are adjacent-line conflicts. They come from #16191 ("keep the queued count exact and cheap"), not from #16020. #16020 (chat-start priority queue) changes neither pump.py nor store.py. It also does not interact with the serial-guard logic. A subagent waiting at the new PrioritySemaphore already has an _agents row, so the probe reads info.done == False and holds the guard without consulting is_queued. The correct resolution keeps both sides in both files:

  • In pump.py, keep the _dispatch_window_ids.add(queued_id) line inside the if queued_id: block, and take main's comment text. The trap is mis-indenting the add outside the if. That would add "" to the set. It would not hold the guard wrongly, because is_queued("") returns False early, but it is still wrong.
  • In store.py, keep _UNSTARTED, the derived _SQL_UNSTARTED, and main's _SQL_CHILD_ONLY.

I checked for semantic interaction with #16191 in the auto-merged result:

  • Every UPDATE tasks SET state= site in the merged store.py is still covered by _note_state: claim (exempt by design), transition, cancel, and both wake_wait branches.
  • #16191's new TaskStore.post() only changes when a store method runs, not which method runs. It does not bypass the write-through.
  • _dispatching_ids has the same discard sites on main as on the PR's base (gate.py failed-claim, pump _unmark_dispatching, the not-a-claim branch, the pop add). The PR's _dispatch_window_ids pairing is therefore still complete after the rebase.

Verified

  • Probe logic. src/kiro_crew/apps/spawn_sdk.py:240-248: a tracked id answers info.done; an untracked but queued id answers False; otherwise True.
  • is_queued_impl. src/kiro_crew/subagent_manager/run.py:381-420: returns False for a live _agents id, skips _resume_id entries, then checks the window set, then store.is_unstarted. It does no SQLite read on the loop (is_unstarted takes only _unstarted_lock).
  • Store write-through. src/kiro_crew/taskq/store.py: _note_state runs after the COMMIT in accept, insert_if_absent, transition, cancel and wake_wait. It is skipped on rollback and on rowcount != 1 early returns, because those paths return before reaching it. claim stays inside the set, so it needs no write. I grepped every UPDATE tasks statement. The only ones that do not write state are lease, progress, wait_json, next_run_at and params updates, and none of them change set membership.
  • "Exactly one TaskStore writes tasks.db" (maintainer's GPT disposition). Confirmed: taskq/__init__.py:195 is the only writer, and doctor_checks/workload.py:103 opens with diagnostic=True for reads only.
  • Non-durable pressure deferral. gate.py _deferred refuses when no durable row exists. A non-durable row therefore never sits in _parked_defers while unnamed, so there is no fail-open gap there.
  • Mutation testing on the targeted files only (test_spawn_done_probe_queued.py, test_taskq_store.py, test_taskq_admission_integration.py, test_subagent.py, -k scoped). The baseline passed. Each mutation was restored afterwards and git status was clean:
    • Probe ignores is_queued → 12 red, including test_a_queued_run_holds_the_guard_despite_no_agents_row and test_is_queued_names_a_store_only_overflow_row.
    • is_queued skips the store → 10 red (test_is_queued_holds_for_a_spawn_deferred_at_accept[*], test_is_queued_holds_for_a_row_deferred_at_drain_time, …).
    • is_queued skips the window set → test_unmark_dispatching_keeps_the_window_for_a_retained_claim red.
    • _note_state removed from transition / wake_wait → test_unstarted_index_follows_every_state_write red.
    • _note_state removed from cancel → 5 red, including test_is_queued_releases_a_store_only_row_ended_without_running[boundary_cancel|cancel_tree].
    • _note_state removed from accept → 13 red.
    • Seed at open removed → test_unstarted_index_is_seeded_from_disk_at_open red.
    • _unmark_dispatching ignores retained → test_unmark_dispatching_keeps_the_window_for_a_retained_claim red.
    • retry_retained_claims never discards the window → test_claim_revalidation_outage_retains_generation_and_slot_until_retry red.
    • Pop-time _dispatch_window_ids.add(queued_id) removed → nothing reddens (see follow-ups).
  • I ran no tests locally beyond these targeted files and mutations. I did not run the full suite.
  • Specs. docs/system-specs/modules/subagent.md (new "An id stays pending wherever the row waits" bullet) and taskq.md (bounded dispatch window) are updated in the same change, as AGENTS.md requires.

Bot claims disposition

  • GPT FINDING run.py:400 ("every path that ends one … unnames it" contradicts the stale cross-connection view). Not real as a defect. Only one writing TaskStore exists, and I verified that above. The wording could be tighter, but it does not gate.
  • Design CONCERNS: description omits TaskStore.is_unstarted / names _dispatching_params. Was real. It is now resolved: the current PR body names is_unstarted, _unstarted_ids and _dispatch_window_ids, and _dispatching_params no longer appears in the code or the body.
  • Design suggestion: drop retained= because the store index already covers retained claims. Correct that it is redundant today. Retention only happens after a committed claim, which leaves the row admitted and therefore in the index. The maintainer's reason for keeping it (a failed seed read, or a future non-durable retain) is reasonable. Advisory, does not gate.
  • First-Principles CONCERNS item 3 (undeclared store index). Was real. It is now resolved by the description update. Item 7 (the is_queued fallback has no consumers) is kept with a sound reason: without it, SpawnSDK.is_done would swallow an AttributeError and hold every guard until timeout.
  • Opus: no findings. I agree. Nothing blocking survived verification.

Follow-ups (non-blocking)

  1. No test pins the pop-time window add. Deleting self._manager._dispatch_window_ids.add(queued_id) (pump.py:664) reddens nothing in the targeted files. The retained-claim outage test seeds the set by hand. For a durable row the store index masks the gap. For a popped non-durable row (incognito / temporary) that add is the only record between pop and registration, and nothing exercises that path. A test should pop an incognito queued row, stop before registration, and assert is_queued is True.
  2. Possible stale window id. In retry_retained_claims (pump.py:191-213), a claim_and_start that raises skips the _dispatch_window_ids.discard. The id then stays named forever, even after the store row goes terminal. This fails closed: the probe holds the caller's guard until timeout. Wrap the discard in try/finally.
  3. Rebase onto current main. It is conflicting. The resolution notes above cover pump.py and store.py. #16191 is the cause, not #16020.
  4. Commit count. The PR carries 2 commits (author's plus maintainer's). If PR Hygiene's single-commit rule applies at merge, squash on merge.
  5. Optional wording. GPT's run.py:400 docstring could say "every path through this store instance". Doc-only.

@chenmingwei23

Copy link
Copy Markdown
Contributor Author

@buluoray rebased onto current main as 625fb2e: the same two commits (mine, then @bolichen97 maintainer follow-up), authorship unchanged, no content change versus the head you approved (4862fcf). Could you re-approve once CI is green? Thanks.

kirocrew-agent and others added 2 commits October 3, 2026 10:09
A spawn admitted behind the concurrency / adaptive cap returns its real
agent_id to the caller but has no _agents entry until it drains. The app
serial-lock done-probe reads an untracked id as finished, so it released
the guard for a queued run that had not started and let the caller queue a
duplicate of that same work.

SubagentManager grows is_queued(agent_id): true while a fresh spawn waits in
_queue, and across the pump's pop-to-claim / retained-claim window, which it
now tracks in _dispatching_params (kept in lockstep with _dispatching_ids and
retained across a retained claim until the retry registers or refuses it). The
done-probe consults it: an untracked id is done unless it is queued.

The existing queued read surfaces (?queued=1, queued_run_async, and
_queued_run_payload) already answer a queued id to spawn_status / spawn_list,
so no new observability projection is added.

Fixes kirodotdev#15668
The done-probe released the serial guard for a durable row that was
waiting on disk with no _queue entry and no _agents row, because the
manager-side id set was filled at only two of the places a row can end
up waiting there. A pressure deferral at accept (posture-critical or
low-memory, from spawn or spawn_async) and a deferral at drain time
(park_defer) added nothing, so is_queued read the row as finished. The
set also kept ids that the store ended without running them, such as a
boundary cancel or a cancel-tree. For those ids the probe held the guard
until its timeout.

There were two problems: some paths that leave a row waiting never
added it, and some paths that end a row never removed it. A
manager-side set cannot track either reliably, because every path that
moves a row would have to remember to update it. The store already
commits each of those transitions, so the store now keeps the record.
TaskStore keeps an in-memory index of the ids in a claimable or
admitted state. It loads the index at open and updates it after the
commit of every write that moves a row into or out of that set: accept,
insert_if_absent, transition, cancel and wake_wait. is_queued reads
that index for durable rows, with a lock and no SQLite access, because
the probe runs on the gateway loop. _accepted_unstarted_ids and its
add and drop sites in the gate, the bridge and the eviction are gone.

Shrink the dispatch window to an id set. Nothing read the params
values, and the comment justifying them cited a reader that does not
exist.
@chenmingwei23

Copy link
Copy Markdown
Contributor Author

@buluoray PR Readiness is green on 94e0689 (same two commits as the head you approved, rebased onto current main; no content change). Ready for your re-approval. Thanks!

@bolichen97 bolichen97 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Checked the main P0 risk, a guard stuck forever: every TaskStore write that moves a row out of claimable/admitted (transition, cancel, wake_wait) calls _note_state after COMMIT, there is no DELETE path, and open() re-seeds the index. The window ids drop when a retained claim registers or is refused. I see no leak that would hold a serial lock and block spawns.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

spawn_run results never reach the session: no completion event, spawn_status "not found", spawn_list empty

3 participants