Repository navigation
fix(taskq): cancel a parked row its current attempt was stopped in - #33155
Conversation
A sub-agent row left in retry_wait with no lease, whose run's tombstone already records a user stop, was never settled: the boot reconcile's queued pass reads retry_wait rows but acts only on the refusal and queued-stop markers, so count_pending kept the parent slot answering 409 slot_subagents_running on every turn until the store was edited by hand. The queued pass now cancels such a row when the stop belongs to the row's current attempt. An attempt is named by the generation of the claim that began it: the registered run keeps it as _taskq_attempt, its tombstone records it as taskq_attempt (carried through the delivery rewrite), and the pass asks the subagent probe about the generation of the row's latest claimed event. A stop from an earlier attempt, or one that names no attempt, leaves the row as it was. The cancel is fenced on retry_wait and the generation read. Builds on #22252 by @cruisercohen. Closes #22244
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change adds one boot-time cancel, fenced on Notes
[DESIGN-REVIEWED] c543e38 Description read[DESCRIPTION-READ] 3d38632e0755c0d2c96b7a37612b69937eb58cfdc5568c85d08261a5bbe10681 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. The fix works at boot, one level after the cause: the stop's settle write can still be lost. The description says so under "Not a goal". Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: a chat slot wedged with 409 by a stopped sub-agent's leftover
[FIRST-PRINCIPLES-REVIEWED] c543e38 Description read[DESCRIPTION-READ] 7fadc3ee0b5bbcbcc612cdf79adfdb8c1568a7c843932b8ab9ba4dd1fc2f6be8 That sha256 names the PR description this verdict read. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
…33155) A sub-agent row left in retry_wait with no lease, whose run's tombstone already records a user stop, was never settled: the boot reconcile's queued pass reads retry_wait rows but acts only on the refusal and queued-stop markers, so count_pending kept the parent slot answering 409 slot_subagents_running on every turn until the store was edited by hand. The queued pass now cancels such a row when the stop belongs to the row's current attempt. An attempt is named by the generation of the claim that began it: the registered run keeps it as _taskq_attempt, its tombstone records it as taskq_attempt (carried through the delivery rewrite), and the pass asks the subagent probe about the generation of the row's latest claimed event. A stop from an earlier attempt, or one that names no attempt, leaves the row as it was. The cancel is fenced on retry_wait and the generation read. Builds on #22252 by @cruisercohen. Closes #22244 Original author: Bolin Chen (cherry picked from commit e7836f7)
Problem / Motivation
Goal: A gateway restart clears a chat slot that a stranded
retry_waitsub-agent row left answering 409 on every turn.A sub-agent run was stopped, but its own cancel never reached the task store, and its row was left in
retry_waitwith no lease while the run's tombstone already recorded the stop.count_pendingkeeps counting that row, so the parent slot is treated as having a child attached and every turn on it gets409 slot_subagents_running. Restarting the gateway does not help: the row was not re-claimed across two restarts in the report, and the only repair was a hand edit of the task store.Builds on #22252 by @cruisercohen.
Why it matters
The parent chat slot is unusable until someone edits the task store by hand, and a restart, the obvious recovery, leaves it wedged. #17161 removed one way to reach this shape; the stop's own settle is still a best-effort write (a busy store drops it), so a row can still be stranded this way.
Not a goal
retry_waitrow on any ending but a stop.failedstays the dispatcher's, because a wait wake or a park also parks a row for a retry;retry_wait -> donestays closed on purpose.queuedrows on a plain ending; the queued pass still acts only on its two markers there.kirocrew doctoralready names such a row once it is long past due.deliveredthe reaper prunes it aftersubagent_result_ttl_secs; a row whose tombstone is gone is left as on main.What changed (motivation → approach → change)
After a gateway restart, a chat slot that a stopped sub-agent left answering 409 on every turn takes turns again: the stopped run's leftover task row is cancelled at boot, so it no longer counts as an attached child.
Symptom → cause. The boot reconcile's queued pass already reads every
retry_waitrow, but it settles a row only on a refusal or queued-stop marker. A plain stop tombstone on a parked row is ignored, so the row stays pending forever.Approach. Settle exactly one more case: a
retry_waitrow whose CURRENT attempt recorded a stop is cancelled, fenced onretry_waitand the generation the pass read. "Current attempt" is matched by identity, never by time, because the run folder is the row id's: a row claimed again after a lost stop runs over the earlier attempt's tombstone, and settling on that would end live work.Change.
SubagentInfo._taskq_attempt(set at registration inadmission/gate.py)_taskq_generation, never this_write_tombstone(subagent_parts/terminal_surface.py)taskq_attempton the run's tombstonesubagent_persistence_tombstones.pytaskq_attempt, as it keepsoutcometaskq/reconcile.pycancelledfor aretry_waitrow, read the row's latestclaimedevent and ask the probe again withattempt=<its generation>; cancel only on acancelledanswer.ArtifactProbebecomes aProtocolwith that optionalattempt(taskq/__init__.pyuses it)taskq_artifact_probe(admission/taskq_bridge.py)attempt, answers only from a tombstone whosetaskq_attemptequals itA tombstone from an earlier attempt or naming none, a row with no claim, and a store that refuses the read or the write leave the row exactly as on main (the last named in
report.errors).docs/system-specs/modules/taskq.mddescribes the case.Alternative considered: a separate
retry_waitpass (as #22252 had, written before the queued pass readretry_waitrows). Rejected: the queued pass already pages those rows, so the check rides in it.Backwards compatibility
Compatible: the only new write is a
cancelledon an unleasedretry_waitrow whose current attempt's own tombstone records a stop; every other row, state and verdict is handled as before, and a tombstone written before this change settles nothing.Removes nothing: no key, field, function or validator is removed or tightened. Tombstones gain an optional
taskq_attemptkey no existing reader requires,SubagentInfogains_taskq_attempt, andtaskq_artifact_probegains an optionalattemptthe existing calls do not pass.Tests
test_taskq_reconcile.py::test_an_unleased_retry_wait_row_is_cancelled_only_on_its_current_attempts_stop, through the real subagent probe across a close and reopen: the reported shape (stop naming the latest claim's attempt, due six hours past) is cancelled, the parent'scount_pendingdrops to 0 and the row cannot be claimed; a row claimed and started again after that stop, and a tombstone naming no attempt, are left inretry_waitwith their generation andcount_pending1.test_taskq_admission_integration.py::test_a_registered_run_carries_its_claims_attempt_into_its_tombstone: a real spawn through the manager;_taskq_attemptequals theclaimedevent's generation, a wake-shaped bump of_taskq_generationleaves it, the run's tombstone records it,mark_delivered's rewrite keeps it, and the boot reconcile with the manager's probe cancels the parked row.test_taskq_bridge_bindings.py:taskq_artifact_probe's signature pin updated forattempt.Proof: PROVEN test/test_taskq_bridge_bindings.py::test_each_member_keeps_its_kind_signature_and_owner[taskq_artifact_probe], test/test_taskq_reconcile.py::test_an_unleased_retry_wait_row_is_cancelled_only_on_its_current_attempts_stop[False-True-cancelled] (head c543e38)
Manual verification
N/A: unit coverage sufficient. The tests drive the real
TaskStorefile across a close and reopen, as a restarted gateway sees it, through the real subagent artifact probe.Related Issues
Closes #22244
Sibling sweep
Swept: git grep -nE ".queued_rows(|.active_rows(" -- src/kiro_crew
Sibling: src/kiro_crew/taskq/reconcile.py -- changed here
Sibling: src/kiro_crew/taskq/adapters/runner.py -- not affected -- runner kinds write no sub-agent tombstone and adopt only active rows
Sibling: src/kiro_crew/dashboard/handlers/tasks.py -- not affected -- a read for the task list, it settles nothing
Sibling: src/kiro_crew/dashboard/session_health.py -- not affected -- a health read, it settles nothing
Pattern harvest
Not generalizable: the miss is one ending one boot pass did not act on for one state, and the fix needs the attempt identity only that sub-agent probe can read; there is no shape a rule could match.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.