Skip to content

fix(taskq): cancel a parked row its current attempt was stopped in - #33155

Merged
bolichen97 merged 1 commit into
mainfrom
fix/22244-retry-wait-boot-settle-v2
Nov 8, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/22244-retry-wait-boot-settle-v2

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A gateway restart clears a chat slot that a stranded retry_wait sub-agent row left answering 409 on every turn.

A sub-agent run was stopped, but its own cancel never reached the task store, and its row was left in retry_wait with no lease while the run's tombstone already recorded the stop. count_pending keeps counting that row, so the parent slot is treated as having a child attached and every turn on it gets 409 slot_subagents_running. Restarting the gateway does not help: the row was not re-claimed across two restarts in the report, and the only repair was a hand edit of the task store.

Builds on #22252 by @cruisercohen.

Why it matters

The parent chat slot is unusable until someone edits the task store by hand, and a restart, the obvious recovery, leaves it wedged. #17161 removed one way to reach this shape; the stop's own settle is still a best-effort write (a busy store drops it), so a row can still be stranded this way.

Not a goal

  • Settling a retry_wait row on any ending but a stop. failed stays the dispatcher's, because a wait wake or a park also parks a row for a retry; retry_wait -> done stays closed on purpose.
  • Settling queued rows on a plain ending; the queued pass still acts only on its two markers there.
  • Settling rows stranded before this change: their tombstones name no attempt, so they are left as they are. kirocrew doctor already names such a row once it is long past due.
  • Making the stop's own settle write durable.
  • Keeping a stopped run's tombstone past its retention window. A stop tombstone is kept for 7 days, but once the parent's delivery rewrites it as delivered the reaper prunes it after subagent_result_ttl_secs; a row whose tombstone is gone is left as on main.

What changed (motivation → approach → change)

After a gateway restart, a chat slot that a stopped sub-agent left answering 409 on every turn takes turns again: the stopped run's leftover task row is cancelled at boot, so it no longer counts as an attached child.

Symptom → cause. The boot reconcile's queued pass already reads every retry_wait row, but it settles a row only on a refusal or queued-stop marker. A plain stop tombstone on a parked row is ignored, so the row stays pending forever.

Approach. Settle exactly one more case: a retry_wait row whose CURRENT attempt recorded a stop is cancelled, fenced on retry_wait and the generation the pass read. "Current attempt" is matched by identity, never by time, because the run folder is the row id's: a row claimed again after a lost stop runs over the earlier attempt's tombstone, and settling on that would end live work.

Change.

Piece What it does
SubagentInfo._taskq_attempt (set at registration in admission/gate.py) the generation of the claim that began this attempt; a wait wake moves _taskq_generation, never this
_write_tombstone (subagent_parts/terminal_surface.py) records it as taskq_attempt on the run's tombstone
subagent_persistence_tombstones.py a later rewrite with no ending of its own (the delivery acknowledgement) keeps taskq_attempt, as it keeps outcome
taskq/reconcile.py when the probe says cancelled for a retry_wait row, read the row's latest claimed event and ask the probe again with attempt=<its generation>; cancel only on a cancelled answer. ArtifactProbe becomes a Protocol with that optional attempt (taskq/__init__.py uses it)
taskq_artifact_probe (admission/taskq_bridge.py) with attempt, answers only from a tombstone whose taskq_attempt equals it

A tombstone from an earlier attempt or naming none, a row with no claim, and a store that refuses the read or the write leave the row exactly as on main (the last named in report.errors). docs/system-specs/modules/taskq.md describes the case.

Alternative considered: a separate retry_wait pass (as #22252 had, written before the queued pass read retry_wait rows). Rejected: the queued pass already pages those rows, so the check rides in it.

Backwards compatibility

Compatible: the only new write is a cancelled on an unleased retry_wait row whose current attempt's own tombstone records a stop; every other row, state and verdict is handled as before, and a tombstone written before this change settles nothing.

Removes nothing: no key, field, function or validator is removed or tightened. Tombstones gain an optional taskq_attempt key no existing reader requires, SubagentInfo gains _taskq_attempt, and taskq_artifact_probe gains an optional attempt the existing calls do not pass.

Tests

  • test_taskq_reconcile.py::test_an_unleased_retry_wait_row_is_cancelled_only_on_its_current_attempts_stop, through the real subagent probe across a close and reopen: the reported shape (stop naming the latest claim's attempt, due six hours past) is cancelled, the parent's count_pending drops to 0 and the row cannot be claimed; a row claimed and started again after that stop, and a tombstone naming no attempt, are left in retry_wait with their generation and count_pending 1.
  • test_taskq_admission_integration.py::test_a_registered_run_carries_its_claims_attempt_into_its_tombstone: a real spawn through the manager; _taskq_attempt equals the claimed event's generation, a wake-shaped bump of _taskq_generation leaves it, the run's tombstone records it, mark_delivered's rewrite keeps it, and the boot reconcile with the manager's probe cancels the parked row.
  • test_taskq_bridge_bindings.py: taskq_artifact_probe's signature pin updated for attempt.

Proof: PROVEN test/test_taskq_bridge_bindings.py::test_each_member_keeps_its_kind_signature_and_owner[taskq_artifact_probe], test/test_taskq_reconcile.py::test_an_unleased_retry_wait_row_is_cancelled_only_on_its_current_attempts_stop[False-True-cancelled] (head c543e38)

Manual verification

N/A: unit coverage sufficient. The tests drive the real TaskStore file across a close and reopen, as a restarted gateway sees it, through the real subagent artifact probe.

Related Issues

Closes #22244

Sibling sweep

Swept: git grep -nE ".queued_rows(|.active_rows(" -- src/kiro_crew
Sibling: src/kiro_crew/taskq/reconcile.py -- changed here
Sibling: src/kiro_crew/taskq/adapters/runner.py -- not affected -- runner kinds write no sub-agent tombstone and adopt only active rows
Sibling: src/kiro_crew/dashboard/handlers/tasks.py -- not affected -- a read for the task list, it settles nothing
Sibling: src/kiro_crew/dashboard/session_health.py -- not affected -- a health read, it settles nothing

Pattern harvest

Not generalizable: the miss is one ending one boot pass did not act on for one state, and the fix needs the attempt identity only that sub-agent probe can read; there is no shape a rule could match.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

A sub-agent row left in retry_wait with no lease, whose run's tombstone
already records a user stop, was never settled: the boot reconcile's
queued pass reads retry_wait rows but acts only on the refusal and
queued-stop markers, so count_pending kept the parent slot answering
409 slot_subagents_running on every turn until the store was edited by
hand.

The queued pass now cancels such a row when the stop belongs to the
row's current attempt. An attempt is named by the generation of the
claim that began it: the registered run keeps it as _taskq_attempt,
its tombstone records it as taskq_attempt (carried through the
delivery rewrite), and the pass asks the subagent probe about the
generation of the row's latest claimed event. A stop from an earlier
attempt, or one that names no attempt, leaves the row as it was. The
cancel is fenced on retry_wait and the generation read.

Builds on #22252 by @cruisercohen.

Closes #22244
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 8, 2026 12:23
@bolichen97
bolichen97 requested a review from Zedmor November 8, 2026 12:23
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 8, 2026
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision c543e381d1746012cef10ea8afe7711f0eb3f9ed touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of c543e381d1746012cef10ea8afe7711f0eb3f9ed — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change adds one boot-time cancel, fenced on retry_wait, the generation the pass read and a matching attempt identity, and a tombstone that names no attempt changes nothing.

Notes

  • The only production probe is taskq_artifact_probe. A future one-argument probe would raise when called with attempt. _probe_verdict catches that and records an error rather than settling the row, so it fails safe.

[DESIGN-REVIEWED] c543e38

Description read

[DESCRIPTION-READ] 3d38632e0755c0d2c96b7a37612b69937eb58cfdc5568c85d08261a5bbe10681

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of c543e381d1746012cef10ea8afe7711f0eb3f9ed — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. The fix works at boot, one level after the cause: the stop's settle write can still be lost. The description says so under "Not a goal".

Notes

  • Item 1 — symptom-level: the cause is that the stop's settle is "a best-effort write (a busy store drops it)", and "Making the stop's own settle write durable" is excluded. Until a restart, the slot still answers 409 on every turn. Accepted and deferred, because a durable settle is larger than this change.

What this change ships

Inventory (5 items) — 4 justified

Intent: a chat slot wedged with 409 by a stopped sub-agent's leftover retry_wait row recovers after a gateway restart. FIX (#22244).

  1. On boot, an unleased retry_wait row whose current attempt recorded a stop is now cancelled, which frees the parent slot — symptom-level (the lost stop settle is left as it is)
  2. A run's tombstone now records taskq_attempt, and a delivery rewrite keeps it — justified
  3. A registered sub-agent run now carries _taskq_attempt, the generation of the claim that began it — justified
  4. taskq_artifact_probe takes an optional attempt and answers only from a tombstone that names that attempt — justified
  5. ArtifactProbe changed from a Callable alias to a Protocol; open_default_store's type now uses it — justified

[FIRST-PRINCIPLES-REVIEWED] c543e38

Description read

[DESCRIPTION-READ] 7fadc3ee0b5bbcbcc612cdf79adfdb8c1568a7c843932b8ab9ba4dd1fc2f6be8

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed c543e381d1746012cef10ea8afe7711f0eb3f9ed — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] c543e38

Verdict parsed from the review's SHA-scoped output markers for commit c543e381d1746012cef10ea8afe7711f0eb3f9ed.

False positive or not applicable? A repository writer can comment:
/ai-review override fable c543e381d1746012cef10ea8afe7711f0eb3f9ed false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of c543e381d1746012cef10ea8afe7711f0eb3f9ed and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] c543e38

False positive or not applicable? A repository writer can comment:
/ai-review override gpt c543e381d1746012cef10ea8afe7711f0eb3f9ed false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 enabled auto-merge (squash) November 8, 2026 12:44
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 8, 2026

@buluoray buluoray left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed read-only at c543e38: change does what the description says, tests cover it, AI lanes pass. Approving.

@bolichen97
bolichen97 merged commit e7836f7 into main Nov 8, 2026
98 checks passed
@bolichen97
bolichen97 deleted the fix/22244-retry-wait-boot-settle-v2 branch November 8, 2026 13:45
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 8, 2026
buluoray added a commit that referenced this pull request Nov 8, 2026
…33155)

A sub-agent row left in retry_wait with no lease, whose run's tombstone
already records a user stop, was never settled: the boot reconcile's
queued pass reads retry_wait rows but acts only on the refusal and
queued-stop markers, so count_pending kept the parent slot answering
409 slot_subagents_running on every turn until the store was edited by
hand.

The queued pass now cancels such a row when the stop belongs to the
row's current attempt. An attempt is named by the generation of the
claim that began it: the registered run keeps it as _taskq_attempt,
its tombstone records it as taskq_attempt (carried through the
delivery rewrite), and the pass asks the subagent probe about the
generation of the row's latest claimed event. A stop from an earlier
attempt, or one that names no attempt, leaves the row as it was. The
cancel is fenced on retry_wait and the generation read.

Builds on #22252 by @cruisercohen.

Closes #22244

Original author: Bolin Chen
(cherry picked from commit e7836f7)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

taskq: boot reconcile never settles a retry_wait row left with no lease, so its slot keeps answering 409

2 participants