Skip to content

fix(subagent): requeue a row whose claimer is cancelled under its claim - #21570

Merged
bolichen97 merged 1 commit into
mainfrom
fix/claim-cancel-release-20391
Oct 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/claim-cancel-release-20391

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A sub-agent spawn whose claimer is cancelled mid-claim goes back to the queue at once instead of waiting for a gateway restart.

claim_and_start awaits taskq_claim on the store's writer thread. When the claimer's task is cancelled (a shutdown, a caller whose client went away) after that job already took the row, the row is admitted under this incarnation's lease, the generation never reached the claimer, and nothing carries it on. It sits admitted until the next boot's reconcile requeues it.

Why it matters

Accepted work silently stalls: the user's spawn shows as queued and never starts until someone restarts the gateway.

Not a goal

What changed (motivation → approach → change)

Symptom: a cancelled claimer leaves its row admitted with no owner. Cause: the claim await was a plain await store.run(...), so the cancel dropped the claim's answer, and with it the generation needed to fence a release.

Approach: the same shape #20319 uses one await later. claim_and_start now calls _claim_or_release_on_cancel in subagent_manager/admission/pump.py. It shields the claim job and, on a cancel, awaits it to its answer. If the claim took the row, it posts one generation-fenced admitted -> queued write (_release_untaken_claim in subagent_manager/admission/taskq_bridge.py) behind it on the same writer, asks the pump for another pass, then re-raises the cancel. No start write is posted. A claim that did not take the row, or a row a stop recorded meanwhile, writes nothing. A claim that fails with the store unavailable took nothing, so it is logged and the cancel itself propagates, never the outage. A release the store cannot take is logged and the row is left to the boot reconcile, as before.

docs/system-specs/modules/taskq.md replaces the sentence that said this case was left to the reconcile.

Backwards compatibility

Compatible: only the cancelled-claimer path changes; a claim that is not cancelled returns the same tuple as before.

Removes nothing: the boot reconcile still requeues any row this release could not write.

Tests

  • test/test_taskq_claim_start.py::test_a_claimer_cancelled_under_its_claim_await_requeues_the_row (new): cancel the claimer after the claim job took the row; the row ends queued at the claim's generation, unleased, no start write posted, reservation returned; the release requests exactly one pump pass (recorded, not run, so no pass reclaims the row while its released state is read); the test's own pump pass then starts it under a new generation. Fails on main (row stays admitted), passes here.
  • test/test_taskq_claim_start.py::test_a_claimer_cancelled_under_a_failing_claim_still_propagates_its_cancel (new): the claim fails with TaskStoreUnavailable after the cancel; the claimer raises CancelledError, the row is unchanged and nothing registers. Fails without the except TaskStoreUnavailable branch (the outage replaced the cancel), passes here.
  • Determinism: both new tests 30/30 repeats under host load, and the whole file passes in collection order and in reversed order (31 passed each).
  • test/test_taskq_bridge_bindings.py: inventory row for the new static method.
  • test/test_subagent_scale.py::TestQueuedDepthReachesZero::test_a_popped_but_unclaimed_row_still_counts_as_pending_work: the shielded claim adds loop hops, so the test now waits for the registration itself (async_wait_until) instead of a fixed count of store barriers.
  • Local gate list (prepare-pr profile): related backend tests, black, isort, flake8, mypy and every repo checker pass. The only reds are main's own (MarkdownRenderer.domGolden in the frontend related set) and test:electron, which fails the same way on a PR with no desktop change on this host.

Manual verification

N/A — unit coverage sufficient: the path is a cancel between two writer jobs, which the new test drives with a real TaskStore.

Related Issues

Closes #20391

Pattern harvest

Rule candidate: review-prompt
Pattern: an await store.run(...) whose job mutates durable state must be shielded and its answer settled when the awaiting task is cancelled.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners October 17, 2026 09:07
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ea889dac73014ef3ed9b39ad9c1d7a745a7be505 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 17, 2026
@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ea889dac73014ef3ed9b39ad9c1d7a745a7be505 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ea889da

Description read

[DESCRIPTION-READ] 72f90d17ad20332e530373483f98daca3455dad0cf6513b2e979e52f71e10895

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ea889dac73014ef3ed9b39ad9c1d7a745a7be505 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Check before merge: a claim that commits and then reports TaskStoreUnavailable takes the "took nothing" branch, so the row is left to the boot reconcile.

What this change ships

Inventory (5 items) — 5 justified

Intent: when a sub-agent spawn's claimer is cancelled mid-claim, the spawn goes straight back to the queue instead of waiting for a gateway restart. This is a FIX (Closes #20391; the new test fails on base).

  1. A cancelled claimer now waits for its claim to finish instead of dropping its answer (_claim_or_release_on_cancel, the only taskq_claim call site) — justified
  2. A row the claim already took goes back to queued with one generation-fenced write (_release_untaken_claim; _release_taken_start covers a different state pair, so this is not a second spelling of it) — justified
  3. After a successful release outside shutdown, the pump gets one more pass, so the row starts right away — justified
  4. When a claim fails because the store is down, the claimer still raises its own cancel, not the outage — justified
  5. The taskq spec sentence that left this case to the boot reconcile is replaced, and the scale test now waits on registration instead of counting barriers — justified

[FIRST-PRINCIPLES-REVIEWED] ea889da

Description read

[DESCRIPTION-READ] 9fd437ede8fa034bf39d5950e664f600fdd67854c2ecd0f7f572e43191941cea

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ea889dac73014ef3ed9b39ad9c1d7a745a7be505 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ea889da

Verdict parsed from the review's SHA-scoped output markers for commit ea889dac73014ef3ed9b39ad9c1d7a745a7be505.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ea889dac73014ef3ed9b39ad9c1d7a745a7be505: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ea889dac73014ef3ed9b39ad9c1d7a745a7be505 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ea889da

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ea889dac73014ef3ed9b39ad9c1d7a745a7be505: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Frontend Tests (2) is red on src/test/MarkdownRenderer.domGolden.test.tsx > renders the prose document (snapshot mismatch). This PR changes no frontend file; it is the main-red tracked in #21535 (stale katex snapshot after #17239). Not rerun: it will stay red until main's fix lands, then this branch gets one rebase.

@bolichen97
bolichen97 enabled auto-merge (squash) October 17, 2026 09:35
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition: rebutted (the PASS punchline's reuse check, answered)

_release_untaken_registration (pump.py) posts _release_taken_start (starting -> queued) and returns True for "committed OR the row moved on", because its caller only needs to know whether to retain the claim.
The claim release posts _release_untaken_claim (admitted -> queued) and needs the commit result itself, because only a committed release requests the extra pump pass.
Sharing it needs a writer-function parameter and a second return mode: more code than the dozen lines it would save.

@bolichen97
bolichen97 force-pushed the fix/claim-cancel-release-20391 branch from 7177e4f to 6625b3c Compare October 17, 2026 10:40
A claimer cancelled while its claim ran on the store's writer thread left the row admitted under this incarnation's lease with nothing carrying it on, so accepted work waited for the next boot's reconcile. The claim job is now shielded and awaited to its answer; a claim that took the row is released by one generation-fenced admitted -> queued write behind it, and the pump is asked for another pass.

Closes #20391
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 17, 2026
@bolichen97
bolichen97 force-pushed the fix/claim-cancel-release-20391 branch from 6625b3c to ea889da Compare October 17, 2026 10:42
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] Disposition: rebutted (the PASS punchline's check, answered: not a defect)

TaskStore.claim (src/kiro_crew/taskq/store.py:889-913) raises TaskStoreUnavailable only from its except sqlite3.Error arm, which runs self._rollback(conn) first; the BEGIN IMMEDIATE ... COMMIT is one transaction.
A COMMIT that raises did not commit, so an outage answer always means the row was not taken; there is no commit-then-outage answer for this branch to mislabel.
Even if one existed, the row would be admitted under this incarnation's lease, which the boot reconcile requeues, the same as main's behavior for every cancelled claim before this PR.

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (6 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bug fix; a claimer cancelled under its claim await now gives the admitted row back to queued under a generation fence instead of leaving it for the boot reconcile, with tests. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/taskq.md.

@bolichen97
bolichen97 merged commit 2d926d9 into main Oct 17, 2026
104 checks passed
@bolichen97
bolichen97 deleted the fix/claim-cancel-release-20391 branch October 17, 2026 12:38
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Subagent claim: release the admitted row when the claimer is cancelled under its claim await

2 participants