Skip to content

fix(taskq): stop fencing a user stop on the generation it read - #17161

Merged
iamwhatever merged 1 commit into
mainfrom
fix/taskq-stale-generation-cancel
Oct 6, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
fix/taskq-stale-generation-cancel

Conversation

@buluoray

@buluoray buluoray commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A user Stop on a sub-agent row that has not started still cancels it when a concurrent write moved only the row's generation, so the parent is never left blocked behind a row nobody will settle.

taskq_cancel_queued reads the row and then cancels it under the generation it read. If a claim (queued -> admitted), a requeue or a wake (into retry_wait) bumps the generation in between, the row is still unstarted but TaskStore.cancel refuses with stale_result. The stop is then dropped. A retry_wait row left behind this way is outside the boot reconciler's ACTIVE set and is still counted by count_pending. The parent's attached-children guard (/continue -> 409 slot_subagents_running) therefore stays wedged across restarts. Full mechanism with file:line evidence is in #17158.

Why it matters

The user pressed Stop and was shown a stop, but the session stays blocked with a 409 that names no task. Restarting does not help. Recovering needs a manual edit of the task database.

Not a goal

What changed (motivation → approach → change)

  • Symptom: a stop is dropped, and a retry_wait row keeps the parent attached forever.
  • Root cause: taskq_cancel_queued takes a None from the conditional cancel to mean "the drain started it" even when only the generation moved and the row is still unstarted. Nothing retries the cancel for a store-only row. _repost_unlanded_cancel covers window entries only.
  • Change: taskq_cancel_queued (src/kiro_crew/subagent_manager/admission/taskq_bridge.py) no longer passes generation=rec.generation to TaskStore.cancel; it keeps only_from=unstarted. The store re-tests only_from inside the cancel's own transaction, so a row that reached starting/running is still never cancelled under its executor, while a row whose generation alone moved (a claim, requeue or wake) is still unstarted and now cancels. docs/system-specs/modules/taskq.md (caller table + test list) is updated in the same commit.

Backwards compatibility

Compatible: the function's contract (params on cancel, None otherwise) and its refusal of a started row are unchanged. It only cancels in one more case, a row that is still unstarted at a newer generation, which is the stop the user asked for.

Removes nothing: no path, record, key or validator is removed or tightened. A stop that used to be dropped now lands.

Tests

  • test/test_taskq_admission_integration.py::test_a_stop_whose_generation_moved_under_an_unstarted_row_still_cancels[queued|retry_wait]: forces a real claim inside taskq_cancel_queued's read, so the generation moves before the cancel. It asserts the row ends cancelled, the params come back, and no stale_result refusal was recorded. Mutation check: restoring generation=rec.generation (the pre-fix code) turns both cases red.
  • The existing test_a_queued_row_the_drain_started_is_not_cancelled_under_the_spawn still passes, so a row that went live is still left to the reap.

Manual verification

N/A. Unit coverage is sufficient: the race is forced deterministically against the real store and real admission. python3 scripts/local-gate.py passed (backend related: 776 passed; frontend related: 10147 passed).

Related Issues

Closes #17158

Pattern harvest

Rule candidate: review-prompt
Pattern: "a read-then-conditional-write caller fences on more than its decision needs; when the in-transaction state predicate (only_from) already guarantees safety, an added generation fence only turns harmless concurrent bumps into dropped operations"

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

@buluoray
buluoray requested a review from a team as a code owner October 5, 2026 22:58
@buluoray
buluoray requested a review from chenmingwei23 October 5, 2026 22:58
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 5, 2026
@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

Nothing to check.

What this change ships

Inventory (3 items) — 3 justified

Intent: when a user presses Stop on a sub-agent that has not started yet, the stop should actually cancel it, so the parent session is not left stuck behind it with a 409. This is a FIX (#17158, plus a parametrized test the description says fails on base).

  1. Stopping an unstarted sub-agent row now works even if a claim, requeue or wake bumped its generation in between: the generation= argument is deleted from the cancel call — justified
  2. The taskq spec's caller table and test list now describe the cancel with no generation fence — justified
  3. A regression test forces a real claim between the read and the cancel, for both queued and retry_wait rows — justified

[FIRST-PRINCIPLES-REVIEWED] fe5c1cd

Description read

[DESCRIPTION-READ] f83f1f9418e08d40f451eeb1763713a78665ad156deb3d82f13cd10191076178

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check whether taskq_cancel_queued_async's docstring still describes the old fence: it says "one only_from under the generation the read returned", and this PR removes that fence.

Suggestions

  • Update the delegation docstring at taskq_bridge.py:2311-2315 in this commit. It says the twin's safety argument is "the same code rather than the same intent restated". After this change it restates an intent the code no longer has.

[DESIGN-REVIEWED] fe5c1cd

Description read

[DESCRIPTION-READ] c48a4721df83b843db7913194d1dd0210c1b9829cfb61456214b305902ee7299

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] fe5c1cd

Verdict parsed from the review's SHA-scoped output markers for commit fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015.

False positive or not applicable? A repository writer can comment:
/ai-review override fable fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 5, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] fe5c1cd

False positive or not applicable? A repository writer can comment:
/ai-review override gpt fe5c1cd31b6c1d4d7da17ad20bee56ddb84a8015: <one-sentence reason>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 5, 2026
@buluoray
buluoray force-pushed the fix/taskq-stale-generation-cancel branch from cfef4ea to b52a2cb Compare October 6, 2026 00:18
@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author
  • Log a stop still refused after all three tries

fixed in b52a2cb: the retry loop's for ... else in taskq_cancel_queued now logs taskq: stop of <id> dropped ... at WARNING when every pass found the row unstarted and every cancel was refused.
Pinned by test_a_stop_refused_on_every_retry_is_logged_not_silent (cancel always refuses -> exactly 3 calls, a warning naming the row, row still queued); it goes red when the warning is changed. Spec caller table + test list updated in the same commit.
self-added: no

@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author
  • After three stale refusals the loop returns None without logging

fixed in b52a2cb: exhaustion now logs a WARNING naming the row and the attempt count (for ... else in taskq_cancel_queued), so a dropped stop is visible instead of silent. Return value is unchanged (None), so callers still fall through to the live reap.
Pinned by test_a_stop_refused_on_every_retry_is_logged_not_silent.
self-added: no

@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

CI note for cfef4ea: both reds were inherited, not caused by this diff. Coverage Gate flagged src/kiro_crew/apps/hooks_integration.py (not in this diff) as graduated from the baseline; main removed that row in #17114, and this rebase picks it up. Backend Tests (Windows) (2) failed in test_dashboard_chat.py::TestFolderCRUD::test_resume_session_unhides_folder and test_slot_ownership_checkpoint.py (resume_conflict 409); the same tests failed on base main 3a18542 (run 37383777848), main went green after #17131/#17122, and this push is rebased onto 782dedd.

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Oct 6, 2026
@buluoray
buluoray force-pushed the fix/taskq-stale-generation-cancel branch from b52a2cb to 3534b29 Compare October 6, 2026 00:58
@buluoray buluoray changed the title fix(taskq): retry a user stop whose generation moved under it fix(taskq): stop fencing a user stop on the generation it read Oct 6, 2026
@buluoray

buluoray commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=7aea2461a2e0 Item 1: retry loop works around a generation fence only_from already covers

fixed in 3534b29: verified in source -- TaskStore.cancel re-tests only_from inside its own BEGIN IMMEDIATE transaction (store.py cancel), so a row that reached starting/running is refused without the generation fence, and params (the only other thing read from rec) is never rewritten across generations except accepted_by.
Took the subtraction: dropped generation=rec.generation, _STALE_CANCEL_ATTEMPTS, the for/else loop and the exhaustion warning; one read plus one only_from cancel remains.
Regression test asserts the cancel lands with no stale_result event; restoring generation=rec.generation turns both parametrized cases red.
self-added: yes
mechanism: generation-fence retry loop (removed)

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Oct 6, 2026
taskq_cancel_queued read the row, then cancelled under that read's
generation. A claim, requeue or wake that bumps the generation while
the row stays unstarted made the store refuse the cancel as
stale_result, and the stop was dropped. A retry_wait row left that way
is outside the boot reconciler's ACTIVE set and stays in the parent's
queued count, so the attached-children guard never clears.

The generation fence added nothing the in-transaction only_from test
does not already guarantee: a row that reached starting/running is
still refused and left to the live reap. Drop the generation argument
and keep only_from.

Closes #17158
@buluoray
buluoray force-pushed the fix/taskq-stale-generation-cancel branch from 3534b29 to fe5c1cd Compare October 6, 2026 01:24
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 6, 2026
@buluoray

buluoray commented Oct 6, 2026

Copy link
Copy Markdown
Contributor Author

CI note for fe5c1cd: Backend Tests (Windows) (2) failed in test_dashboard_chat.py::TestFolderCRUD::test_resume_session_unhides_folder and test_slot_ownership_checkpoint.py::TestTranscriptOwnershipWithoutALiveSlot (resume 409 resume_conflict). Not this diff: those tests exercise dashboard/chat_api/resume.py, which this PR does not touch; the same tests failed on base main 3a18542 (run 37383777848) and on unrelated PR #17151 (run 37397669414, same shard). Rerun with gh run rerun --failed.

@github-actions github-actions Bot removed the readiness: checking Automated validation is still running label Oct 6, 2026
@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 6, 2026
@iamwhatever
iamwhatever enabled auto-merge (squash) October 6, 2026 03:07

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, SAST annotations clean, security checklist all-NO, AI reviewers green. Category: taskq_cancel_queued drops the generation fence on a user stop so a row whose generation moved while still unstarted (admitted / retry_wait) is cancelled instead of refused as stale_result; only_from still limits the cancel to unstarted rows; one-line root-cause fix plus integration test. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/taskq.md.

@iamwhatever
iamwhatever merged commit e281eca into main Oct 6, 2026
193 of 195 checks passed
@iamwhatever
iamwhatever deleted the fix/taskq-stale-generation-cancel branch October 6, 2026 03:08
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

taskq: a user Stop lost to a concurrent generation bump leaves a retry_wait row attached forever

2 participants