Skip to content

fix(taskq): start a re-claimed attempt without the old attempt's progress - #19819

Merged
javenciu merged 1 commit into
mainfrom
fix/taskq-claim-clears-stale-progress
Oct 13, 2026
Merged

javenciu merged 1 commit into
mainfrom
fix/taskq-claim-clears-stale-progress

Conversation

@javenciu

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A task row that is claimed again as a new attempt starts with no progress marker, instead of carrying the marker the lost attempt left behind.

  • TaskStore.claim (src/kiro_crew/taskq/store.py:1056-1064 on 54b0c64) re-takes a claimable row (queued, retry_wait or recovering) as generation N+1. One UPDATE sets the state, the lease, generation=generation+1, attempts=attempts+1 and next_run_at=NULL. It does not name progress_json.
  • TaskStore.record_progress (src/kiro_crew/taskq/store.py:1702 on 54b0c64) overwrites that column with the latest marker and refuses any generation other than the row's current one.
  • After a re-claim the row therefore has the new generation, a new lease and attempts one higher, together with the last marker the lost attempt recorded. TaskStore.get and the ClaimResult.record that claim returns both show it, until the new attempt records its own.

Why it matters

generation says which attempt a row describes, and a marker from generation N on a row at generation N+1 says something false about it. Nothing renders the stale marker in this tree yet: row_payload (src/kiro_crew/dashboard/handlers/tasks.py:219) does not emit progress, and the only reader of the column is TaskRecord.from_row (src/kiro_crew/taskq/model.py:433). The commands that show this are under Tests. A surface or a caller that reads record.progress would show the lost attempt's last step as the new attempt's.

Not a goal

  • Showing progress in the task API or the dashboard.
  • Any other writer of the column: transition(progress=...) and record_progress are unchanged, and so is the generation fence on record_progress.
  • wake_wait and cancel, the other statements that bump generation (src/kiro_crew/taskq/store.py:1632, :1639, :1399). A wake to running continues the same attempt with its runtime resident and does not bump attempts, and the stop-recovery marker is written after it; a wake to retry_wait leaves the row claimable, and the claim that follows clears the marker. cancel ends the row, so its last marker is the final record.
  • A schema change or a migration. The column and its encoding are the same.

What changed (motivation → approach → change)

  • Symptom: record progress under attempt one, lose its owner, claim the row again. claim returns generation 2 with progress == {'step': 3}, the marker of generation 1.
  • Root cause: the claim UPDATE resets the lease and next_run_at for the new attempt and leaves progress_json untouched.
  • Change: progress_json=NULL joins the SET list of that UPDATE (src/kiro_crew/taskq/store.py:1061), and the claim docstring says so (:1047-1049). NULL is already how a row with no progress is stored: TaskRecord.to_row writes it for a falsy progress (src/kiro_crew/taskq/model.py:390), and TaskRecord.from_row maps the column back to progress (src/kiro_crew/taskq/model.py:433), so a row that never recorded progress holds the same NULL. docs/system-specs/modules/taskq.md names the column in the claim statement and in the paragraph below it.
  • Rejected: clearing the marker when the row moves to recovering or retry_wait. While the row waits, the marker is the record of where the lost attempt got to; the claim is the one statement that starts a new attempt. Also rejected: stamping the marker with its generation so readers can compare. It changes what record_progress stores for a problem the claim can fix in one column.
  • result_ref is the other column a lost attempt could leave, and the claim cannot meet one: only the terminal finish path writes it, and the terminal states DONE, FAILED and CANCELLED have no outgoing transition while UNKNOWN_SIDE_EFFECT goes only to DONE or FAILED (src/kiro_crew/taskq/model.py:151-155). wait_json is cleared on every exit from a wait (src/kiro_crew/taskq/store.py:1210, :1632, :1639).

Backwards compatibility

Compatible: no stored value changes shape. The column keeps its type and encoding. A row written before this change keeps whatever marker it holds until the next claim clears it.

Reader: src/kiro_crew/taskq/model.py:433 TaskRecord.from_row -- every store read (subagent, runner and TaskRunner rows) -- maps NULL to progress=None, the same as a row that never recorded progress (test_a_new_attempt_starts_without_the_old_attempts_progress).
Reader: test/test_subagent_stop_reason_consistency.py:316 -- subagent stop recovery -- reads phase == "readmitted", which record_progress writes after wake_wait, not after claim; the file passes unchanged.

The column holds only the latest marker (record_progress overwrites it), so no history is lost: the earlier attempt's markers are already replaced by each later one.

Tests

In short: the new-attempt test fails on 54b0c64 with assert {'step': 3} is None and passes with the change; the stale-attempt test passes on both and fails when the generation fence is removed. Red 3 of 3 runs, green 3 of 3 runs, 20 of 20 at -n0, 10 of 10 at -n 4.

Added to test/test_taskq_store.py, over a real SQLite TaskStore and the file's Clock (no sleeps, no patching):

  • _reclaim_after_a_lost_owner (:844): claims, starts and runs attempt one, records {"step": 3}, advances the clock past the lease, moves the row to recovering, and claims again.
  • test_a_new_attempt_starts_without_the_old_attempts_progress (:867): the second claim's generation is the first plus one, and both the ClaimResult.record and get() show progress is None.
  • test_a_stale_attempt_cannot_record_progress_after_the_reclaim (:885): the control. After the re-claim, record_progress with the first generation returns False and leaves the row as it was; with the current generation it returns True and stores the marker.

Before the change, on 54b0c64 with this branch's test file, python -m pytest -n0 -p no:cacheprovider -q test/test_taskq_store.py -k "a_new_attempt_starts_without or a_stale_attempt_cannot_record", RC=1 in 3 of 3 runs:

>       assert record.progress is None
E       AssertionError: assert {'step': 3} is None
test/test_taskq_store.py:879: AssertionError
FAILED test/test_taskq_store.py::test_a_new_attempt_starts_without_the_old_attempts_progress - AssertionError: assert {'step': 3} is None
============ 1 failed, 1 passed, 62 deselected, 2 warnings in 1.07s ============

With the change, the same command, RC=0 in 3 of 3 runs: 2 passed, 62 deselected.

Mutants, each one line, 3 of 3 runs each, RC=1:

  • progress_json=NULL removed from the claim UPDATE: only test_a_new_attempt_starts_without_the_old_attempts_progress fails.
  • AND generation=? in record_progress made AND (generation=? OR 1=1): only test_a_stale_attempt_cannot_record_progress_after_the_reclaim fails.

Repeats of the two new tests: 20 of 20 at -n0; 10 of 10 at -n 4 --dist loadgroup --max-worker-restart=2 with test/test_taskq_claim_fencing.py and test/test_taskq_reconcile.py as co-tenants; one shuffled run of those three files in seed 1354, 121 passed.

Neighbours, on this branch: test/test_taskq_store.py test/test_taskq_claim_fencing.py test/test_taskq_reconcile.py at -n0, 121 passed; the 67 test/test_*.py files that import taskq or name TaskStore, taskq_store, .claim(, claim_next( or record_progress, at -n 8, 3337 passed. The five ratchets (wait discipline, flake pattern, pid reader, tmpdir helpers, dir-fd create race) pass at -n0. The new tests patch nothing, so there is no module-global to leak.

Absence claims, with the commands: git grep -n progress_json -- . ':!docs' ':!test' lists only src/kiro_crew/taskq/migrate.py:65, src/kiro_crew/taskq/model.py:361, src/kiro_crew/taskq/model.py:433, src/kiro_crew/taskq/store.py:1061, src/kiro_crew/taskq/store.py:1215 and src/kiro_crew/taskq/store.py:1711; git grep -n -E '\.progress\b' -- src/kiro_crew/taskq src/kiro_crew/subagent_manager src/kiro_crew/dashboard/handlers/tasks.py src/kiro_crew/taskrunner.py src/kiro_crew/workflows lists only src/kiro_crew/taskq/model.py:390, the serializer.

Gates, from the repo's resolved profile, each exit code read before any pipe: black baseline gate, isort, flake8, mypy src/kiro_crew/, the comment-history gate with COMMENT_HISTORY_BASE_REF at the merge-base, docs lint, and the other diff-scoped gates, all RC=0.

Manual verification

N/A: the tests run the real UPDATE against a real SQLite file through TaskStore. No gateway or dashboard was started.

Related Issues

None open. Searched issues and PRs, open and closed, for progress_json, progress_json claim, taskq claim progress, stale progress marker claim, re-claimed progress marker and record_progress; none describes this defect.

Pattern harvest

Not generalizable: the claim statement is the one place a row starts a new attempt, and progress_json is the only per-attempt column it left behind (see result_ref and wait_json above). A lint would need to know which columns are per-attempt.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

…ress

TaskStore.claim hands a claimable row to a new generation. It bumps
generation and attempts and clears next_run_at, but it leaves
progress_json as the earlier attempt wrote it. A row that is claimed
again after its owner was lost therefore reads, through get() and the
record claim() returns, as a new attempt carrying the old attempt's
progress marker until the new attempt records its own.

Clear progress_json in the same UPDATE. A marker belongs to the attempt
that recorded it, and NULL is already how a row with no progress is
stored. The generation fence is unchanged: record_progress still
refuses any generation other than the row's current one, so a stale
attempt cannot write a marker after the re-claim.

One test records progress under the first attempt, lets the lease lapse
on a manual clock, claims again and checks that the marker is gone and
the generation advanced. A second test checks that the stale attempt is
still refused after the re-claim. The spec's claim statement names the
cleared column.
@javenciu
javenciu requested review from a team and KiroCrewCaptain as code owners October 13, 2026 13:07
@javenciu
javenciu requested a review from mbriones98 October 13, 2026 13:07
@javenciu

Copy link
Copy Markdown
Contributor Author

Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683).

@javenciu javenciu closed this Oct 13, 2026
@javenciu javenciu reopened this Oct 13, 2026
@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision ca84059c9607bd49e0176f1e4411d1a512e0e70a touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of ca84059c9607bd49e0176f1e4411d1a512e0e70a — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: PASS

No production code reads TaskRecord.progress (grep \.progress\b in src/kiro_crew/taskq: zero readers outside from_row), so the fix only matters later. Confirm no resume path expects the lost attempt's marker.

What this change ships

Inventory (4 items) — 4 justified

Intent: a task row taken up again as a new attempt should not show the lost attempt's last progress step as its own. This is a FIX.

  1. A re-claimed task row (generation N+1) now starts with no progress marker, instead of keeping generation N's marker — justified
  2. The claim docstring and the taskq spec now say the claim clears progress_json — justified
  3. A test that fails on base with assert {'step': 3} is None and passes with the fix — justified
  4. A control test showing the generation fence on record_progress still holds after a re-claim — justified

[FIRST-PRINCIPLES-REVIEWED] ca84059

Description read

[DESCRIPTION-READ] b1e8d681d153aabcfc5a3110d9006662311a4f32385c07b6645d436dc815ae58

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of ca84059c9607bd49e0176f1e4411d1a512e0e70a and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] ca84059

False positive or not applicable? A repository writer can comment:
/ai-review override gpt ca84059c9607bd49e0176f1e4411d1a512e0e70a: <one-sentence reason>

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of ca84059c9607bd49e0176f1e4411d1a512e0e70a — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Nothing to check.

[DESIGN-REVIEWED] ca84059

Description read

[DESCRIPTION-READ] b890738a6867abe3295184c71c44aaa0295edd61aa72615f5526562cb5346c7c

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed ca84059c9607bd49e0176f1e4411d1a512e0e70a — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] ca84059

Verdict parsed from the review's SHA-scoped output markers for commit ca84059c9607bd49e0176f1e4411d1a512e0e70a.

False positive or not applicable? A repository writer can comment:
/ai-review override fable ca84059c9607bd49e0176f1e4411d1a512e0e70a: <one-sentence reason>

@github-actions github-actions Bot added the readiness: passed Eligible automated validation passed for the current revision label Oct 13, 2026
@javenciu
javenciu enabled auto-merge (squash) October 13, 2026 13:45

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the task-store claim UPDATE now also sets progress_json=NULL, so a re-claimed attempt no longer carries the lost attempt's progress marker; two regression tests cover the reset and the generation fence. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/taskq.md.

@javenciu
javenciu merged commit f4b419e into main Oct 13, 2026
104 of 177 checks passed
@javenciu
javenciu deleted the fix/taskq-claim-clears-stale-progress branch October 13, 2026 14:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants