Repository navigation
fix(taskq): start a re-claimed attempt without the old attempt's progress - #19819
Conversation
…ress TaskStore.claim hands a claimable row to a new generation. It bumps generation and attempts and clears next_run_at, but it leaves progress_json as the earlier attempt wrote it. A row that is claimed again after its owner was lost therefore reads, through get() and the record claim() returns, as a new attempt carrying the old attempt's progress marker until the new attempt records its own. Clear progress_json in the same UPDATE. A marker belongs to the attempt that recorded it, and NULL is already how a row with no progress is stored. The generation fence is unchanged: record_progress still refuses any generation other than the row's current one, so a stale attempt cannot write a marker after the re-claim. One test records progress under the first attempt, lets the lease lapse on a manual clock, claims again and checks that the marker is gone and the generation advanced. A second test checks that the stale attempt is still refused after the re-claim. The spec's claim statement names the cleared column.
|
Closing and reopening right after opening so CI runs on hosted runners: same-repo runs from this account queue on the fleet label without a runner (details on #19683). |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS No production code reads What this change shipsInventory (4 items) — 4 justifiedIntent: a task row taken up again as a new attempt should not show the lost attempt's last progress step as its own. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] ca84059 Description read[DESCRIPTION-READ] b1e8d681d153aabcfc5a3110d9006662311a4f32385c07b6645d436dc815ae58 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing to check. [DESIGN-REVIEWED] ca84059 Description read[DESCRIPTION-READ] b890738a6867abe3295184c71c44aaa0295edd61aa72615f5526562cb5346c7c That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: fix (3 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: the task-store claim UPDATE now also sets progress_json=NULL, so a re-claimed attempt no longer carries the lost attempt's progress marker; two regression tests cover the reset and the generation fence. Spec files changed as a ride-along (a minority of the diff on both file count and changed lines), not reviewed as a design decision: docs/system-specs/modules/taskq.md.
Problem / Motivation
Goal: A task row that is claimed again as a new attempt starts with no progress marker, instead of carrying the marker the lost attempt left behind.
TaskStore.claim(src/kiro_crew/taskq/store.py:1056-1064on 54b0c64) re-takes a claimable row (queued,retry_waitorrecovering) as generation N+1. OneUPDATEsets the state, the lease,generation=generation+1,attempts=attempts+1andnext_run_at=NULL. It does not nameprogress_json.TaskStore.record_progress(src/kiro_crew/taskq/store.py:1702on 54b0c64) overwrites that column with the latest marker and refuses any generation other than the row's current one.attemptsone higher, together with the last marker the lost attempt recorded.TaskStore.getand theClaimResult.recordthatclaimreturns both show it, until the new attempt records its own.Why it matters
generationsays which attempt a row describes, and a marker from generation N on a row at generation N+1 says something false about it. Nothing renders the stale marker in this tree yet:row_payload(src/kiro_crew/dashboard/handlers/tasks.py:219) does not emitprogress, and the only reader of the column isTaskRecord.from_row(src/kiro_crew/taskq/model.py:433). The commands that show this are under Tests. A surface or a caller that readsrecord.progresswould show the lost attempt's last step as the new attempt's.Not a goal
progressin the task API or the dashboard.transition(progress=...)andrecord_progressare unchanged, and so is the generation fence onrecord_progress.wake_waitandcancel, the other statements that bumpgeneration(src/kiro_crew/taskq/store.py:1632,:1639,:1399). A wake torunningcontinues the same attempt with its runtime resident and does not bumpattempts, and the stop-recovery marker is written after it; a wake toretry_waitleaves the row claimable, and the claim that follows clears the marker.cancelends the row, so its last marker is the final record.What changed (motivation → approach → change)
claimreturns generation 2 withprogress == {'step': 3}, the marker of generation 1.UPDATEresets the lease andnext_run_atfor the new attempt and leavesprogress_jsonuntouched.progress_json=NULLjoins theSETlist of thatUPDATE(src/kiro_crew/taskq/store.py:1061), and theclaimdocstring says so (:1047-1049).NULLis already how a row with no progress is stored:TaskRecord.to_rowwrites it for a falsyprogress(src/kiro_crew/taskq/model.py:390), andTaskRecord.from_rowmaps the column back toprogress(src/kiro_crew/taskq/model.py:433), so a row that never recorded progress holds the sameNULL.docs/system-specs/modules/taskq.mdnames the column in the claim statement and in the paragraph below it.recoveringorretry_wait. While the row waits, the marker is the record of where the lost attempt got to; the claim is the one statement that starts a new attempt. Also rejected: stamping the marker with its generation so readers can compare. It changes whatrecord_progressstores for a problem the claim can fix in one column.result_refis the other column a lost attempt could leave, and the claim cannot meet one: only the terminalfinishpath writes it, and the terminal statesDONE,FAILEDandCANCELLEDhave no outgoing transition whileUNKNOWN_SIDE_EFFECTgoes only toDONEorFAILED(src/kiro_crew/taskq/model.py:151-155).wait_jsonis cleared on every exit from a wait (src/kiro_crew/taskq/store.py:1210,:1632,:1639).Backwards compatibility
Compatible: no stored value changes shape. The column keeps its type and encoding. A row written before this change keeps whatever marker it holds until the next claim clears it.
Reader:
src/kiro_crew/taskq/model.py:433TaskRecord.from_row-- every store read (subagent, runner and TaskRunner rows) -- mapsNULLtoprogress=None, the same as a row that never recorded progress (test_a_new_attempt_starts_without_the_old_attempts_progress).Reader:
test/test_subagent_stop_reason_consistency.py:316-- subagent stop recovery -- readsphase == "readmitted", whichrecord_progresswrites afterwake_wait, not afterclaim; the file passes unchanged.The column holds only the latest marker (
record_progressoverwrites it), so no history is lost: the earlier attempt's markers are already replaced by each later one.Tests
In short: the new-attempt test fails on 54b0c64 with
assert {'step': 3} is Noneand passes with the change; the stale-attempt test passes on both and fails when the generation fence is removed. Red 3 of 3 runs, green 3 of 3 runs, 20 of 20 at-n0, 10 of 10 at-n 4.Added to
test/test_taskq_store.py, over a real SQLiteTaskStoreand the file'sClock(no sleeps, no patching):_reclaim_after_a_lost_owner(:844): claims, starts and runs attempt one, records{"step": 3}, advances the clock past the lease, moves the row torecovering, and claims again.test_a_new_attempt_starts_without_the_old_attempts_progress(:867): the second claim's generation is the first plus one, and both theClaimResult.recordandget()showprogress is None.test_a_stale_attempt_cannot_record_progress_after_the_reclaim(:885): the control. After the re-claim,record_progresswith the first generation returnsFalseand leaves the row as it was; with the current generation it returnsTrueand stores the marker.Before the change, on 54b0c64 with this branch's test file,
python -m pytest -n0 -p no:cacheprovider -q test/test_taskq_store.py -k "a_new_attempt_starts_without or a_stale_attempt_cannot_record", RC=1 in 3 of 3 runs:With the change, the same command, RC=0 in 3 of 3 runs:
2 passed, 62 deselected.Mutants, each one line, 3 of 3 runs each, RC=1:
progress_json=NULLremoved from the claimUPDATE: onlytest_a_new_attempt_starts_without_the_old_attempts_progressfails.AND generation=?inrecord_progressmadeAND (generation=? OR 1=1): onlytest_a_stale_attempt_cannot_record_progress_after_the_reclaimfails.Repeats of the two new tests: 20 of 20 at
-n0; 10 of 10 at-n 4 --dist loadgroup --max-worker-restart=2withtest/test_taskq_claim_fencing.pyandtest/test_taskq_reconcile.pyas co-tenants; one shuffled run of those three files in seed 1354,121 passed.Neighbours, on this branch:
test/test_taskq_store.py test/test_taskq_claim_fencing.py test/test_taskq_reconcile.pyat-n0,121 passed; the 67test/test_*.pyfiles that importtaskqor nameTaskStore,taskq_store,.claim(,claim_next(orrecord_progress, at-n 8,3337 passed. The five ratchets (wait discipline, flake pattern, pid reader, tmpdir helpers, dir-fd create race) pass at-n0. The new tests patch nothing, so there is no module-global to leak.Absence claims, with the commands:
git grep -n progress_json -- . ':!docs' ':!test'lists onlysrc/kiro_crew/taskq/migrate.py:65,src/kiro_crew/taskq/model.py:361,src/kiro_crew/taskq/model.py:433,src/kiro_crew/taskq/store.py:1061,src/kiro_crew/taskq/store.py:1215andsrc/kiro_crew/taskq/store.py:1711;git grep -n -E '\.progress\b' -- src/kiro_crew/taskq src/kiro_crew/subagent_manager src/kiro_crew/dashboard/handlers/tasks.py src/kiro_crew/taskrunner.py src/kiro_crew/workflowslists onlysrc/kiro_crew/taskq/model.py:390, the serializer.Gates, from the repo's resolved profile, each exit code read before any pipe: black baseline gate, isort, flake8,
mypy src/kiro_crew/, the comment-history gate withCOMMENT_HISTORY_BASE_REFat the merge-base, docs lint, and the other diff-scoped gates, all RC=0.Manual verification
N/A: the tests run the real
UPDATEagainst a real SQLite file throughTaskStore. No gateway or dashboard was started.Related Issues
None open. Searched issues and PRs, open and closed, for
progress_json,progress_json claim,taskq claim progress,stale progress marker claim,re-claimed progress markerandrecord_progress; none describes this defect.Pattern harvest
Not generalizable: the claim statement is the one place a row starts a new attempt, and
progress_jsonis the only per-attempt column it left behind (seeresult_refandwait_jsonabove). A lint would need to know which columns are per-attempt.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)