Repository navigation
fix(taskq): settle an unleased retry_wait row its tombstone proves ended - #22252
cruisercohen wants to merge 2 commits into
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the [DESIGN-REVIEWED] 29c9ef4 Description read[DESCRIPTION-READ] 7b8fa969d2a9da6b35f59d51cabbc37299ad701d58cfaf8f1f8172a53138cbc5 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
First Principles Review (Opus 5.5) — 🟡 CONCERNSPremise-level review of First-Principles-Verdict: CONCERNS The new boot pass won't clear the row that was actually reported, and #17161 already removed what produced it. Nothing names a remaining cause it would catch. Not justified as shipped
What this change shipsInventory (6 items) — 5 justifiedIntent: a gateway restart should free a chat slot that a stopped sub-agent's stranded
[FIRST-PRINCIPLES-REVIEWED] 29c9ef4 Description read[DESCRIPTION-READ] ecdab70486a2a05cc775f16cd12c19b8bd82bfd108c2b909419b92de98d0c789 That sha256 names the PR description this verdict read. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
|
Design review note on 85e99b5 (a parked row whose probe says |
|
HOLD (AI review check not answered; boot-time task ending reserved for human review) at 85e99b5. The pass is narrow. It writes only Blockers:
— crew-pr-reviewer (r113) |
a5b65c1 to
632155e
Compare
632155e to
29c9ef4
Compare
|
Reply to the HOLD at 85e99b5, on 29c9ef4. 1. First Principles: can a Fix, in the second commit:
Tests, through
(a) and (c) fail on 85e99b5 and on 632155e. A remaining path to this shape after #17161: the stop's settle is a best-effort store write. 2. A boot-time task ending reserved for human review: that part is a maintainer call, left to you. |
|
HOLD (reserved for human review: a boot-time pass that ends queued work) at 29c9ef4. The r113 question is answered. The pass now settles a parked row only on a stop recorded by its current attempt: the probe matches the tombstone's Blocker:
— crew-pr-reviewer (r114) |
reconcile_on_boot examined only the ACTIVE states, and retry_wait is CLAIMABLE, not ACTIVE. A retry_wait row left with no lease after its run's own terminal write was lost (a Stop whose cancel was dropped) was never looked at on restart, and count_pending kept counting it, so the parent slot's attached-children guard answered 409 slot_subagents_running on every turn until the store was edited by hand. Add a second, narrower boot pass over every unleased retry_wait row: read all pages first, then settle a row whose artifact probe proves an ending the transition table accepts from retry_wait (failed or cancelled), fenced on the state and generation the pass read. done is not written (retry_wait -> done stays closed), and started or no verdict leaves the row to its dispatcher. Fixes #22244
A claim reads no artifact and the run folder is the row id's, so a retry_wait row whose ending was lost can be claimed and run again over the folder its earlier attempt ended in. The boot pass then read that earlier ending as the parked attempt's and ended live work. An attempt is now named by the generation of the claim that began it: the claimed event records it, the registered run keeps it as _taskq_attempt, and the run's tombstone records it as taskq_attempt, carried through a delivery rewrite. The retry_wait pass calls the artifact probe with attempt=<latest claim generation>, and the subagent probe answers only from a tombstone naming that attempt; no wall clock is compared. A tombstone that names no attempt, written before this change, settles nothing. The pass also writes only cancelled: a wait wake or a park parks a row for a retry without a claim, so a failed ending there is left to the dispatcher.
29c9ef4 to
a033334
Compare
|
Thanks for this PR. It now conflicts with main. Could you rebase onto the latest main and resolve the conflicts? We will review it again once it is rebased and CI is green. |
|
[operator: bolichen97] Continuing this in #33155; thanks for the start. |
…33155) A sub-agent row left in retry_wait with no lease, whose run's tombstone already records a user stop, was never settled: the boot reconcile's queued pass reads retry_wait rows but acts only on the refusal and queued-stop markers, so count_pending kept the parent slot answering 409 slot_subagents_running on every turn until the store was edited by hand. The queued pass now cancels such a row when the stop belongs to the row's current attempt. An attempt is named by the generation of the claim that began it: the registered run keeps it as _taskq_attempt, its tombstone records it as taskq_attempt (carried through the delivery rewrite), and the pass asks the subagent probe about the generation of the row's latest claimed event. A stop from an earlier attempt, or one that names no attempt, leaves the row as it was. The cancel is fenced on retry_wait and the generation read. Builds on #22252 by @cruisercohen. Closes #22244
…33155) A sub-agent row left in retry_wait with no lease, whose run's tombstone already records a user stop, was never settled: the boot reconcile's queued pass reads retry_wait rows but acts only on the refusal and queued-stop markers, so count_pending kept the parent slot answering 409 slot_subagents_running on every turn until the store was edited by hand. The queued pass now cancels such a row when the stop belongs to the row's current attempt. An attempt is named by the generation of the claim that began it: the registered run keeps it as _taskq_attempt, its tombstone records it as taskq_attempt (carried through the delivery rewrite), and the pass asks the subagent probe about the generation of the row's latest claimed event. A stop from an earlier attempt, or one that names no attempt, leaves the row as it was. The cancel is fenced on retry_wait and the generation read. Builds on #22252 by @cruisercohen. Closes #22244 Original author: Bolin Chen (cherry picked from commit e7836f7)
Pull request was closed
Problem / Motivation
Goal: A gateway restart clears a chat slot that a stranded
retry_waitsub-agent row left answering 409 on every turn.When a sub-agent run's own terminal write is lost while its task row sits in
retry_waitwith no lease (the shape #17158 described: a Stop whose cancel was refused, with the run's tombstone already recording the stop), the row is never settled.count_pendingkeeps counting it,_attached_verdictkeeps treating the parent slot as attached, and every turn on that slot gets409 slot_subagents_running. In the reported case the row was also not re-claimed across two gateway restarts, and the only repair was a hand edit of the task store.Why it matters
The parent chat slot is unusable until someone edits the task store by hand, and restarting the gateway, the obvious recovery, does not help. #17161 stops the dropped cancel that produced this shape; a row already stranded, or one that reaches the same shape another way, stays wedged.
Not a goal
queuedrows or any other claimable state at boot (open fix(subagent): settle a refused task-queue row before publishing #18165 adds a separatequeuedpass for refusal markers).retry_wait -> doneedge. It is closed on purpose, so a parked row whose artifacts prove it finished is left as it is.failedending. A wait wake or a park also parks a row for a retry, so a failure there is left to the dispatcher.What changed (motivation → approach → change)
Symptom: a slot answers 409 forever after a stopped child. Root cause:
reconcile_on_boot(taskq/reconcile.py) readsstore.active_rows(), which selects theACTIVEstates;retry_waitis inCLAIMABLE, notACTIVE, so a restart never looks at the row.Change: after the existing
ACTIVEpass, a second, narrower pass reads everyretry_waitrow with no lease (all pages first, then the writes, so a settled row cannot shift a later page) and settles it on ONE ending: a stop its current attempt recorded, written ascancelwithonly_from={retry_wait}and the generation the pass read. Every other verdict leaves the row to its dispatcher as today:donehas no edge fromretry_wait;startedor no verdict is the dispatcher's to re-run; andfailedis not written, because a wait wake or a park also lands a row inretry_waitfor a retry without writing a claim, so a failure there cannot be told apart from one the dispatcher was meant to retry (#22244 reports a stop). A probe that raises is recorded inreport.errors, so_log_unsettlednames the row at boot, and leaves that row alone. The pass is counted in a newReconcileReport.parked_examined, soexaminedkeeps meaning theACTIVEpass.docs/system-specs/modules/taskq.mddescribes the pass.The stop must be the CURRENT attempt's, matched by identity, not by time (second commit). A claim reads no artifact and the run folder is the row id's, so a row whose ending was lost can be claimed and run again over the folder its earlier attempt ended in; the in-memory guard against that,
_stopped_while_claimed, reads only_agents, which a restart or a dismissed card empties. So an attempt is named by the generation of the claim that began it:claimedevent already records that generation;SubagentInfo._taskq_attempt, set at registration beside_taskq_generation(a wait wake moves_taskq_generation, never the attempt);_write_tombstone) records it astaskq_attempt, and a later bookkeeping rewrite that carries no ending of its own (the delivery acknowledgement) carries it forward, as it carriesoutcomeandleaked_process.The pass reads each row's latest
claimedevent (store.latest_events, one query per page) and calls the sameartifact_probewithattempt=<that generation>; theArtifactProbetype says so, and theACTIVEpass still calls it without. The subagent probe (taskq_artifact_probe) then answers only from a tombstone whosetaskq_attemptequals that attempt. No wall clock is compared anywhere.Every case it cannot match leaves the row untouched, exactly as on main:
kirocrew doctor(feat(doctor): name task rows that are not terminal, unleased and long past due #22270) already names such a row once its due time is long past, the reported shape, as stranded with no lease;claimedevent, or a page whose claim history cannot be read (each row named inreport.errors);attempt(it raises, which is recorded inreport.errors).With no
artifact_probethe pass reads no row. The one production caller,taskq_open, passes the subagent probe, so the boot open gets the fix.Alternative considered: adding
retry_waittoACTIVE. Rejected, because theACTIVEpass also requeues and recovers rows by side-effect class, which would re-run work the dispatcher already owns.Backwards compatibility
Compatible: the pass only writes
cancelled, and only on an unleasedretry_waitrow whose current attempt's own tombstone records a stop; every other row, state and verdict is handled exactly as before, and a tombstone written before this change settles nothing.Removes nothing: no key, field, function or validator is removed or tightened.
ReconcileReportgainsparked_examined,SubagentInfogains_taskq_attempt, tombstones gain an optionaltaskq_attemptkey that no existing reader requires, andtaskq_artifact_probegains an optionalattempt, which theretry_waitpass passes and theACTIVEpass does not.Tests
Failing first on origin/main 3e39e7a, passing with the fix (
test/test_taskq_reconcile.py):test_an_unleased_retry_wait_row_whose_run_was_stopped_is_cancelled_at_boot: the reported shape (claimed,starting, the stop'stombstone.jsonwithcause: user_stop, outcome: stoppedandtaskq_attemptnaming that claim, released toretry_wait, no lease, due time six hours past) read through the real subagent probe. On main the row staysretry_waitand the parent'scount_pendingstays 1; with the fix the row iscancelled,count_pendingis 0 and the row cannot be claimed.test_an_unleased_retry_wait_row_is_settled_only_on_a_proven_stop:cancelledsettles (red on main);failed,done,startedand no verdict leave the row inretry_waitwith its generation and no lease. The probe is asked once, with the generation of the row's latest claim.test_a_retry_wait_probe_that_raises_leaves_its_row_and_settles_the_rest: a raising probe is named inreport.errors, its row is untouched, and the next row is still settled (red on main).The attempt match (second commit), through the gateway's own boot open (
taskq_open),test_the_boot_pass_settles_a_parked_row_only_on_its_current_attempts_tombstone:[False-True-0.0-cancelled], the stop names the attempt the latest claim began: the row is cancelled.[True-True-3600.0-retry_wait], clock rollback: the row is claimed and started again after the stop with the wall clock stepped back an hour, so the old tombstone'sdiedis LATER than the new claim. The row staysretry_waitwith its generation andcount_pending1. Fails on 85e99b5 and on the previous head 632155e (assert 'cancelled' == 'retry_wait'): both cancelled the live later attempt.[False-False-0.0-retry_wait], a tombstone with notaskq_attempt(written before this change): the row is left untouched. Fails on 85e99b5 and 632155e.[True-True-0.0-retry_wait], re-claimed with no rollback: untouched. Fails on 85e99b5.Also:
test_a_failed_run_woken_or_parked_into_retry_wait_is_left_to_its_dispatcher[wake|park](a run that timed out while waiting, its row woken or parked intoretry_wait, its tombstone naming the current attempt: left alone; fails on 632155e);test_the_subagent_probe_answers_a_parked_row_only_for_its_own_attempt(matching, earlier, missing, boolean and stringtaskq_attempt, and theACTIVEcall with no attempt);test_a_parked_row_with_no_claim_is_not_asked_about;test_a_probe_that_cannot_name_the_attempt_settles_no_parked_row;test_a_retry_wait_row_whose_claims_cannot_be_read_is_named_and_left.The claim path carries the identity,
test_taskq_admission_integration.py::test_a_registered_run_carries_its_claims_attempt_into_its_tombstone: a real spawn through the manager and its store. The registered run's_taskq_attemptequals the generation of the row'sclaimedevent; a wake-shaped bump of_taskq_generationleaves it unchanged; the run's own_write_tombstonerecords it;mark_delivered's rewrite keeps it; and with the row released toretry_wait,reconcile_on_bootwith the manager's probe cancels it.Run locally: all 17
test/test_taskq*.pyfiles plus the 81 suites that open the store, call the boot reconcile, wait for the manager's store open, write or read tombstones, or read the run record's taskq fields, with the dependency coordinator and session health (4,692 passed together);test_taskq_bridge_bindings.pywithtaskq_artifact_probe's new signature pinned;test_taskq_unconditional_cancel_callers.pypasses with no new allowance because the new cancel carriesonly_from. black, isort and flake8 on the touched files, mypy over the whole package; the comment-history, brand, harness-parity, feature-map and docs-lint gates.Manual verification
N/A: unit coverage sufficient. The tests drive the real
TaskStorefile across a close and reopen, as a restarted gateway sees it, through the real subagent artifact probe.Related Issues
Fixes #22244
Pattern harvest
Not generalizable: the miss is one state outside one sweep's selection set, and the state machine already documents why
retry_waitis notACTIVE; the fix is a narrow pass for that state, not a rule other code can break the same way.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.