Repository navigation
fix(taskq): end the step when the store refuses a dependency wait - #38412
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks: this is a one-branch fix for issue #36003, and the coordinator path already takes the same exit ( Inventory (3 items) — 3 justifiedIntent: a task step whose queue row is now another owner's, or has been cancelled, stops when the store refuses its dependency wait, instead of running the call a second time. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 0d62893 Description read[DESCRIPTION-READ] 1052c4c25590984eea75e5e2c0d1b5ca2de33c7b5e0ad3970db3ac9d1acef775 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. Both callers ( Notes
[DESIGN-REVIEWED] 0d62893 Description read[DESCRIPTION-READ] 4d03e435efde00512ce7e7e3ab766e28164d76330859fc0a07b6a484b18eac23 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Rebased: main moved in docs/system-specs/modules/taskq.md since this head was pushed (c22557c -> ae205cb); CI re-running. The patch is unchanged. The re-sync clears SAST (Semgrep): on the old base its diff-only scan covered every file main had changed since c22557c (about 6,000) and the engine ran out of memory. [operator: bolichen97] |
4072d9e to
0f8e0fc
Compare
[operator: bolichen97] |
Human judgment recorded@bolichen97 recorded a false-positive release (class
This decision applies only to this commit. A new push requires a new judgment. |
|
Rebased: main moved in docs/system-specs/modules/ (the directory of this PR's taskq.md; none of this PR's own files) since this head was pushed (ae205cb -> 5df30b7); CI re-running. The patch is unchanged. The re-sync picks up the main fix for [operator: bolichen97] |
0f8e0fc to
917e2ad
Compare
When the task store refused a coordinator-less dependency wait (the handle's generation is stale or the row has ended), yield_dependency still returned the signal's retryable flag after a zero sleep, so the caller ran the call again at once from an owner that no longer holds a running row. That could run the same call twice. A refused wait now ends the step: the lane slot is taken back and yield_dependency returns False, the same as the coordinator path. The caller's fenced terminal write settles the handle or is refused. The no-store and store-outage paths keep their local retry at retry_at. The test that pinned the immediate retry is replaced by one driving both real store fences (a stale generation and a row cancelled before the wait), and the taskq spec describes the new behaviour. Closes #36003
|
Rebased: main moved in 252 commits, none touching this PR's three files, since this head was pushed (5df30b7 -> e06be9f); CI re-running. The patch is unchanged. The previous head's reds were not code failures: Static Ratchet Gate and Flake Pattern Ratchet stopped because the CI runner could not install Checked locally on the new head: black, isort and flake8 on the changed files, [operator: bolichen97] |
917e2ad to
0d62893
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
I reviewed head 0d62893. When the store refuses a dependency wait because the row is now another owner's or has ended, the step now ends instead of running the call again at once. A store outage, or having no store at all, still waits until the signal's retry time and then retries. The store returns a refusal only for a missing row, a stale generation or a forbidden edge, never for an outage, and every caller writes the running mark before the step can wait. Both callers handle the new False result with a fenced terminal write, and that write gives the lane slot back. With the fix reverted the new parametrized test fails; on this head it passes along with the rest of the task queue wait suites, so nothing blocks.
Problem / Motivation
Goal: A task step whose dependency wait the task store refuses (its row is now another owner's or has ended) stops instead of running the call again.
When a step hits a dependency signal such as a rate limit and no dependency coordinator is attached,
RunnerAdmission.yield_dependencyasks the task store to park the row inwaiting_dependency. If the store refused that wait because the handle's generation was stale or the row had already ended, the step still returned "retry" after a zero sleep. The caller then ran the call again at once.Why it matters
Re-running the call from an owner that no longer holds a running row could run the same call twice. The coordinator path already stops in this case; the coordinator-less path did not.
Not a goal
What changed (motivation → approach → change)
A task step that hits a rate limit after its queue row stopped being its own (another owner took it, or it was cancelled) stops instead of running the call a second time.
The cause: with no dependency coordinator attached,
RunnerAdmission.yield_dependencytreated the store refusing the wait like a store outage, but with a0.0wait, so the caller retried at once. A refusal is the store's answer that this owner's row is no longer running under its generation. Following the maintainer decision on the issue, a refused wait ends the step:yield_dependencyreturns False without sleeping and hands the lane slot back to the caller, whose terminal write then settles the handle. That write is generation-fenced too, so the store refuses it and the row stays as the other owner or the cancel left it. A store outage and the no-store case still wait out the signal'sretry_atand retry, as before.The taskq spec describes the refusal in its waits row, and its coordinator paragraph names the outage as the case the coordinator-less arm retries locally.
Backwards compatibility
Compatible: a step that used to re-run once more against a refused row now ends (fails) instead; nothing is removed from any API or config.
Removes nothing: only the return value on one refusal branch changes; no key, field, row, path or public function is removed or renamed.
Tests
test_a_wait_the_store_refuses_ends_the_stepreplacestest_a_wait_the_store_refuses_keeps_its_immediate_retry. It drives a real SQLite store with its real fences (no monkeypatchedenter_wait) and is parametrized over a stale handle generation and a row cancelled before the wait. It pins that the call returns False, nothing sleeps, the slot is held for the caller, the store records the refusal, the row is untouched, and the caller'sfail_asyncis refused too.test_a_wait_the_store_cannot_record_still_honours_the_retry_instantis unchanged and still passes.Proof: PROVEN test/test_taskq_runner_adapter.py::test_a_wait_the_store_refuses_ends_the_step[stale_generation], test/test_taskq_runner_adapter.py::test_a_wait_the_store_refuses_ends_the_step[cancelled_before_the_wait] (head 0d62893)
Determinism: the new test passed 20/20 consecutive runs. The whole file (71 tests) passed in a seeded shuffled order.
Gates at this head (Linux x86_64, Python 3.12): black, isort and flake8 on the changed files,
mypy src/kiro_crew, docs lint, the black and subprocess-encoding checks, and the flake-pattern ratchet pass. The 96 test files that importkiro_crew.taskqor name its runner admission pass (2,863 passed, 18 skipped). The repository's semgrep rules found nothing on the diff. The full gate list, the related backend run (1,283 test files) and the related frontend run (626 files) included, passed on the identical patch at 917e2ad (Ubuntu 24.04, x86_64).Manual verification
There is no dashboard or CLI action that triggers this race on demand, so it was reproduced at the exact code path. A script admits a row in a real temporary SQLite
TaskStore, marks it running, makes the store refuse the wait with its own fences, and callsyield_dependencywith a rate-limit signal (retry_at = now + 30).harness.pyHappy path, unchanged:
test_rate_limit_parks_in_waiting_dependency_and_tick_wakes_it(a wait the store takes still parks andtick()wakes it), the store-outage test (still waits out 30 s and retries) and the coordinator schedule test all pass:5 passed.Real-machine verification
Machine: EC2 m7i.2xlarge, Ubuntu 24.04.5 LTS, x86_64, 8 vCPU / 32 GiB, Python 3.12
Before (main 5df30b7):
KC_TREE=$T PYTHONPATH=$T/src python harness.py-> both fences return True after a zero sleep, so the caller re-runs the call:After (head 917e2ad): the same command -> both return False with no sleep, the slot held for the caller and the row as the fence left it:
Both runs also print the store's on-loop connection warning, raised by the script's own synchronous
adm.accept()setup call; it is left out above.Cleanup: no gateway or service was started; the store lived in a temporary directory the script deletes; the instance was a disposable verification host with no inbound access, and it has been terminated.
Related Issues
Closes #36003
Sibling sweep
Swept: git grep -n "enter_wait" -- src/
Sibling: src/kiro_crew/taskq/adapters/runner.py -- changed here
Sibling: src/kiro_crew/subagent_manager/admission/waits.py -- not affected -- a posted wait mark that makes no retry decision
Sibling: src/kiro_crew/taskq/waits.py -- not affected -- WaitLedger.enter returns the store's answer to its caller unchanged
waiting_inputin the same file already ends the step on a refused wait.Pattern harvest
Not generalizable: the one branch here that collapsed "store refused" into "store unavailable" is now split; the sibling waits already tell the two apart.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.