Skip to content

fix(taskq): end the step when the store refuses a dependency wait - #38412

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/36003-refused-wait-ends-step
Nov 20, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/36003-refused-wait-ends-step

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A task step whose dependency wait the task store refuses (its row is now another owner's or has ended) stops instead of running the call again.

When a step hits a dependency signal such as a rate limit and no dependency coordinator is attached, RunnerAdmission.yield_dependency asks the task store to park the row in waiting_dependency. If the store refused that wait because the handle's generation was stale or the row had already ended, the step still returned "retry" after a zero sleep. The caller then ran the call again at once.

Why it matters

Re-running the call from an owner that no longer holds a running row could run the same call twice. The coordinator path already stops in this case; the coordinator-less path did not.

Not a goal

What changed (motivation → approach → change)

A task step that hits a rate limit after its queue row stopped being its own (another owner took it, or it was cancelled) stops instead of running the call a second time.

The cause: with no dependency coordinator attached, RunnerAdmission.yield_dependency treated the store refusing the wait like a store outage, but with a 0.0 wait, so the caller retried at once. A refusal is the store's answer that this owner's row is no longer running under its generation. Following the maintainer decision on the issue, a refused wait ends the step: yield_dependency returns False without sleeping and hands the lane slot back to the caller, whose terminal write then settles the handle. That write is generation-fenced too, so the store refuses it and the row stays as the other owner or the cancel left it. A store outage and the no-store case still wait out the signal's retry_at and retry, as before.

The taskq spec describes the refusal in its waits row, and its coordinator paragraph names the outage as the case the coordinator-less arm retries locally.

Backwards compatibility

Compatible: a step that used to re-run once more against a refused row now ends (fails) instead; nothing is removed from any API or config.

Removes nothing: only the return value on one refusal branch changes; no key, field, row, path or public function is removed or renamed.

Tests

  • test_a_wait_the_store_refuses_ends_the_step replaces test_a_wait_the_store_refuses_keeps_its_immediate_retry. It drives a real SQLite store with its real fences (no monkeypatched enter_wait) and is parametrized over a stale handle generation and a row cancelled before the wait. It pins that the call returns False, nothing sleeps, the slot is held for the caller, the store records the refusal, the row is untouched, and the caller's fail_async is refused too.
  • The outage test test_a_wait_the_store_cannot_record_still_honours_the_retry_instant is unchanged and still passes.

Proof: PROVEN test/test_taskq_runner_adapter.py::test_a_wait_the_store_refuses_ends_the_step[stale_generation], test/test_taskq_runner_adapter.py::test_a_wait_the_store_refuses_ends_the_step[cancelled_before_the_wait] (head 0d62893)

Determinism: the new test passed 20/20 consecutive runs. The whole file (71 tests) passed in a seeded shuffled order.

Gates at this head (Linux x86_64, Python 3.12): black, isort and flake8 on the changed files, mypy src/kiro_crew, docs lint, the black and subprocess-encoding checks, and the flake-pattern ratchet pass. The 96 test files that import kiro_crew.taskq or name its runner admission pass (2,863 passed, 18 skipped). The repository's semgrep rules found nothing on the diff. The full gate list, the related backend run (1,283 test files) and the related frontend run (626 files) included, passed on the identical patch at 917e2ad (Ubuntu 24.04, x86_64).

Manual verification

There is no dashboard or CLI action that triggers this race on demand, so it was reproduced at the exact code path. A script admits a row in a real temporary SQLite TaskStore, marks it running, makes the store refuse the wait with its own fences, and calls yield_dependency with a rate-limit signal (retry_at = now + 30).

harness.py
"""Drive RunnerAdmission.yield_dependency against a real sqlite TaskStore."""
import asyncio, os, sys, tempfile
from pathlib import Path
sys.path.insert(0, os.path.join(os.environ["KC_TREE"], "test"))
from overload_fakes import Clock, open_task_store
from taskq_runner_adapter_helpers import _admission
from kiro_crew.taskq import model as m
from kiro_crew.taskq.dependency import KIND_RATE_LIMITED, DependencySignal

async def case(fence):
    clock = Clock()
    with tempfile.TemporaryDirectory() as d:
        gen = open_task_store(Path(d), clock)
        store = next(gen)
        adm, sleeps = _admission(store, clock, cap=1)
        rec = adm.accept(kind=m.KIND_WORKFLOW_AGENT, task_id="workflow:w:agent1")
        h = await adm.admit(rec.id, kind=m.KIND_WORKFLOW_AGENT)
        h.running()
        if fence == "stale_generation":
            h.generation += 1
        else:
            store.cancel(rec.id, reason="user_stop")
        sig = DependencySignal(kind=KIND_RATE_LIMITED, dependency_scope="api", source="repro", retry_at=clock.t + 30)
        out = await adm.yield_dependency(h, sig)
        row = store.get(rec.id)
        evs = [(e.kind, e.data) for e in store.events(rec.id)][-1]
        print(f"{fence}: yield_dependency -> {out}  (True = caller re-runs the call)  sleeps={sleeps.calls} "
              f"lane.running={adm.lane.running} row.state={row.state} last_event={evs}")
        try: next(gen)
        except StopIteration: pass

async def main():
    for f in ("stale_generation", "cancelled_before_wait"):
        await case(f)
asyncio.run(main())

Happy path, unchanged: test_rate_limit_parks_in_waiting_dependency_and_tick_wakes_it (a wait the store takes still parks and tick() wakes it), the store-outage test (still waits out 30 s and retries) and the coordinator schedule test all pass: 5 passed.

Real-machine verification

Machine: EC2 m7i.2xlarge, Ubuntu 24.04.5 LTS, x86_64, 8 vCPU / 32 GiB, Python 3.12

Before (main 5df30b7): KC_TREE=$T PYTHONPATH=$T/src python harness.py -> both fences return True after a zero sleep, so the caller re-runs the call:

stale_generation: yield_dependency -> True  (True = caller re-runs the call)  sleeps=[0.0] lane.running=1 row.state=running last_event=('stale_result', {'current': 1, 'from_generation': 2, 'wanted': 'waiting_dependency'})
cancelled_before_wait: yield_dependency -> True  (True = caller re-runs the call)  sleeps=[0.0] lane.running=1 row.state=cancelled last_event=('stale_result', {'current': 2, 'from_generation': 1, 'wanted': 'waiting_dependency'})

After (head 917e2ad): the same command -> both return False with no sleep, the slot held for the caller and the row as the fence left it:

stale_generation: yield_dependency -> False  (True = caller re-runs the call)  sleeps=[] lane.running=1 row.state=running last_event=('stale_result', {'current': 1, 'from_generation': 2, 'wanted': 'waiting_dependency'})
cancelled_before_wait: yield_dependency -> False  (True = caller re-runs the call)  sleeps=[] lane.running=1 row.state=cancelled last_event=('stale_result', {'current': 2, 'from_generation': 1, 'wanted': 'waiting_dependency'})

Both runs also print the store's on-loop connection warning, raised by the script's own synchronous adm.accept() setup call; it is left out above.

Cleanup: no gateway or service was started; the store lived in a temporary directory the script deletes; the instance was a disposable verification host with no inbound access, and it has been terminated.

Related Issues

Closes #36003

Sibling sweep

Swept: git grep -n "enter_wait" -- src/
Sibling: src/kiro_crew/taskq/adapters/runner.py -- changed here
Sibling: src/kiro_crew/subagent_manager/admission/waits.py -- not affected -- a posted wait mark that makes no retry decision
Sibling: src/kiro_crew/taskq/waits.py -- not affected -- WaitLedger.enter returns the store's answer to its caller unchanged

waiting_input in the same file already ends the step on a refused wait.

Pattern harvest

Not generalizable: the one branch here that collapsed "store refused" into "store unavailable" is now split; the sibling waits already tell the two apart.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributions are accepted under the project's Apache-2.0 license; see the Licensing section of CONTRIBUTING.md.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks: this is a one-branch fix for issue #36003, and the coordinator path already takes the same exit (runner.py:1046-1049).

Inventory (3 items) — 3 justified

Intent: a task step whose queue row is now another owner's, or has been cancelled, stops when the store refuses its dependency wait, instead of running the call a second time. This is a FIX.

  1. When the store refuses the wait, yield_dependency now returns False with no sleep, so the step ends and the call does not run again. — justified
  2. The retry-on-outage and no-store paths are unchanged: they still wait until retry_at, then retry. — justified
  3. The waits row and the coordinator paragraph in the taskq spec now say a refusal ends the step and an outage retries. — justified

[FIRST-PRINCIPLES-REVIEWED] 0d62893

Description read

[DESCRIPTION-READ] 1052c4c25590984eea75e5e2c0d1b5ca2de33c7b5e0ad3970db3ac9d1acef775

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. Both callers (task_executor.py:1398, workflows/agent_pool.py:726) already handle False with a fenced terminal write, which settles the handle and gives back the slot this branch takes back.

Notes

[DESIGN-REVIEWED] 0d62893

Description read

[DESCRIPTION-READ] 4d03e435efde00512ce7e7e3ab766e28164d76330859fc0a07b6a484b18eac23

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0d62893

Verdict parsed from the review's SHA-scoped output markers for commit 0d628932c9e6bdbb6c2bcfa927241835d3fc3689.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0d62893

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0d628932c9e6bdbb6c2bcfa927241835d3fc3689 false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 17, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/taskq.md since this head was pushed (c22557c -> ae205cb); CI re-running.

The patch is unchanged. The re-sync clears SAST (Semgrep): on the old base its diff-only scan covered every file main had changed since c22557c (about 6,000) and the engine ran out of memory.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/36003-refused-wait-ends-step branch from 4072d9e to 0f8e0fc Compare November 18, 2026 02:52
@bolichen97

bolichen97 commented Nov 18, 2026 •

Copy link
Copy Markdown
Contributor Author
  • span=eb3c6dd75d2b Superseding decisions require the maintainer override gate

Rebutted: not a defect in this PR. git diff --name-only ae205cb86d...0f8e0fc62371 lists only docs/system-specs/modules/taskq.md, src/kiro_crew/taskq/adapters/runner.py and test/test_taskq_runner_adapter.py; this PR adds, edits or supersedes no decision entry.
Both decision files the finding cites reached main through other PRs after this head's base (#38936 and #36888, both ancestors of the d4e7e9f base the lane read, neither of ae205cb), so they are main's lines, not this diff's.
The lane's own adjudication found the same: nothing here for the supersession override gate to approve.

[operator: bolichen97]

@bolichen97

Copy link
Copy Markdown
Contributor Author

/ai-review override gpt 0f8e0fc false-positive: agent: the cited decision files reached main through #38936 and #36888 after this head's base; this PR changes only the taskq spec, runner.py and its test, so it adds no supersession for the override gate to approve.

@github-actions

Copy link
Copy Markdown
Contributor

Human judgment recorded

@bolichen97 recorded a false-positive release (class false-positive) for the gpt AI finding on 0f8e0fc62371b08892e9cf469f3f716f6888e4dc.

agent: the cited decision files reached main through #38936 and #36888 after this head's base; this PR changes only the taskq spec, runner.py and its test, so it adds no supersession for the override gate to approve.

This decision applies only to this commit. A new push requires a new judgment.

@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/system-specs/modules/ (the directory of this PR's taskq.md; none of this PR's own files) since this head was pushed (ae205cb -> 5df30b7); CI re-running.

The patch is unchanged. The re-sync picks up the main fix for test/test_kas_policy_hidden_readiness.py (#38941, #38961): Backend Tests shard 2 failed on those two tests here and on main at the time. The review lanes on the old head failed reading the API while the shared rate limit was spent, and run again on this head.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/36003-refused-wait-ends-step branch from 0f8e0fc to 917e2ad Compare November 19, 2026 12:31
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
When the task store refused a coordinator-less dependency wait (the
handle's generation is stale or the row has ended), yield_dependency
still returned the signal's retryable flag after a zero sleep, so the
caller ran the call again at once from an owner that no longer holds a
running row. That could run the same call twice.

A refused wait now ends the step: the lane slot is taken back and
yield_dependency returns False, the same as the coordinator path. The
caller's fenced terminal write settles the handle or is refused. The
no-store and store-outage paths keep their local retry at retry_at.

The test that pinned the immediate retry is replaced by one driving
both real store fences (a stale generation and a row cancelled before
the wait), and the taskq spec describes the new behaviour.

Closes #36003
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in 252 commits, none touching this PR's three files, since this head was pushed (5df30b7 -> e06be9f); CI re-running. The patch is unchanged.

The previous head's reds were not code failures: Static Ratchet Gate and Flake Pattern Ratchet stopped because the CI runner could not install lsof while the Ubuntu package mirror answered 503, and Await Fast Gate, Coverage Gate and PR Readiness failed only because they wait on those two. Main's Fast Gate is green again, so nothing in the diff needed fixing.

Checked locally on the new head: black, isort and flake8 on the changed files, mypy src/kiro_crew, docs lint, the repository's semgrep rules, the flake-pattern ratchet, and the 96 test files that use the task queue (2,863 passed, 18 skipped). The fix's tests still fail with the fix reverted and pass with it.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/36003-refused-wait-ends-step branch from 917e2ad to 0d62893 Compare November 19, 2026 18:43
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 19, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention labels Nov 20, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed head 0d62893. When the store refuses a dependency wait because the row is now another owner's or has ended, the step now ends instead of running the call again at once. A store outage, or having no store at all, still waits until the signal's retry time and then retries. The store returns a refusal only for a missing row, a stale generation or a forbidden edge, never for an outage, and every caller writes the running mark before the step can wait. Both callers handle the new False result with a fenced terminal write, and that write gives the lane slot back. With the fix reverted the new parametrized test fails; on this head it passes along with the rest of the task queue wait suites, so nothing blocks.

@KiroCrewCaptain
KiroCrewCaptain merged commit d8cdc06 into main Nov 20, 2026
103 of 105 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/36003-refused-wait-ends-step branch November 20, 2026 02:35
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task queue: a refused dependency wait still re-runs the step at once

2 participants