Skip to content

test(taskq): pin nine runner adapter invariants that carried no test - #36508

Merged
bolichen97 merged 1 commit into
mainfrom
fix/11314-taskq-runner-pins
Nov 17, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
fix/11314-taskq-runner-pins

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Nine lines in the task queue's runner adapters that guard real behaviour get a test that fails when someone deletes or weakens them.

#11314 lists lines in the task queue whose deletion fails zero tests. #11721 and #31798 pinned the reachable lines of the store shard (S01-taskq-store). This PR takes the next shard, S02-taskq-runner: the runner adapters in kiro_crew.taskq.adapters (runner.py and its runner_lane.py and runner_handle.py owners). On the base this branch is cut from, nine of their reachable lines are still unobserved: deleting any one of them leaves every test that covers the adapters green.

Why it matters

Each of these lines can be simplified away by a refactor with nothing going red, and each one costs something real when it goes:

the line if it silently disappears
retry_terminal_writes's continue on TaskStoreUnavailable a finished step's outcome is dropped while the store is down; the row stays running until the next boot calls it interrupted
release_slot's if self._slot_held: a fail() after recovering() frees another row's slot, and the lane runs one body over its cap
self_blocking_holders's pause guard (effective <= 0) a descendant that arrives while the lane is paused is refused instead of parked
self_blocking_holders's spare-capacity guard a descendant is refused while a slot is free
acquire's not self._waiters fast-path guard a late arrival takes a free slot ahead of a waiter that was parked first
_defer_for_pressure's min(..., _MAX_ADMIT_SLEEP_SECS) a pressure deferral sleeps the whole admit wait (up to an hour) blind, and a cancel meanwhile is not seen
admit's backoff min(..., _MAX_ADMIT_SLEEP_SECS) a backoff is slept in one piece, the same way
admit's and parks gate on the ancestor read every admission makes a store round trip between the saturation check and the grant, and another acquirer can take the slot in between
_settle_apply's owed-write guard a settle whose terminal write was deferred drops the row from the dependency schedule while the store still holds it running

Not a goal

What changed (motivation → approach → change)

The suite ran these lines only on their happy path, so every one was covered and none was observed. A line is only pinned by a test that watches it make a difference.

Following AUTOSDE.yaml's an-invariant-line-needs-a-mutation-pin, each line gets one class in test/test_taskq_runner_invariant_pins.py that fails when the line is deleted, fails when it is loosened to the next-weakest thing a reader might write, and asserts the healthy side. The two sleep bounds share one class; the two lane park guards get one class each, shown on one row below.

class deleted loosened healthy side
TestARetryWhileTheStoreIsStillDownKeepsTheWrite the write counts as settled the write is dropped before the continue the next replay commits done
TestASecondSlotReleaseGivesBackNothing the second release frees another row's slot guard reads the lane instead of the handle the lane refills to its cap
TestAPausedLaneParksADescendant, TestSpareCapacityParksADescendant a descendant is refused (paused lane, free slot) ceiling misses a lane override; <= is held by the existing test_runner_lane_refuses_a_wait_only_its_own_ancestor_could_end the raise or the pump serves it
TestALateArrivalQueuesBehindAParkedWaiter the late arrival overtakes none weaker: the guard is one emptiness test FIFO order kept
TestAnAdmitSleepIsBoundedSoTheRowIsReread one sleep of the whole wait bounded by the admit wait instead (for the pressure sleep, the same as deletion) admitted once the wait has passed
TestAnIdleLaneGrantsWithoutReadingTheAncestry ancestry read onto an idle lane gated on running > 0 read on a saturated lane, then served
TestATerminalWriteStillOwedIsNotForgotten a deferred settle is forgotten if ok: keeps a fenced row for ever committed and fenced settles are forgotten

The tests drive the adapters through the shared taskq_runner_adapter_helpers fixtures, an injected clock and recorded sleeps; no real time passes.

Backwards compatibility

Compatible: the diff only adds tests.

Removes nothing: no production line changes.

Tests

test/test_taskq_runner_invariant_pins.py adds 13 tests in the 8 classes above.

Measured on Linux (Ubuntu 24.04, x86_64, Python 3.12.3) at base 2b01afa510 with this file applied. Each mutation was applied to the source, the tests were run, and the source was restored:

line deleted: the 525 covering tests deleted: this file loosened: this file
retry_terminal_writes's continue 0 failed 1 failed 1 failed
release_slot's if self._slot_held: 0 failed 1 failed 1 failed
pause guard effective <= 0 0 failed 2 failed 1 failed (ceiling)
spare-capacity guard 0 failed 1 failed 0 failed; <= reds the existing test_runner_lane_refuses_a_wait_only_its_own_ancestor_could_end
acquire's not self._waiters 0 failed 2 failed n/a
pressure sleep bound 0 failed 1 failed 1 failed (same as deletion)
backoff sleep bound 0 failed 1 failed 1 failed
and parks gate 0 failed 1 failed 1 failed
owed-write forget guard 0 failed 1 failed 1 failed (if ok:)

The covering set is the 20 test files that reference the runner adapters: 525 tests, all green on the base.

Determinism: the 13 new tests passed 20 of 20 runs at -n0; the file passed 10 of 10 runs at -n 4 --dist loadgroup --max-worker-restart=2 beside test_taskq_runner_adapter.py and test_taskq_runner_adapter_durability.py; the 13 ids passed in a shuffled order (seed 15241); and the covering set plus this file passed together (538 tests). The tests use tmp_path stores, an injected clock and recorded sleeps: no wall-clock read or real sleep decides a verdict.

Proof: not provable -- test-only change

Manual verification

N/A — unit coverage sufficient: the change is tests only, and the mutation tables above are the evidence that each test observes its line. The smoke run on the Linux host, with this file on base 2b01afa510:

(a) -n0 x20: 20/20 green; last: 13 passed, 1 warning in 2.19s
(b) -n 4 co-tenants x10: 10/10 green; last: 111 passed, 5 warnings in 7.61s
(c) shuffled seed=15241, 13 ids: 13 passed, 1 warning in 2.23s
(d) covering set + new file: 538 passed, 332 warnings in 41.61s

Screenshots / video

N/A — no user-visible surface; the diff is one test file.

Related Issues

Part of #11314

One more S02 line stays unpinned on purpose: self_blocking_holders's untracked-holder guard (sum(self._holders.values()) < self._running). Only RunnerLane.acquire() with no task_id makes it true, and both product callers (runner.py's admit and runner_handle.py's reacquire_slot) always pass the row id, so a pin there would freeze a path nothing reaches.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

Contribution License Agreement

Contributed under the project's Apache-2.0 license.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The PR only adds one test file, which pins behaviour that is already correct, and it renders no UI and removes nothing.

Notes

  • Some pins check private details (the _ancestor_ids spy and _pending_finish membership) instead of what the system visibly does. That is acceptable for a mutation pin, but a refactor that keeps the behaviour and renames those internals will turn these tests red. Name the behaviour each one guards in its failure message so the person who breaks it knows what to restore.

[DESIGN-REVIEWED] 410a2b4

Description read

[DESCRIPTION-READ] f955ee7edff1762dc433f70212950c0af03defffa5053049be150644cb415fd6

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 410a2b4

Verdict parsed from the review's SHA-scoped output markers for commit 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 410a2b4

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 410a2b4c59fb9c57f0e7a0ca5a7a2a5fcd44a4cc false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 13, 2026 08:50
@bolichen97
bolichen97 force-pushed the fix/11314-taskq-runner-pins branch from 1f4b70b to da6b323 Compare November 13, 2026 14:31
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in docs/request-for-change/README.md since this head's CI ran (2f7e9f2 -> c000f31): the Source Gates docs-lint failure (docs/request-for-change/rfc-app-agent-tools-third-party-grant.md not reachable from any index, also red on main) was fixed on main by #36504, and Await Fast Gate, Coverage Gate and PR Readiness followed it; the diff is unchanged. CI re-running on da6b323.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 13, 2026
@KiroCrewCaptain

KiroCrewCaptain commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Code change approver recommendation: A person should decide

Why: the decider estimates a 4% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 415 lines, more than the 400 it may approve on its own.

The PR adds one test file and changes no production code. It pins nine runner-adapter invariant lines in runner.py, runner_lane.py and runner_handle.py. I traced the guards against the source and they match what the tests claim: the pause and spare-capacity guards in self_blocking_holders, acquire's FIFO fast path, both sleep bounds of 30 s (_MAX_ADMIT_SLEEP_SECS), the and parks gate, and the owed-write forget guard. The helpers and fixtures the file imports all exist. The tests use tmp_path stores, an injected clock and recorded sleeps, so no real time passes. CI is fully green, including the new-test stability jobs on Linux and Windows.

Tier: pending: being reviewed.

An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit 410a2b4c59 · 415 changed lines. If this recommendation is wrong, reply here: replies are how the approver is evaluated.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 13, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The red Backend Tests shard 3 (Linux and Windows) on da6b323271 is main's: test/test_mate_conductor_parity.py::test_every_shared_rule_has_an_ultra_counterpart_or_a_named_gap failed on main the same way and was fixed there by #36652, which landed after this head's base. Shard 7 lost its self-hosted runner, and the same shard also failed on main at that time.

Current main is red on test/test_search_docs.py::test_search_docs_ranks_the_page_the_query_is_about_first (tracked in #37060, fix in #37063). PR CI tests this branch merged into the current main tip, so a re-sync now would only swap one main red for the other. This PR waits for that fix, then re-syncs once onto a main that carries both. The diff here is unchanged: one new test file.

@bolichen97

Copy link
Copy Markdown
Contributor Author

[operator: bolichen97] The search-docs red that held the last re-sync is fixed on main (#37063), but main is red again, now on the row-scrub tests: test/test_one_conversation_one_session.py::TestOneRedactionRule::test_a_user_row_arriving_through_the_shared_helper_is_scrubbed, four tests in test/test_autonudge_row_scrub.py, test/test_history.py::TestAMiscomposedHostPersistsAWholeTurn::test_both_rows_persist and test/test_telegram_sessions_resume.py::TestTelegramColdResume::test_the_live_projection_receives_scrubbed_text_not_raw (main run https://github.com/kirodotdev/KiroCrew/actions/runs/43654430807). It is tracked in #37421, with the fix in #37423.

PR CI tests this branch merged into the current main tip, so a re-sync now would come back red on those tests. This PR waits for #37423, then re-syncs once onto a main that carries both #36652 and #37423. The diff here is unchanged: one new test file, and it still merges cleanly.

@buluoray

Copy link
Copy Markdown
Contributor

Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 503 commits behind main, and its red shards come from the 11-13 run (main's own red at that time, since fixed, plus one runner that lost its connection). The approval will follow once the rebased head is green.

Part of #11314 (shard S02-taskq-runner). Nine lines in the runner
adapters and their lane and handle owners guard real behaviour, yet
deleting any one of them failed none of the tests that cover them:
the deferred terminal-write replay, the idempotent slot release, two
lane park guards, the lane's FIFO fast-path guard, the two admit
sleep bounds, the idle-lane gate on the ancestor read, and the
no-forget-while-a-write-is-owed guard.

Each line gets one class in test_taskq_runner_invariant_pins.py that
fails when the line is deleted, fails when it is loosened, and asserts
the healthy side.
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in test/test_mate_conductor_parity.py since this head's CI ran (c000f31 -> e72cf2e): the Backend Tests shard 3 red (Linux and Windows) on the old head was main's own test_every_shared_rule_has_an_ultra_counterpart_or_a_named_gap failure, fixed on main by #36652, and the new base also carries #37423. Shard 7 on the old head lost its runner. The diff is unchanged: one new test file, the same patch as the reviewed head. CI re-running. @buluoray this is the rebase you asked for.

[operator: bolichen97]

@bolichen97
bolichen97 force-pushed the fix/11314-taskq-runner-pins branch from da6b323 to 410a2b4 Compare November 17, 2026 18:50
@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: passed Eligible automated validation passed for the current revision and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 17, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This review checked head 410a2b4. The PR adds one test file and changes no production code. Each of the nine pinned lines exists at head in runner.py, runner_lane.py and runner_handle.py. All 15 mutations named in the body were reapplied on this head, and each one fails this file with the count the body states. The existing 507 runner adapter tests stay green under the four key deletions, so those lines really had no test before. The tests are deterministic: they use an injected clock, recorded sleeps, tmp_path stores and bounded waits.

Done well: every pin checks the deleted form, the next weakest loosening and the healthy side. The "lane override" case is the only one that catches the ceiling <= 0 loosening, and it is included on purpose.

Optional:

  • The two terminal write pins repeat the outage setup from test_settle_marks_settled_only_after_the_terminal_write_or_a_durable_retry in test_taskq_runner_adapter_durability.py. You could instead add a retry call made while the store is still down, plus a forget spy, to that test. Keeping a separate pins file also follows the existing precedent.

@bolichen97
bolichen97 merged commit afa8f41 into main Nov 17, 2026
100 checks passed
@bolichen97
bolichen97 deleted the fix/11314-taskq-runner-pins branch November 17, 2026 20:44
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: passed Eligible automated validation passed for the current revision readiness: checking Automated validation is still running labels Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants