Repository navigation
test(taskq): pin nine runner adapter invariants that carried no test - #36508
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The PR only adds one test file, which pins behaviour that is already correct, and it renders no UI and removes nothing. Notes
[DESIGN-REVIEWED] 410a2b4 Description read[DESCRIPTION-READ] f955ee7edff1762dc433f70212950c0af03defffa5053049be150644cb415fd6 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
1f4b70b to
da6b323
Compare
|
Rebased: main moved in docs/request-for-change/README.md since this head's CI ran (2f7e9f2 -> c000f31): the Source Gates docs-lint failure (docs/request-for-change/rfc-app-agent-tools-third-party-grant.md not reachable from any index, also red on main) was fixed on main by #36504, and Await Fast Gate, Coverage Gate and PR Readiness followed it; the diff is unchanged. CI re-running on da6b323. [operator: bolichen97] |
Code change approver recommendation: A person should decideWhy: the decider estimates a 4% chance that this change introduces a defect, within the 12% limit for approving on its own, but it changes 415 lines, more than the 400 it may approve on its own. The PR adds one test file and changes no production code. It pins nine runner-adapter invariant lines in runner.py, runner_lane.py and runner_handle.py. I traced the guards against the source and they match what the tests claim: the pause and spare-capacity guards in self_blocking_holders, acquire's FIFO fast path, both sleep bounds of 30 s (_MAX_ADMIT_SLEEP_SECS), the Tier: pending: being reviewed. An automated recommendation, not a review: a maintainer still decides. It approves on its own only when the estimated chance of a defect is at most 12% and the change has at most 400 changed lines. Decider v3c (217fe607) · commit |
|
[operator: bolichen97] The red Backend Tests shard 3 (Linux and Windows) on Current main is red on |
|
[operator: bolichen97] The search-docs red that held the last re-sync is fixed on main (#37063), but main is red again, now on the row-scrub tests: PR CI tests this branch merged into the current main tip, so a re-sync now would come back red on those tests. This PR waits for #37423, then re-syncs once onto a main that carries both #36652 and #37423. The diff here is unchanged: one new test file, and it still merges cleanly. |
|
Review passed on this head. Please rebase onto the latest main so CI can run fresh; this head is 503 commits behind main, and its red shards come from the 11-13 run (main's own red at that time, since fixed, plus one runner that lost its connection). The approval will follow once the rebased head is green. |
Part of #11314 (shard S02-taskq-runner). Nine lines in the runner adapters and their lane and handle owners guard real behaviour, yet deleting any one of them failed none of the tests that cover them: the deferred terminal-write replay, the idempotent slot release, two lane park guards, the lane's FIFO fast-path guard, the two admit sleep bounds, the idle-lane gate on the ancestor read, and the no-forget-while-a-write-is-owed guard. Each line gets one class in test_taskq_runner_invariant_pins.py that fails when the line is deleted, fails when it is loosened, and asserts the healthy side.
|
Rebased: main moved in test/test_mate_conductor_parity.py since this head's CI ran (c000f31 -> e72cf2e): the Backend Tests shard 3 red (Linux and Windows) on the old head was main's own [operator: bolichen97] |
da6b323 to
410a2b4
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
This review checked head 410a2b4. The PR adds one test file and changes no production code. Each of the nine pinned lines exists at head in runner.py, runner_lane.py and runner_handle.py. All 15 mutations named in the body were reapplied on this head, and each one fails this file with the count the body states. The existing 507 runner adapter tests stay green under the four key deletions, so those lines really had no test before. The tests are deterministic: they use an injected clock, recorded sleeps, tmp_path stores and bounded waits.
Done well: every pin checks the deleted form, the next weakest loosening and the healthy side. The "lane override" case is the only one that catches the ceiling <= 0 loosening, and it is included on purpose.
Optional:
- The two terminal write pins repeat the outage setup from test_settle_marks_settled_only_after_the_terminal_write_or_a_durable_retry in test_taskq_runner_adapter_durability.py. You could instead add a retry call made while the store is still down, plus a forget spy, to that test. Keeping a separate pins file also follows the existing precedent.
Problem / Motivation
Goal: Nine lines in the task queue's runner adapters that guard real behaviour get a test that fails when someone deletes or weakens them.
#11314 lists lines in the task queue whose deletion fails zero tests. #11721 and #31798 pinned the reachable lines of the store shard (
S01-taskq-store). This PR takes the next shard,S02-taskq-runner: the runner adapters inkiro_crew.taskq.adapters(runner.pyand itsrunner_lane.pyandrunner_handle.pyowners). On the base this branch is cut from, nine of their reachable lines are still unobserved: deleting any one of them leaves every test that covers the adapters green.Why it matters
Each of these lines can be simplified away by a refactor with nothing going red, and each one costs something real when it goes:
retry_terminal_writes'scontinueonTaskStoreUnavailablerunninguntil the next boot calls it interruptedrelease_slot'sif self._slot_held:fail()afterrecovering()frees another row's slot, and the lane runs one body over its capself_blocking_holders's pause guard (effective <= 0)self_blocking_holders's spare-capacity guardacquire'snot self._waitersfast-path guard_defer_for_pressure'smin(..., _MAX_ADMIT_SLEEP_SECS)admit's backoffmin(..., _MAX_ADMIT_SLEEP_SECS)admit'sand parksgate on the ancestor read_settle_apply's owed-write guardrunningNot a goal
What changed (motivation → approach → change)
The suite ran these lines only on their happy path, so every one was covered and none was observed. A line is only pinned by a test that watches it make a difference.
Following
AUTOSDE.yaml'san-invariant-line-needs-a-mutation-pin, each line gets one class intest/test_taskq_runner_invariant_pins.pythat fails when the line is deleted, fails when it is loosened to the next-weakest thing a reader might write, and asserts the healthy side. The two sleep bounds share one class; the two lane park guards get one class each, shown on one row below.TestARetryWhileTheStoreIsStillDownKeepsTheWritecontinuedoneTestASecondSlotReleaseGivesBackNothingTestAPausedLaneParksADescendant,TestSpareCapacityParksADescendantceilingmisses a lane override;<=is held by the existingtest_runner_lane_refuses_a_wait_only_its_own_ancestor_could_endTestALateArrivalQueuesBehindAParkedWaiterTestAnAdmitSleepIsBoundedSoTheRowIsRereadTestAnIdleLaneGrantsWithoutReadingTheAncestryrunning > 0TestATerminalWriteStillOwedIsNotForgottenif ok:keeps a fenced row for everThe tests drive the adapters through the shared
taskq_runner_adapter_helpersfixtures, an injected clock and recorded sleeps; no real time passes.Backwards compatibility
Compatible: the diff only adds tests.
Removes nothing: no production line changes.
Tests
test/test_taskq_runner_invariant_pins.pyadds 13 tests in the 8 classes above.Measured on Linux (Ubuntu 24.04, x86_64, Python 3.12.3) at base
2b01afa510with this file applied. Each mutation was applied to the source, the tests were run, and the source was restored:retry_terminal_writes'scontinuerelease_slot'sif self._slot_held:effective <= 0ceiling)<=reds the existingtest_runner_lane_refuses_a_wait_only_its_own_ancestor_could_endacquire'snot self._waitersand parksgateif ok:)The covering set is the 20 test files that reference the runner adapters: 525 tests, all green on the base.
Determinism: the 13 new tests passed 20 of 20 runs at
-n0; the file passed 10 of 10 runs at-n 4 --dist loadgroup --max-worker-restart=2besidetest_taskq_runner_adapter.pyandtest_taskq_runner_adapter_durability.py; the 13 ids passed in a shuffled order (seed 15241); and the covering set plus this file passed together (538 tests). The tests usetmp_pathstores, an injected clock and recorded sleeps: no wall-clock read or real sleep decides a verdict.Proof: not provable -- test-only change
Manual verification
N/A — unit coverage sufficient: the change is tests only, and the mutation tables above are the evidence that each test observes its line. The smoke run on the Linux host, with this file on base
2b01afa510:Screenshots / video
N/A — no user-visible surface; the diff is one test file.
Related Issues
Part of #11314
One more S02 line stays unpinned on purpose:
self_blocking_holders's untracked-holder guard (sum(self._holders.values()) < self._running). OnlyRunnerLane.acquire()with notask_idmakes it true, and both product callers (runner.py'sadmitandrunner_handle.py'sreacquire_slot) always pass the row id, so a pin there would freeze a path nothing reaches.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)Contribution License Agreement
Contributed under the project's Apache-2.0 license.