Repository navigation
test: bound held-side event waits by the lost-run bound and ratchet to_thread waits - #20418
Conversation
UX Review (Opus 5.5, fork) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing to check. What this change shipsInventory (7 items) — 6 justifiedIntent: stop the #20346-touched tests from failing on slow runners on the held side, and stop new tests from adding the parked
[FIRST-PRINCIPLES-REVIEWED] 8943e2f Description read[DESCRIPTION-READ] b5c49ba7e506b571c847c6b021d35a064c4afdc0390eb81cedebd683f4235b83 That sha256 names the PR description this verdict read. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the duplicate test fails by its own message when the hold times out: with a lost-run-sized bound, the job timeout could fire first and give no name. [DESIGN-REVIEWED] 8943e2f Description read[DESCRIPTION-READ] fdb8805aa5d74ada38db556dde8b8c5c502f5346b112fe272245c2ab633814bd That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings blocking merge. FINDING — src/kiro_crew/apps/builtins/spec_builder/tests/test_routes_delete.py:47 — the joins run one after another and each [OPUS-REVIEWED] 8943e2f |
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
cda3ae8 to
559d707
Compare
Independent review of cda3ae8, all items on 559d707:
Rules matched on the changed files:
First-pass check: R1-R8 OK |
First-pass check: R1-R8 OK |
559d707 to
8e5bc1e
Compare
|
Rebased onto current main (fbca1a1) as 8e5bc1e, still one commit. Conflict: main split the tail of Baseline: total unchanged at 184 sites, now 98 files instead of 97. Main's meetings route split (#20460) moved three sites out of
|
8e5bc1e to
6515a1c
Compare
|
Disposition for head 6515a1c (rebased onto main 9a77e12, one commit, test-only). It answers the independent review of 8e5bc1e. SF-1, two tree-scan timeout constants (fixed by deletion).
SF-2, positive control needed a live
NIT, missed shapes (done). The module docstring now lists three more misses: a function alias of NIT, the two-burn-down test never merges (declined). It stays as a baseline-composition test. The review ran a real Also fixed: mypy reported two errors in Per-file baseline scenarios on this head (scratch tree, each reverted):
Gates:
The lanes on 8e5bc1e had two reds:
First-pass check: R1-R8 OK. AUTOSDE rules that apply to the changed test file:
|
First-pass check: R1-R8 OK. Local gates on this head: both ratchet files plus every changed test file at |
|
|
Disposition for the delta review of c546711, now at head 2063959.
Gates on this head: First-pass check: R1-R8 OK. |
2063959 to
1798880
Compare
…o_thread waits The settlement tests in test_auto_research.py and the spec_builder route tests were moved off `asyncio.to_thread(<event>.wait, N)` earlier. Their held side still bet on the runner's speed: a patched write blocked on `assert <release>.wait(5)` while the test drove executor-backed I/O. In test_slow_settlement_cannot_remove_resumed_replacement and test_duplicate_reserves_its_name_before_populating_the_destination that I/O is a store write or a full create, so a slow runner made the held thread give up first and the test failed with a bare assert or a 500. With a one-thread default executor both fail every time at those lines. Every held-side fixed bound in test_auto_research.py and the spec_builder route tests now waits on default_timeout(), the lost-run bound, and a hold the test never releases fails by name at that bound, including one raised inside a request handler or a plain thread. The one-thread executor pin moves into kiro_crew.testing as the pytest fixture one_pool_thread (kiro_crew.testing.fixtures). Both test files import it, so the duplicated fixtures are deleted. test_flake_pattern_ratchet.py gains the K15 detector for the parked-wait shape: `asyncio.to_thread(<obj>.wait, ...)` and `loop.run_in_executor(<any>, <obj>.wait, ...)`, counted at any bound, whether a literal, a call such as default_timeout(), or none. Callees are matched by their last name, so an aliased asyncio module counts, and a name bound by `from asyncio import to_thread as tt` resolves to to_thread. Like the other detectors it holds each touched file to its count at the change's merge-base, forgives a site that moved, and is excused by `# flake-ok: <reason>`. No baseline is stored, so merged changes that add sites elsewhere cannot red this one.
1798880 to
8943e2f
Compare
KiroCrewCaptain
left a comment
There was a problem hiding this comment.
Tier 1 auto-approve: test (9 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: bounds test waits and adds a pytest-only one_pool_thread fixture plus a ratchet test; test helpers only.
Problem / Motivation
Goal: The tests #20346 touched no longer depend on runner speed on either side of their waits, and no new test adds the
asyncio.to_thread(<event>.wait, N)shape.#20346 fixed the waiting side of the spec-builder nudge-pause and auto-research settlement tests. That covered half of the class. The held side of the same tests still made the same D1 bet (
docs/system-specs/common/testing-conventions.md): a patched write blocks onassert <release>.wait(5)while the test drives executor-backed I/O. In two of them that I/O is real durable work:test_slow_settlement_cannot_remove_resumed_replacementholds the status write while the test awaits an AutoNudge store write;test_duplicate_reserves_its_name_before_populating_the_destinationholds a save while the test runs a fullPOST /specscreate.When that I/O takes longer than 5 s, the held thread gives up first, and the test fails with a bare
assert Falseor a 500 from the handler: the #20345 symptom again. With a one-thread default executor, both fail in 30 of 30 runs on main.Why it matters
These two tests can still red unrelated pull requests on a slow Windows shard. Nothing stops new tests from adding the parked-wait shape #20345 traced either, because no ratchet counts it.
Not a goal
kiro_crew.testingchange.What changed (motivation → approach → change)
test_auto_research.pyand thespec_builder/tests/test_routes*.pyfiles waits ondefault_timeout(), the lost-run bound. A hold the test never releases fails by name at that bound, including a hold inside a request handler or on a plain thread, which records the failure for the test to assert.one_pool_threadfixtures become one pytest fixture insrc/kiro_crew/testing/fixtures.py, behind its existing pytest guard.test_auto_research.pyandtest_routes_nudge_loops.pyimport it.test/test_flake_pattern_ratchet.pygains detector K15,K15-parked-pool-wait, for athreading.Eventwait a test parks on an executor.The release is the test's own signal, so the larger bound loosens nothing. It turns "the test never released" into a named failure. The two load-bearing tests do not get the one-thread pin, because they hold a pool thread on purpose while the test needs a second one, so with one thread they would sit out the lost-run bound.
K15 counts
to_thread(<obj>.wait, ...)andrun_in_executor(<any>, <obj>.wait, ...)whatever the bound: a literal, a call such asdefault_timeout(), or none. The parked wait holds a pool thread the code under test may need to make the event fire, so a larger bound only changes how the starvation fails. The callee is matched by its last name, so an aliased module counts too, and a name bound byfrom asyncio import to_thread as ttresolves toto_thread. Known misses, listed in the docstring:functools.partial, alambda, a reboundw = ev.wait, a rebound callee,executor.submitandgetattr. Like every other detector it holds each file a change touches to its count at the change's merge-base, forgives a site that moved, and is excused by# flake-ok: <reason>. The module docstring lists it with the other shapes.Why not a stored baseline: the flake ratchet's docstring records that judging against a stored count would red a change for sites that other, already merged changes added, which is the churn the earlier per-file baseline cost this PR on every rebase.
test/test_wait_discipline_ratchet.pyis back to main's shape andtest/to_thread_wait_baseline.txtis gone.Backwards compatibility
Compatible. The
one_pool_threadfixture is an addition, and nothing is removed fromkiro_crew.testing.Tests
the test never released the status writeandthe test never released the held save. With two threads they pass. With the release removed, the duplicate test fails at 63 s withAssertionError: the test never released the held save, where it used to fail with a 500.-n 8, and 30 of 30 under 32 CPU burners.default_timeout(), a from-import, an aliased from-import, a module alias,run_in_executorwith and without a bound) each count once; its near miss (another callable,is_set, polling from the loop, a held-side wait, a docstring mention) counts nothing. The suite's mutation proof turns a blind K15 into a failed positive case. Ignoring the from-import aliases fails the aliased case (0 == 1), as does the detector at c546711.await asyncio.to_thread(ev.wait, default_timeout())totest/test_auto_research.pyfailstest_no_changed_file_gains_a_flaky_shapewithK15-parked-pool-wait 1 > 0; removing it passes.test_auto_research.py, thespec_builder/tests/test_routes*.pyfiles,test_flake_pattern_ratchet.py,test_wait_discipline_ratchet.pyand thekiro_crew.testingtests (test_testing_determinism_helpers.py,test_fixtures_helper.py) pass at-n 8(636 passed). A bareimport kiro_crew.testingwith pytest blocked loads no wait helper and no fixture.src/kiro_crew), the black gate and the comment-history gate pass. Semgrep 1.78.0 with the repository rules plus p/python, p/security-audit, p/secrets and p/owasp-top-ten reports 0 findings on changed lines (7 pre-existing elsewhere in those files).Manual verification
N/A: unit coverage is enough for a test and test-helper change, proven by the one-thread runs and the mutations above.
Related Issues
Part of #20345
Follow-up to #20346 (merged): the held-side remainder, and the ratchet shape #20345 recorded as a burn-down candidate.
test_app_mcp_tool_policy_scope.py; that is main-side, tracked in Windows: managed-tool policy scope treats an outside handler as managed (main red since #20530) #20682, and unrelated to this change.Pattern harvest
Rule candidate: lint
Pattern: a test parking a
threading.Eventwait on an executor under a fixed bound (asyncio.to_thread(<event>.wait, N),run_in_executor(<any>, <event>.wait, N)). This PR counts it as detector K15 intest/test_flake_pattern_ratchet.py, whatever the bound.Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)