Skip to content

test: wait on the store-write signal, not a 2-5 s thread wait - #20346

Merged
KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/windows-event-wait-starvation
Oct 14, 2026
Merged

KiroCrewCaptain merged 1 commit into
kirodotdev:mainfrom
jeeshofone:fix/windows-event-wait-starvation

Conversation

@jeeshofone

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: The spec-builder nudge-pause tests and the auto-research watchdog settlement tests pass on every runner instead of failing on slow Windows shards.

Backend Tests (Windows) (7) fails with a bare assert False on pull requests that touch none of these modules. Each failing test waits for an executor-backed durable write with assert await asyncio.to_thread(<event>.wait, 2 or 5). The failure recurred 3 times since 11 Oct, always as a pair on two xdist workers of the same shard (#20345 has the runs).

Why it matters

An unrelated red shard costs every open pull request a rerun or a fresh head, and the bare assert False names neither the write nor the state it reached.

Not a goal

  • Migrating the other asyncio.to_thread(<event>.wait, N) sites in the tree (about 168), or teaching the wait-discipline ratchet to count this shape. Flaky: nudge-pause and watchdog settlement tests on Windows (7) #20345 records both as burn-down candidates.
  • Any change to the AutoNudge store, the auto-research watchdog or the spec-builder routes. The production behaviour the tests check is correct.

What changed (motivation → approach → change)

Symptom: the wait returns False and the test fails. Cause: the bound is a fixed 2 s or 5 s bet on the runner's speed (rule D1 in docs/system-specs/common/testing-conventions.md), and the write behind it is a chain of temp-file creates, fsyncs, a retried rename and a directory sync. The 4-vCPU Windows runner does not always finish that inside the bound. The wait also parks on the loop's default executor, the pool the write itself needs. That is not what fails on CI (each test loop gets a fresh 8-thread pool, and at most 2 jobs are in flight in these tests). With a one-thread pool, though, the old shape fails every time.

Change: every asyncio.to_thread(<event>.wait, N) in the two files now uses async_wait_until(<event>.is_set, describe=...) from kiro_crew.testing.wait: 6 sites in test_routes.py and 5 in test_auto_research.py. The wait polls from the loop, holds no pool thread, and fails by name at the lost-run bound (half the test's timeout). The four tests that flaked request a new one_pool_thread fixture (one per file), which gives the loop a one-thread default executor.

#20345 shows why #20060's atomic_write change is not involved: none of these write paths call atomic_write().

Backwards compatibility

Compatible: test-only change. No production module is touched.

Removes nothing: the tests keep every assertion they had and wait on the same signals.

Tests

  • The four tests that flaked run on a one-thread default executor. With the old waits restored, all four fail at the reverted lines (mutation run on this head). With the new waits, all four pass.
  • src/kiro_crew/apps/builtins/spec_builder/tests/test_routes.py and test/test_auto_research.py in full: 809 passed with -n 4.
  • Determinism: the 21 tests around the changed waits passed 10 of 10 repeats with -n 4, and passed with -n 8 under a 32-process CPU spinner load. The four that flaked, plus their class siblings (13 tests), passed in reverse order with -n0.
  • With max_workers=1 on the loop, main's versions of the four tests fail at exactly the CI lines.

Manual verification

N/A — unit coverage sufficient: test-only change, proven by the one-thread pin and its mutation run.

Related Issues

Closes #20345

#20327 moves the nudge tests into test_routes_nudge_loops.py. Whichever of the two lands second needs a small rebase.

Pattern harvest

Rule candidate: lint
Pattern: asyncio.to_thread(<threading.Event>.wait, <number under 30>) in a test, a fixed short wait that also takes a default-executor thread from the code under test. test/test_wait_discipline_ratchet.py does not count it today.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

The spec-builder nudge-pause tests and the auto-research watchdog
settlement tests waited for an executor-backed durable write with
`assert await asyncio.to_thread(<event>.wait, 2 or 5)`. That bound is a
bet on the runner's speed (rule D1): the write it waits for is a chain of
temp-file creates, fsyncs, a retried rename and a directory sync, which
the 4-vCPU Windows runner does not always finish inside it, so the wait
returned False and the test failed with a bare `assert False`.

The wait also parked itself on the loop's default executor, the same
pool the write needs. Each wait now polls the threading.Event from the
loop with async_wait_until, which holds no pool thread and fails by name
at the lost-run bound. The four tests that flaked run on a one-thread
default executor, which makes the old shape fail every time and pins
that the wait takes nothing from the pool the code under test uses.
@jeeshofone
jeeshofone requested review from a team and KiroCrewCaptain as code owners October 14, 2026 15:14
@jeeshofone
jeeshofone requested a review from mbriones98 October 14, 2026 15:14
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: checking Automated validation is still running labels Oct 14, 2026
@github-actions

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ⏭️ skipped

Revision a8190326503cb9d17802925fc62fdc2b58468da6 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ⏭️ skipped

Revision a8190326503cb9d17802925fc62fdc2b58468da6 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of a8190326503cb9d17802925fc62fdc2b58468da6 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

The description says pool starvation isn't what fails on CI. Check that the longer lost-run bound alone covers the observed Windows timeouts.

[DESIGN-REVIEWED] a819032

Description read

[DESCRIPTION-READ] 26c2a60cb9ac4516bcccfa908afa5d2f50c5eb7ef6ae13321937a3d89886e9d0

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed a8190326503cb9d17802925fc62fdc2b58468da6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] a819032

@github-actions

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed a8190326503cb9d17802925fc62fdc2b58468da6 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] a819032

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 14, 2026

@mbriones98 mbriones98 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checks green and readiness passed at a819032; no behavior change.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 14, 2026 18:08

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tier 1 auto-approve: test (2 files). Criteria: no conflict, no requested changes, security path denylist clean, design-doc gate clean, security checklist all-NO, no reviewer-lane findings, readiness signal fresh for this head SHA, declared category matches the diff. Category: replaces thread-parked waits with async_wait_until and a one-thread executor fixture in two test modules; test files only.

@KiroCrewCaptain
KiroCrewCaptain merged commit f28a573 into kirodotdev:main Oct 14, 2026
103 checks passed
@jeeshofone

Copy link
Copy Markdown
Contributor Author

Follow-up for the independent review of a819032, which merged before this round could be pushed: #20418 (head cda3ae8).

  • S1 + N1, fixed: every held-side fixed bound in the two files (9 sites) now waits on default_timeout() with a message naming the hold. On a819032 the two load-bearing tests fail 60 of 60 runs under a one-thread executor. With 2 threads they pass 20 of 20, and a hold that is never released fails by name at 60.1 s. They keep no one-thread pin, because they hold a pool thread on purpose.
  • N2, fixed: one_pool_thread now lives in kiro_crew.testing (wait.use_one_pool_thread and the fixtures.one_pool_thread fixture), and the two duplicated copies are deleted.
  • N3 and the ratchet: test_wait_discipline_ratchet.py now counts to_thread(<obj>.wait, N) across test/ and src/**/tests, pinned at the measured 173. A planted site fails the gate.

First-pass check: R1-R8 OK.

@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 14, 2026
jeeshofone added a commit to jeeshofone/KiroCrew that referenced this pull request Oct 14, 2026
…o_thread waits

kirodotdev#20346 moved the waiting side of the spec-builder nudge-pause and
auto-research settlement tests off `asyncio.to_thread(<event>.wait, N)`.
The held side of the same tests still bet on the runner's speed: a
patched write blocked on `assert <release>.wait(5)` while the test drove
executor-backed I/O. In
test_slow_settlement_cannot_remove_resumed_replacement and
test_duplicate_reserves_its_name_before_populating_the_destination that
I/O is a store write or a full create, so a slow runner made the held
thread give up first and the test failed with a bare assert or a 500.
With a one-thread default executor both fail every time at those lines.

Every held-side fixed bound in test_auto_research.py and
spec_builder/tests/test_routes.py now waits on default_timeout(), the
lost-run bound, and a hold the test never releases fails by name at that
bound, including one raised inside a request handler or a plain thread.

The one-thread executor pin moves into kiro_crew.testing as the pytest
fixture one_pool_thread (kiro_crew.testing.fixtures); both test files
import it, so the duplicated fixtures are deleted.

test/test_wait_discipline_ratchet.py gains a second per-file ratchet,
test/to_thread_wait_baseline.txt, for a threading.Event wait parked on an
executor under a fixed bound: `asyncio.to_thread(<obj>.wait, N)`
(asyncio however imported or aliased) and
`loop.run_in_executor(<any>, <obj>.wait, N)`, across test/ and every
src/**/tests package. 184 sites in 97 files. It shares the existing
gate's baseline machinery, so two burn-downs in different files merge
without conflict and moving a site between files fails.
jeeshofone added a commit to jeeshofone/KiroCrew that referenced this pull request Oct 14, 2026
…o_thread waits

kirodotdev#20346 moved the waiting side of the spec-builder nudge-pause and
auto-research settlement tests off `asyncio.to_thread(<event>.wait, N)`.
The held side of the same tests still bet on the runner's speed: a
patched write blocked on `assert <release>.wait(5)` while the test drove
executor-backed I/O. In
test_slow_settlement_cannot_remove_resumed_replacement and
test_duplicate_reserves_its_name_before_populating_the_destination that
I/O is a store write or a full create, so a slow runner made the held
thread give up first and the test failed with a bare assert or a 500.
With a one-thread default executor both fail every time at those lines.

Every held-side fixed bound in test_auto_research.py and
spec_builder/tests/test_routes.py now waits on default_timeout(), the
lost-run bound, and a hold the test never releases fails by name at that
bound, including one raised inside a request handler or a plain thread.

The one-thread executor pin moves into kiro_crew.testing as the pytest
fixture one_pool_thread (kiro_crew.testing.fixtures); both test files
import it, so the duplicated fixtures are deleted.

test/test_wait_discipline_ratchet.py gains a second per-file ratchet,
test/to_thread_wait_baseline.txt, for a threading.Event wait parked on an
executor under a fixed bound: `asyncio.to_thread(<obj>.wait, N)`
(asyncio however imported or aliased) and
`loop.run_in_executor(<any>, <obj>.wait, N)`, across test/ and every
src/**/tests package. 184 sites in 97 files. It shares the existing
gate's baseline machinery, so two burn-downs in different files merge
without conflict and moving a site between files fails.
jeeshofone added a commit to jeeshofone/KiroCrew that referenced this pull request Oct 15, 2026
…o_thread waits

kirodotdev#20346 moved the waiting side of the spec-builder nudge-pause and
auto-research settlement tests off `asyncio.to_thread(<event>.wait, N)`.
The held side of the same tests still bet on the runner's speed: a
patched write blocked on `assert <release>.wait(5)` while the test drove
executor-backed I/O. In
test_slow_settlement_cannot_remove_resumed_replacement and
test_duplicate_reserves_its_name_before_populating_the_destination that
I/O is a store write or a full create, so a slow runner made the held
thread give up first and the test failed with a bare assert or a 500.
With a one-thread default executor both fail every time at those lines.

Every held-side fixed bound in test_auto_research.py and
spec_builder/tests/test_routes.py now waits on default_timeout(), the
lost-run bound, and a hold the test never releases fails by name at that
bound, including one raised inside a request handler or a plain thread.

The one-thread executor pin moves into kiro_crew.testing as the pytest
fixture one_pool_thread (kiro_crew.testing.fixtures); both test files
import it, so the duplicated fixtures are deleted.

test/test_wait_discipline_ratchet.py gains a second per-file ratchet,
test/to_thread_wait_baseline.txt, for a threading.Event wait parked on an
executor under a fixed bound: `asyncio.to_thread(<obj>.wait, N)`
(asyncio however imported or aliased) and
`loop.run_in_executor(<any>, <obj>.wait, N)`, across test/ and every
src/**/tests package. 184 sites in 97 files. It shares the existing
gate's baseline machinery, so two burn-downs in different files merge
without conflict and moving a site between files fails.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: nudge-pause and watchdog settlement tests on Windows (7)

3 participants