Skip to content

test: ratchet fixed short waits; add await_until and LoopHop helpers - #19248

Merged
iamwhatever merged 1 commit into
mainfrom
unflaky/wait-discipline-ratchet
Oct 12, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
unflaky/wait-discipline-ratchet

Conversation

@CrysisDeu

@CrysisDeu CrysisDeu commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A test that waits a fixed short time for something the code signals can no longer be added to test/; the ones already there are counted, file by file, and that count only goes down.

Of the last 18 flaky backend tests fixed here, 7 had one cause: the test bet on a short clock -- a 5 s asyncio.wait_for (#19093), a .result(timeout=5) (#19144), a poll of 200 five-millisecond turns (#19177), a 50 ms module constant (#19205) -- and a CI runner was slower than the dev box. test/ holds 2,912 more such bets in 528 files. Nothing stops the next PR adding one.

Why it matters

Every one of these reds a pull request that did not touch it, and each fix costs a day of someone's time to find the race, size a ceiling and prove it. Stopping the number growing is cheaper than fixing them one at a time after they fire.

Not a goal

  • Lowering the 2,912 existing sites (eleven are converted here, to show the helpers in use). That migration is a separate tracked effort; this PR only freezes the count.
  • Replacing test/test_flake_pattern_ratchet.py or semgrep/test-determinism.yaml. They judge a diff; this judges the tree. The # flake-ok: <reason> marker is shared so no site needs two.
  • Changing any product code. kiro_crew.testing.wait is the test helper module that ships in the wheel; nothing outside tests reads it.
  • Scanning the in-package app suites under src/kiro_crew/apps/builtins/*/tests.

What changed (motivation → approach → change)

Part What it does
test/test_wait_discipline_ratchet.py AST scan of test/**/*.py. Counts asyncio.wait_for, asyncio.timeout and .result(timeout=) under 30 s, and time.sleep / asyncio.sleep under 5 s inside a loop. A literal, or a module-level constant bound to one. sleep(0), a sleep outside a loop, a site under pytest.raises(TimeoutError) and a # flake-ok: <reason> line do not count.
test/wait_discipline_baseline.txt One <count> <file> line per file that has any, measured on main at d6341a8b99: 2,912 sites in 528 files, minus the 11 this PR converts. A count may only go down; a file not listed starts at zero. --write-baseline lowers lines and drops gone files, never raises or adds one; --move OLD NEW carries a renamed file's line to its new path, count unchanged, and the failure messages say so.
await_until in src/kiro_crew/testing/wait.py The replacement for asyncio.wait_for(signal, <short number>): an asyncio.Event or an asyncio future (a task, a gather). Default bound is half the test's timeout; expiry raises an AssertionError naming what and the elapsed time. The test's own future is never cancelled. A state with neither behind it stays with async_wait_until.
LoopHop in src/kiro_crew/testing/wait.py _LoopHop from test/test_slot_ownership_checkpoint.py, moved here with the same behaviour and the same 60 s bound (half the test timeout); that test imports it.
First consumers test_subagent_manager_boundaries.py and test_workflows_async_persistence.py await their Events through await_until instead of a 2 s / 4 s wait_for; test_members_dm_thread.py hops its winner's publish through LoopHop instead of .result(timeout=10). Their baseline lines drop accordingly.
D1 in testing-conventions.md One sentence: wait on signals, not clocks; a ceiling of 30 s or more only names a hang; the helpers and the ratchet.

Why a committed per-file baseline when test_flake_pattern_ratchet.py already counts short waits: that gate compares against the merge-base, which CI's depth-1 checkout cannot read, so it is a local pre-push gate, and its "short" is under 2 s. A baseline file is readable on any checkout, and a rise names the exact file and line.

Why one floor at 30 s: D1 says a ceiling of 30 s or more only names a hang. timeout=5 is the commonest value in the tree (1,013 wait_for sites) and the one that lost on CI, so a floor that let 5 through would make raising 2 to 5 the cheapest way past a red gate; the off-loop hop onto a loop doing durable work (a SQLite commit, an fsync) exceeds 5 s on the Windows runner. The helpers default to 60 s, half the suite's --timeout=120.

Why this gate beside K9 in test_flake_pattern_ratchet.py: K9 counts the same call shapes under 2 s, against a merge-base, and so cannot run on CI's depth-1 checkout. Trimming K9 is that gate's own change, not this one's; the # flake-ok marker is shared so a site never needs two.

Backwards compatibility

Compatible: two names are added to kiro_crew.testing.wait.__all__; nothing it exported changes. test_slot_ownership_checkpoint.py keeps the same hop behaviour through the shared class.

Removes nothing: the _LoopHop class and its 60 s constant move from one test file into the shared helper module; the test that used them imports the shared class.

Tests

  • test_wait_discipline_ratchet.py: the gate (test_no_file_gains_a_short_fixed_wait), staleness (test_the_baseline_is_not_stale), canonical form of the baseline, 14 positive detector cases and 15 near misses (floor values, non-literals, pytest.raises(TimeoutError), sleep(0), a sleep in a nested def, a name in a docstring, the helper with a short bound), a planted-file end-to-end on a scratch tree, the baseline reader refusing a malformed line, and lowered_baseline never raising or adding.
  • test_testing_determinism_helpers.py: TestAwaitUntil (event set / already set / never set; future result / exception / left pending at the bound; a gather; TypeError for a callable or a bare coroutine; the default bound is half the test timeout) and TestLoopHop (runs on the loop from a thread, the coroutine's own exception, an expired hop recorded for check(), the default ceiling).
  • test_slot_ownership_checkpoint.py and test_members_dm_thread.py: unchanged assertions with the shared LoopHop. test_subagent_manager_boundaries.py and test_workflows_async_persistence.py: unchanged assertions through await_until.
  • The LoopHop expiry test forces expiry only after the coroutine has entered, through a stand-in future whose result() waits for the entry mark and then raises TimeoutError; a real short bound would race the loop's scheduling. The two await_until negatives that take real time use a 0.05 s bound on a signal that never fires, so nothing can arrive late.
  • Proof the gate bites: a planted test/test_zz_planted_probe.py with one wait_for(..., timeout=1) fails test_no_file_gains_a_short_fixed_wait as test/test_zz_planted_probe.py: 0 -> 1 / :5 wait_for(1s) plus the helper hint; removing it is green again.

Manual verification

python test/test_wait_discipline_ratchet.py prints the burn-down (on this branch 2,901 in 526 files: 2,230 wait_for, 598 sleep-in-loop, 67 result, 6 asyncio.timeout); the scan takes about 19 s on a dev box. The new helper tests pass 10 of 10 repeats at -n0 and beside their neighbours at -n 4. Black, isort, flake8, mypy, comment-history, docs-lint and the rest of the prepare-pr static floor pass locally.

Related Issues

Fixes #19246

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@CrysisDeu
CrysisDeu requested review from a team and KiroCrewCaptain as code owners October 12, 2026 03:19
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 12, 2026
@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision f739119b6d3d4fa053d665d88240efeab3edeb25 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of f739119b6d3d4fa053d665d88240efeab3edeb25 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check the 19 s whole-tree AST scan on the slowest CI runner (Windows): it must stay well under the 120 s per-test timeout.

[DESIGN-REVIEWED] f739119

Description read

[DESCRIPTION-READ] a4c489f2500e54aceeb391a6139fbab1071e703bbe5a8577a17bf205638f7e75

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — 🟡 CONCERNS

Premise-level review of f739119b6d3d4fa053d665d88240efeab3edeb25 — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

First-Principles-Verdict: CONCERNS

For any wait under 2 s, the new ratchet counts the same wait_for and .result(timeout=) calls that K9 already counts, so each such site now has two counters.

Not justified as shipped

  • Item 1 — duplicate of test/test_flake_pattern_ratchet.py::_k9_short_literal_wait for the band under 2 s. The description says "K9 counts the same call shapes under 2 s… Trimming K9 is that gate's own change, not this one's". _k9_short_literal_wait (line 408) still matches wait_for and .result under 2 s. The new _short_waits matches the same calls under 30 s. That makes 1 overlapping detector, and the two have different baselines that will drift apart.
    Subtraction: in _k9_short_literal_wait, delete the wait_for branch and drop result from the attribute tuple, keeping only .wait/.join. Each short site then has one counter.
    Clears when: K9 is narrowed that way, or a named site is shown that only the merge-base check under 2 s catches.
  • Item 4 — oversized: --move OLD NEW is a CLI path for something a person can do by hand, since a rename is a one-line edit to wait_discipline_baseline.txt. Its only consumers are its own hints and the D1 sentence.
    Subtraction: delete the --move branch in __main__, moved_baseline and MOVE_HINT, and have the failure message say "edit the baseline line".
    Clears when: the flag is removed.

What this change ships

Inventory (8 items) — 6 justified

Intent: stop new fixed short waits from entering test/ and make sure the existing count only goes down — ADDITION.

  1. CI fails a test file that gains a short fixed wait or short poll sleep — duplicate of K9 for waits under 2 s
  2. Committed per-file baseline of 2,901 sites — justified
  3. A staleness test makes a file's baseline line drop when its count drops; --write-baseline writes the lower counts — justified
  4. --move OLD NEW rename command — oversized
  5. await_until helper in the shipped testing module — justified
  6. LoopHop moved from one test file into the shared module — justified
  7. Eleven waits in three test files converted to the helpers — justified
  8. D1 sentence in testing-conventions plus the module docstring — justified

[FIRST-PRINCIPLES-REVIEWED] f739119

Description read

[DESCRIPTION-READ] 7f40070ab84d2fdb5556974e4da8badd882a00f49b3c9240ea7285ebe5c29099

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed f739119b6d3d4fa053d665d88240efeab3edeb25 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] f739119

Verdict parsed from the review's SHA-scoped output markers for commit f739119b6d3d4fa053d665d88240efeab3edeb25.

False positive or not applicable? A repository writer can comment:
/ai-review override fable f739119b6d3d4fa053d665d88240efeab3edeb25: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 12, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of f739119b6d3d4fa053d665d88240efeab3edeb25 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] f739119

False positive or not applicable? A repository writer can comment:
/ai-review override gpt f739119b6d3d4fa053d665d88240efeab3edeb25: <one-sentence reason>

@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-discipline-ratchet branch 2 times, most recently from 57d5728 to c882292 Compare October 12, 2026 04:17
@CrysisDeu
CrysisDeu force-pushed the unflaky/wait-discipline-ratchet branch from c882292 to f739119 Compare October 12, 2026 04:29
@CrysisDeu

Copy link
Copy Markdown
Contributor Author

Disposition of the advisory findings on c882292fec, now at f739119b6d:

  • Design — a moved test file looks like a regression. Fixed. A renamed file keeps its line under the new path, count unchanged: python test/test_wait_discipline_ratchet.py --move OLD NEW; a split file splits its line so the parts sum to the old count. Both failure messages, the baseline header and the D1 sentence say so. Neither raises a count, so neither is what the ratchet forbids, and a reviewer reads a move as a changed path with the same number.
  • First Principles — K9 in test_flake_pattern_ratchet.py also counts wait_for/.result. Declined here. K9 counts those two shapes under 2 s plus .wait( and .join(, against the merge-base with move-forgiveness, and cannot run on CI's depth-1 checkout. Trimming it is that gate's own change (its detectors carry positive/negative/mutation pins and a semgrep mapping); the shared # flake-ok marker means a site never needs two. Noted in the description.
  • First Principles — await_until too wide, LoopHop one consumer. Taken on the previous push: Event or asyncio future only; test_members_dm_thread.py is the second LoopHop user.
  • Design — timeout=5 passed the floor. Taken on the previous push: one 30 s floor for wait_for / asyncio.timeout / .result.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 12, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR adds a per-file ratchet (test/test_wait_discipline_ratchet.py plus a committed baseline) that stops new test files gaining fixed short waits: wait_for, asyncio.timeout and .result under 30 s, and sleeps under 5 s inside loops. It also adds await_until and LoopHop to kiro_crew.testing.wait and converts eleven sites to use them. That attacks the root cause the description names (tests betting on a short clock), which caused 7 of the last 18 flaky-test fixes, and does it at the tree level CI can read on a depth-1 checkout. I read wait.py in full. await_until never cancels the caller's future, cancels only its own Event waiter, and raises a named AssertionError. LoopHop's except TimeoutError is correct because requires-python is >=3.12, where concurrent.futures.TimeoutError is the builtin. I also read the AST detector: it handles import aliases and module constants, exempts pytest.raises(TimeoutError), sleep(0) and flake-ok. A non-literal timeout gets past it, which is acceptable for a ratchet and stated as scope. All checks are green, GPT/Opus/Design pass, and the First Principles K9-overlap item has an answer on the PR. The --move subtraction is advisory and can stay, since it is what answers Design's rename concern. — crew-pr-reviewer (deep re-check)

@iamwhatever
iamwhatever merged commit 5a87891 into main Oct 12, 2026
101 checks passed
@iamwhatever
iamwhatever deleted the unflaky/wait-discipline-ratchet branch October 12, 2026 06:47
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 12, 2026
CrysisDeu added a commit that referenced this pull request Oct 12, 2026
…ng, compaction_stop, taskq_admission

Four backend test files bet on fixed short waits for things the code
signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate
every counted site to a wait on the real signal, using the
kiro_crew.testing.wait helpers from #19248.

Site counts (test/wait_discipline_baseline.txt): all four lines removed.
  test_queue_depth_reconcile.py       43 -> 0
  test_codex_steering_request.py      42 -> 0
  test_compaction_stop_declined.py    35 -> 0
  test_taskq_admission_integration.py 20 -> 0

No test's intent changed; no assertion was weakened; no site moved to
another file. Part of the wait-discipline burn-down behind #19248.
CrysisDeu added a commit that referenced this pull request Oct 12, 2026
…ances, test_lifecycle_hooks

Migrate every short fixed wait in the three session-area test files to the
signal the code under test emits, using the kiro_crew.testing.wait helpers
from #19248 (await_until, async_wait_until). Lowers the three
wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22).

Every test's intent is preserved; no timeout was raised to 30, no assertion
weakened, no site moved or skipped.

Closes #19526
Part of #19246
CrysisDeu added a commit that referenced this pull request Oct 12, 2026
…ng, compaction_stop, taskq_admission

Four backend test files bet on fixed short waits for things the code
signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate
every counted site to a wait on the real signal, using the
kiro_crew.testing.wait helpers from #19248.

Site counts (test/wait_discipline_baseline.txt): all four lines removed.
  test_queue_depth_reconcile.py       43 -> 0
  test_codex_steering_request.py      42 -> 0
  test_compaction_stop_declined.py    35 -> 0
  test_taskq_admission_integration.py 20 -> 0

No test's intent changed; no assertion was weakened; no site moved to
another file. Part of the wait-discipline burn-down behind #19248.
CrysisDeu added a commit that referenced this pull request Oct 12, 2026
…ances, test_lifecycle_hooks

Migrate every short fixed wait in the three session-area test files to the
signal the code under test emits, using the kiro_crew.testing.wait helpers
from #19248 (await_until, async_wait_until). Lowers the three
wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22).

Every test's intent is preserved; no timeout was raised to 30, no assertion
weakened, no site moved or skipped.

Closes #19526
Part of #19246
CrysisDeu added a commit that referenced this pull request Oct 13, 2026
…ng, compaction_stop, taskq_admission

Four backend test files bet on fixed short waits for things the code
signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate
every counted site to a wait on the real signal, using the
kiro_crew.testing.wait helpers from #19248.

Site counts (test/wait_discipline_baseline.txt): all four lines removed.
  test_queue_depth_reconcile.py       43 -> 0
  test_codex_steering_request.py      42 -> 0
  test_compaction_stop_declined.py    35 -> 0
  test_taskq_admission_integration.py 20 -> 0

No test's intent changed; no assertion was weakened; no site moved to
another file. Part of the wait-discipline burn-down behind #19248.
CrysisDeu added a commit that referenced this pull request Oct 14, 2026
…ances, test_lifecycle_hooks

Migrate every short fixed wait in the three session-area test files to the
signal the code under test emits, using the kiro_crew.testing.wait helpers
from #19248 (await_until, async_wait_until). Lowers the three
wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22).

Every test's intent is preserved; no timeout was raised to 30, no assertion
weakened, no site moved or skipped.

Closes #19526
Part of #19246
KiroCrewCaptain pushed a commit that referenced this pull request Oct 14, 2026
…ances, test_lifecycle_hooks (#19527)

Migrate every short fixed wait in the three session-area test files to the
signal the code under test emits, using the kiro_crew.testing.wait helpers
from #19248 (await_until, async_wait_until). Lowers the three
wait_discipline_baseline.txt lines to zero (85 sites: 39 + 24 + 22).

Every test's intent is preserved; no timeout was raised to 30, no assertion
weakened, no site moved or skipped.

Closes #19526
Part of #19246
CrysisDeu added a commit that referenced this pull request Oct 14, 2026
…ng, compaction_stop, taskq_admission

Four backend test files bet on fixed short waits for things the code
signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate
every counted site to a wait on the real signal, using the
kiro_crew.testing.wait helpers from #19248.

Site counts (test/wait_discipline_baseline.txt): all four lines removed.
  test_queue_depth_reconcile.py       43 -> 0
  test_codex_steering_request.py      42 -> 0
  test_compaction_stop_declined.py    35 -> 0
  test_taskq_admission_integration.py 20 -> 0

No test's intent changed; no assertion was weakened; no site moved to
another file. Part of the wait-discipline burn-down behind #19248.
iamwhatever pushed a commit that referenced this pull request Oct 14, 2026
…ng, compaction_stop, taskq_admission (#19496)

Four backend test files bet on fixed short waits for things the code
signals. A slow CI shard loses the bet and reds an unrelated PR. Migrate
every counted site to a wait on the real signal, using the
kiro_crew.testing.wait helpers from #19248.

Site counts (test/wait_discipline_baseline.txt): all four lines removed.
  test_queue_depth_reconcile.py       43 -> 0
  test_codex_steering_request.py      42 -> 0
  test_compaction_stop_declined.py    35 -> 0
  test_taskq_admission_integration.py 20 -> 0

No test's intent changed; no assertion was weakened; no site moved to
another file. Part of the wait-discipline burn-down behind #19248.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Ratchet fixed short waits in tests; add await_until/LoopHop helpers

2 participants