Repository navigation
test(auto_improvement): wait on the lost-run bound in test_pr_watchers - #39545
Conversation
UX Review (Opus 5.5) — ⏭️ skippedRevision |
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks: the change fixes the cause (one shared 10 s cap in the poll helpers) in the helpers themselves, so no test needs its own bump. A stuck watcher still fails by name under the lost-run bound. Notes
[DESIGN-REVIEWED] 414b259 Description read[DESCRIPTION-READ] 59033338f357211b9c23271e921a1774d7174892c7c930f60a10274fe5fc5af9 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
This PR and #39544 both close #39485 with the same change to |
…s poll helpers
The file's _await_status/_await_gone helpers and five is_alive loops bounded
every wait at a file-wide WAIT_S = 10 s. A pass is a real git clone plus several
git subprocesses; on Windows backend shards with six xdist workers that ran past
10 s while the watcher was still mid-pass, so
test_the_pass_fix_is_exported_to_the_pr_queue and
test_the_head_branch_comes_from_the_live_status failed with
'watcher never reached {exhausted}; last status nudging, pass 1/1'.
Route every wait through kiro_crew.testing.wait.wait_until with its default
lost-run bound (half the test timeout) and drop the per-test WAIT_S * 3
overrides. testing-conventions records why the per-test multiplier did not fix
the file.
4a4b1b3 to
414b259
Compare
The Windows backend shards are CPU-bound, not memory-bound: at 4 xdist workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and peaked at 8,603-8,901 of 16,111 MiB. Grant one worker per 2 GiB after 2 GiB for the host instead of one per 3 GiB, still capped at the core count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. At 6 workers a shard peaked at 9,858-11,234 MiB (runs 44434126975 and 44440752272). The first attempt (#39390) was reverted (#39516) because tests that depend on wall-clock time failed at 6 workers on Windows shards 5, 7 and 8. #39505, #39545 and #39618 removed those dependencies from the tests. This change is measured over several full Windows runs before review: one run is not evidence that load-sensitive tests stay green. Related: #39365 Co-authored-by: Akim Akimov <zedmor@amazon.com>
Problem / Motivation
Goal:
test_pr_watchers.pystops failing Windows backend shard 5 on pull requests that do not touch it.Since the Windows backend shards moved to six xdist workers (#39390),
TestCloneLifecycleAndExport::test_the_pass_fix_is_exported_to_the_pr_queueandtest_the_head_branch_comes_from_the_live_statusfail withwatcher never reached {'exhausted'}; last={... 'status': 'nudging', 'lastNote': 'pass 1/1' ...}. The watcher is still mid-pass, not stuck.Why it matters
In seven days the export test turned Windows shard 5 red on seven pull requests targeting main (#36705, #36757, #38370, #39461, #39504, #39506, #39507) and once on main (60a1064). Linux is green. Every author has to rerun a shard they did not break.
Not a goal
pr_watchers.py) or its production timeouts.test_ai_pr_watchers_coverage.py, which fakes git (see Sibling sweep).What changed (motivation → approach → change)
Every wait in
test_pr_watchers.pyused a file-wide 10-second ceiling (WAIT_S). One watcher pass is a realgit cloneplus several git subprocesses. On a busy Windows runner that takes longer than 10 seconds, so the wait gave up while the pass was still running. Two tests already carried aWAIT_S * 3patch for the same reason.The file's wait helpers now call
wait_untilfromkiro_crew.testing.waitwith no timeout. That gives every wait the lost-run bound (half the test timeout, rule D1), so a slow pass waits and a truly stuck watcher still fails by name. The twoWAIT_S * 3patches and the five hand-written "is the thread dead yet" loops use the same helpers.testing-conventions.mdrecords why a per-test multiplier did not fix the file.Backwards compatibility
Compatible: test-only change; no product code, config or public API is touched.
Removes nothing: only test helpers and a docs paragraph change.
Tests
Changed: the wait helpers
_await_status,_await_goneand the new_await_deadinsrc/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py. The assertions are unchanged; only how long a test may wait for the state it asserts.Proof: not provable -- test-only change
Manual verification
N/A -- the change is test-only; the real-machine repeats below are the verification.
Real-machine verification
Machine: EC2 c6i.2xlarge, Windows Server 2022 (ami-074ba7d38e80fcb2e), x86_64, 8 vCPU / 16 GiB, Python 3.12.10. Load: each round runs 3 concurrent
pytest -n 6sessions (18 workers) overTestCloneLifecycleAndExportandTestNudgeLoop, plus 6 busy-loop CPU burners, 25 rounds = 75 sessions per side.Before (main 906ee87):
python -m pytest -q -n 6 -p no:cacheprovider --timeout=180 --no-cov <file>::TestCloneLifecycleAndExport <file>::TestNudgeLoop-> 57 of 75 sessions red:test_the_pass_fix_is_exported_to_the_pr_queuefailed 56 times andtest_the_head_branch_comes_from_the_live_status17 times, bothwatcher never reached {'exhausted'}withstatus: nudging,pass 1/1(the CI signature).After (head 4a4b1b3): the same command and load -> 75 of 75 sessions green (16 passed each). 84 of the 150 recorded calls of those two tests took over 10 s (max 15.0 s), so each would have failed under the old ceiling. The current head 414b259 differs from 4a4b1b3 only in the helper's not-yet value (
{}instead ofNone, both falsy towait_until) to satisfy mypy.Cleanup: instance i-05fcbac05800bc330 shut itself down via
shutdown /s /t 60with--instance-initiated-shutdown-behavior terminate; no key pair or security group was created.Related Issues
Closes #39485
Sibling sweep
Swept: git grep -nE "^WAIT_S\s*=" -- src/kiro_crew/apps/builtins/auto_improvement/
Sibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py -- changed here
Sibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_ai_pr_watchers_coverage.py -- not affected -- it replaces
_git/_ghwith FakeGit, so a pass runs no real subprocess and stays far under 10 sSibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_ai_runner_coverage.py -- not affected -- its waits gate on stub events released by the test, with no git work behind them
Pattern harvest
Rule candidate: review-prompt
Pattern: a file-wide short wait ceiling (
WAIT_S = 10) shared by poll helpers that wait on real subprocess workChecklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits