Skip to content

test(auto_improvement): wait on the lost-run bound in test_pr_watchers - #39545

Merged
bolichen97 merged 1 commit into
mainfrom
crew/mate/deflake-pr-watchers-export
Nov 19, 2026
Merged

bolichen97 merged 1 commit into
mainfrom
crew/mate/deflake-pr-watchers-export

Conversation

@bolichen97

@bolichen97 bolichen97 commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: test_pr_watchers.py stops failing Windows backend shard 5 on pull requests that do not touch it.

Since the Windows backend shards moved to six xdist workers (#39390), TestCloneLifecycleAndExport::test_the_pass_fix_is_exported_to_the_pr_queue and test_the_head_branch_comes_from_the_live_status fail with watcher never reached {'exhausted'}; last={... 'status': 'nudging', 'lastNote': 'pass 1/1' ...}. The watcher is still mid-pass, not stuck.

Why it matters

In seven days the export test turned Windows shard 5 red on seven pull requests targeting main (#36705, #36757, #38370, #39461, #39504, #39506, #39507) and once on main (60a1064). Linux is green. Every author has to rerun a shard they did not break.

Not a goal

  • Changing the watcher (pr_watchers.py) or its production timeouts.
  • Changing the Windows xdist worker count (#39516 is a separate decision).
  • The sibling coverage file test_ai_pr_watchers_coverage.py, which fakes git (see Sibling sweep).

What changed (motivation → approach → change)

Every wait in test_pr_watchers.py used a file-wide 10-second ceiling (WAIT_S). One watcher pass is a real git clone plus several git subprocesses. On a busy Windows runner that takes longer than 10 seconds, so the wait gave up while the pass was still running. Two tests already carried a WAIT_S * 3 patch for the same reason.

The file's wait helpers now call wait_until from kiro_crew.testing.wait with no timeout. That gives every wait the lost-run bound (half the test timeout, rule D1), so a slow pass waits and a truly stuck watcher still fails by name. The two WAIT_S * 3 patches and the five hand-written "is the thread dead yet" loops use the same helpers. testing-conventions.md records why a per-test multiplier did not fix the file.

Backwards compatibility

Compatible: test-only change; no product code, config or public API is touched.

Removes nothing: only test helpers and a docs paragraph change.

Tests

Changed: the wait helpers _await_status, _await_gone and the new _await_dead in src/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py. The assertions are unchanged; only how long a test may wait for the state it asserts.

Proof: not provable -- test-only change

Manual verification

N/A -- the change is test-only; the real-machine repeats below are the verification.

Real-machine verification

Machine: EC2 c6i.2xlarge, Windows Server 2022 (ami-074ba7d38e80fcb2e), x86_64, 8 vCPU / 16 GiB, Python 3.12.10. Load: each round runs 3 concurrent pytest -n 6 sessions (18 workers) over TestCloneLifecycleAndExport and TestNudgeLoop, plus 6 busy-loop CPU burners, 25 rounds = 75 sessions per side.
Before (main 906ee87): python -m pytest -q -n 6 -p no:cacheprovider --timeout=180 --no-cov <file>::TestCloneLifecycleAndExport <file>::TestNudgeLoop -> 57 of 75 sessions red: test_the_pass_fix_is_exported_to_the_pr_queue failed 56 times and test_the_head_branch_comes_from_the_live_status 17 times, both watcher never reached {'exhausted'} with status: nudging, pass 1/1 (the CI signature).
After (head 4a4b1b3): the same command and load -> 75 of 75 sessions green (16 passed each). 84 of the 150 recorded calls of those two tests took over 10 s (max 15.0 s), so each would have failed under the old ceiling. The current head 414b259 differs from 4a4b1b3 only in the helper's not-yet value ({} instead of None, both falsy to wait_until) to satisfy mypy.
Cleanup: instance i-05fcbac05800bc330 shut itself down via shutdown /s /t 60 with --instance-initiated-shutdown-behavior terminate; no key pair or security group was created.

Related Issues

Closes #39485

Sibling sweep

Swept: git grep -nE "^WAIT_S\s*=" -- src/kiro_crew/apps/builtins/auto_improvement/
Sibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_pr_watchers.py -- changed here
Sibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_ai_pr_watchers_coverage.py -- not affected -- it replaces _git/_gh with FakeGit, so a pass runs no real subprocess and stays far under 10 s
Sibling: src/kiro_crew/apps/builtins/auto_improvement/tests/test_ai_runner_coverage.py -- not affected -- its waits gate on stub events released by the test, with no git work behind them

Pattern harvest

Rule candidate: review-prompt
Pattern: a file-wide short wait ceiling (WAIT_S = 10) shared by poll helpers that wait on real subprocess work

Checklist

  • A Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...); PR Hygiene does not count commits
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 19, 2026
@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 414b2591db11611b4f4c48d36d0ed14df7dffafb touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 414b2591db11611b4f4c48d36d0ed14df7dffafb ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Does not block merge.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 414b2591db11611b4f4c48d36d0ed14df7dffafb — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the change fixes the cause (one shared 10 s cap in the poll helpers) in the helpers themselves, so no test needs its own bump. A stuck watcher still fails by name under the lost-run bound.

Notes

  • apps/adoption-pins.json pins auto-improvement, so AGENTS.md wants this fix landed in KiroCrewCaptain/kirocrew-app-auto-improvement first, with the PR naming it as App repo: <owner>/<repo>@<sha>. Until it lands there, that repo's copy of this test still carries the 10 s cap.

[DESIGN-REVIEWED] 414b259

Description read

[DESCRIPTION-READ] 59033338f357211b9c23271e921a1774d7174892c7c930f60a10274fe5fc5af9

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 414b2591db11611b4f4c48d36d0ed14df7dffafb — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 414b259

Verdict parsed from the review's SHA-scoped output markers for commit 414b2591db11611b4f4c48d36d0ed14df7dffafb.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 414b2591db11611b4f4c48d36d0ed14df7dffafb false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 414b2591db11611b4f4c48d36d0ed14df7dffafb and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 414b259

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 414b2591db11611b4f4c48d36d0ed14df7dffafb false-positive: <reason, 20 to 500 characters>

@bolichen97
bolichen97 marked this pull request as ready for review November 19, 2026 18:05
@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 19, 2026 18:05
@bolichen97
bolichen97 requested a review from buluoray November 19, 2026 18:05
@bolichen97
bolichen97 enabled auto-merge (squash) November 19, 2026 18:05
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@KiroCrewCaptain

Copy link
Copy Markdown
Collaborator

This PR and #39544 both close #39485 with the same change to test_pr_watchers.py. Each drops WAIT_S = 10.0 and the WAIT_S * 3 overrides, and moves _await_status, _await_gone and a new _await_dead onto kiro_crew.testing.wait.wait_until. Both also edit testing-conventions.md, and #39544 also changes test_ai_pr_watchers_coverage.py. Please keep one of #39544 and #39545 and close the other, carrying over anything only this one has.

…s poll helpers

The file's _await_status/_await_gone helpers and five is_alive loops bounded
every wait at a file-wide WAIT_S = 10 s. A pass is a real git clone plus several
git subprocesses; on Windows backend shards with six xdist workers that ran past
10 s while the watcher was still mid-pass, so
test_the_pass_fix_is_exported_to_the_pr_queue and
test_the_head_branch_comes_from_the_live_status failed with
'watcher never reached {exhausted}; last status nudging, pass 1/1'.

Route every wait through kiro_crew.testing.wait.wait_until with its default
lost-run bound (half the test timeout) and drop the per-test WAIT_S * 3
overrides. testing-conventions records why the per-test multiplier did not fix
the file.
@bolichen97
bolichen97 force-pushed the crew/mate/deflake-pr-watchers-export branch from 4a4b1b3 to 414b259 Compare November 19, 2026 18:40
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 19, 2026
@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 19, 2026
@bolichen97
bolichen97 merged commit 96ed43d into main Nov 19, 2026
101 of 105 checks passed
@bolichen97
bolichen97 deleted the crew/mate/deflake-pr-watchers-export branch November 19, 2026 19:48
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Nov 19, 2026
Zedmor added a commit that referenced this pull request Nov 20, 2026
The Windows backend shards are CPU-bound, not memory-bound: at 4 xdist
workers a shard kept 3.6-3.7 of the fleet runner's 8 vCPU busy and
peaked at 8,603-8,901 of 16,111 MiB. Grant one worker per 2 GiB after
2 GiB for the host instead of one per 3 GiB, still capped at the core
count: 6 on the 16 GiB fleet runner, 4 on a 4-core hosted runner. At 6
workers a shard peaked at 9,858-11,234 MiB (runs 44434126975 and
44440752272).

The first attempt (#39390) was reverted (#39516) because tests that
depend on wall-clock time failed at 6 workers on Windows shards 5, 7
and 8. #39505, #39545 and #39618 removed those dependencies from the
tests.

This change is measured over several full Windows runs before review:
one run is not evidence that load-sensitive tests stay green.

Related: #39365

Co-authored-by: Akim Akimov <zedmor@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flaky: test_the_pass_fix_is_exported_to_the_pr_queue (watcher never reached exhausted, Windows)

2 participants