Repository navigation
test: set or wait for state in five Windows/macOS-flaky test files - #17828
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Check that the markdown guards' ratio floor, which rises from 0.05 s to the shared helper's 0.25 s, still catches the regression class those guards were written for. [DESIGN-REVIEWED] 6d98123 Description read[DESCRIPTION-READ] e916553954beed082d875fa0d3922fe71d76fe49671939105bf3b0c404b0dc34 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
|
Design Review note on
No code change proposed; a maintainer who wants the 0.05 s constant kept can ask for one absolute assertion at the shipped size beside the helper call. |
|
The fixes go after real causes rather than adding retries. Seeding off the event loop fixes the WinError 5 setup failure, a pinned monotonic clock fixes the diag rate test, the skills catalog now waits for |
3e1eaa6 to
d76f46e
Compare
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ⏭️ skippedRevision |
Five test families went red on loaded Windows and macOS runners for reasons unrelated to the change under test, and a rerun went green. Each bet on runner speed: - test_telegram_sessions: _log() seeded its transcript on the event loop, where replace_with_retry re-raises a Windows sharing violation instead of retrying (WinError 5). It now seeds through off_loop. - test_diag_procs: main already pins both scans of the too-close rate test and ages the window to 0.5 s by hand; the decline note is now pinned to that hand-aged gap as well. - test_skills: a fresh or just-mutated skills root serves [] marked "building" once its 2 s cold budget runs out. Every loader, and every read after a mutation, goes through _settled(), which waits for catalog_status() == "complete" under a named lost-run bound. - test_denied_commands_security: the stubbed DNS hold and the publish poll were both 5 s. The hold now outlasts the wait_until poll, and the resolver join asserts the thread exited. - test_markdown_link_parentheses: the linear-time guards read one wall-clock sample. They use assert_rejected_without_backtracking, which gains a keyword-only budget_seconds (default unchanged), so each guard keeps its original ceiling on thread CPU: 0.5 s for the two opener guards, 1.0 s for the reading guard. Part of #16536 Fixes #17782 Fixes #17447 Fixes #15503
d76f46e to
6d98123
Compare
|
|
E2E (stub ACP backend, offline) on |
|
Design Review note on
No code change. |
iamwhatever
left a comment
There was a problem hiding this comment.
This test-only PR removes five runner-speed bets that turned Windows and macOS shards red. Each test now sets the state it asserts, or waits for it under a named lost-run bound, so the root cause is fixed rather than papered over with retries. I read the whole diff. Here is what each fix does. The Telegram _log() now seeds its log off the event loop, so replace_with_retry's bounded retry applies. The skills tests settle catalog_status() to "complete" via wait_until, so a slow walk can no longer turn an empty catalog into a false pass on the negative test, and the new autouse close_skills_loaders fixture releases the workers. The DNS hold (60 s, released in finally) now outlasts the 30 s publish poll, and _join_resolver joins only the resolver this test started, then asserts it exited. The markdown guards move to assert_rejected_without_backtracking with their original absolute ceilings (0.5 s / 1.0 s) kept through the new keyword-only budget_seconds, whose default leaves every other caller unchanged. On the Design lane's question about the ratio floor: it rises from 0.05 s to 0.25 s, but the shared helper now ramps over 10x steps on thread CPU with a 25x per-step bound, which still catches a quadratic regression (100x per step), and the absolute ceiling is unchanged. The GPT ceiling finding was fixed in 6d98123. No production code or policy changes, so touching test_denied_commands_security.py is not a security-policy change. All checks are green, and Opus and GPT report no findings. — crew-pr-reviewer
Problem / Motivation
Goal: Five test families that went red on Windows and macOS runners for reasons unrelated to the PR under test now give the same verdict on every runner, because each test sets or waits for the state it asserts instead of betting on runner speed.
These failed on PRs that touched none of their code, and a rerun went green:
test_telegram_sessions.py::TestTelegramSessionPicker::test_a_press_serializes_against_the_next_messagePermissionError: [WinError 5] Access is denied: '...\sessions\tmp….tmp' -> '...\sessions\dashboard_chat-1.jsonl'in setup (Windows shard 3, #17782)test_diag_procs.py::test_two_reads_too_close_together_decline_the_rateassert 410.17 is None(Windows shard 4, #17447)test_skills.py::TestDisabledAppSkillsAreNotTriggered::test_only_the_disabled_apps_skill_is_withheldassert [] == ['ops_mc/ops-mission-control'](Windows shard 8, #15503)test_denied_commands_security.py::TestSandboxEscapeSshSelf::test_slow_dns_does_not_hold_the_netlink_publishtest_markdown_link_parentheses.pylinear-time guardsWhy it matters
Each red lands on whichever PR shares the shard and costs a rerun, which spends shared runner time and trains people to ignore red. Some of these tests also passed for the wrong reason when they lost the race: an empty skills catalog makes "the disabled app's skill is withheld" true, so on a slow runner they could not catch the regression they exist for.
Not a goal
kiro_crew.testing.wait.wait_until,test/off_loop_helpers.off_loopandconftest.assert_rejected_without_backtracking, which already exist.test_denied_commands_security.py(theelapsed <sites outsideTestSandboxEscapeSshSelf) are left for the planned complexity-guard batch.What changed (motivation → approach → change)
Symptom → cause. Each test bet on runner speed in one of five ways. The Telegram fixture wrote its seed transcript on the event-loop thread, where
replace_with_retryre-raises a Windows sharing violation on the first attempt instead of sleeping the loop, so a scanner holding the fresh temp file for an instant failed the setup. The diag test measured the gap between two real-clock scans, so runner load decided whether it crossed the 1 s rate floor. A fresh skills root, or one just mutated, waits at most_COLD_CATALOG_WAIT_SECS(2 s) for its background walk and then serves[]marked "building", which a slow walk turned into an empty catalog under test. The DNS test's stubbed lookup held for 5 s and its publish poll gave up after 5 s, so a slow publish let DNS return before the "still in flight" check. The markdown guards compared oneperf_counterreading to an absolute budget, which charges the time a worker spends descheduled.Approach. Set the state the assertion reads, or wait for it under a named lost-run bound that fails by name, and leave every assertion as strict or stricter.
test_telegram_sessions.py_log()seeds throughoff_loop, so the bounded retry applies; this covers the 92 tests that build a log through ittest_diag_procs.pySCAN_MONOTONICand ages the window to 0.5 s by hand (#17507); this change adds that the decline note must quotereads 0.5s apart, so the declined gap is the one the test settest_skills.py_settled(loader)pollscatalog_status()to"complete"underwait_until(_CATALOG_SETTLE_SECS = 30); every loader whose test enumerates goes through it, and so does every read that follows a mutation. A module autouse fixture requestsclose_skills_loadersso the settle's catalog worker and SQLite index are releasedtest_denied_commands_security.py_DNS_HOLD_SECS = 60, released infinally) outlasts thewait_untilpublish poll (30 s);_join_resolverjoins the resolver this test started — identified against the set_open_windowrecords — and asserts it exited within 30 stest_markdown_link_parentheses.pyassert_rejected_without_backtracking(thread CPU, ramp, then 10x pumps with a ratio bound), each keeping its original absolute ceiling:budget_seconds=0.5for the two opener guards,1.0for the reading guardtest/conftest.pyassert_rejected_without_backtrackinggains one keyword-onlybudget_seconds(defaultREDOS_LARGE_BUDGET_SECONDS, unchanged), used where it applies the long pumps' absolute budget; every other caller behaves exactly as beforeBackwards compatibility
Compatible: the diff changes test files only; no production module, fixture, configuration or pinned list changes.
Removes nothing: every assertion is kept or made stricter (
test_iter_result_is_cachedtightenscalls["n"] <= 1to== 1). The fivetest_each_reading_stays_linearparameter ids change from the raw 5,000-character payloads to names (openers,brackets, ...); nothing references them.The absolute ceilings stay at their original constants, now read on thread CPU instead of wall time: 0.5 s at the 20,000-unit pump for the two opener guards, 1.0 s for the reading guard. One bound is wider and the measurement says why: the old ratio check was
large <= max(8 * small@5k, 0.05), and the shared helper's ismax(25 * cost@2k, 0.25). With the shipped cost at 0.0017 s of thread CPU the binding bound was that 0.05 s floor. A floor that low on a 15.6 ms Windows clock tick is the shape class 5 says reads a linear cost as unbounded, and the helper's floor is shared, not per call. The ratio exponent itself tightens (25x per 10x = 1.40 against 8x per 4x = 1.50), and the helper adds a 1-to-24 ramp the old shape did not have, which is the only thing that catches the exponential class.Tests
All proofs ran on Linux CPython 3.12.13; host 1-minute load 4-6. The probes are out-of-tree pytest plugins and are not committed. The commit is rebased onto
db083f5d3. On that base #17507 had already applied the same pin to the #17447 test, so its forced condition (+2 s per clock read) no longer reds the new parent; it reds the original parentcd0298429, and this head adds only the note assertion there. After the rebase:test_diag_procs.py20x-n0(54/54 each) and 10x-n 4besidetest_diag_threads.pyandtest_skills.py(308/308 each), the 646 touched tests once, and the flake-pattern ratchet all pass. With the ceilings restored:test_markdown_link_parentheses.py20x-n0(1,418/1,418 each) and 10x-n 4beside two other ramp-helper files (1,503/1,503 each), every one of the 12 test files that callassert_rejected_without_backtrackingonce at the default budget (2,500 passed), the sleep probe still red oncd0298429(30 of 31) and green on this head, and semgrep 0 new onconftest.pyand the markdown file.Forced condition: parent red, head green (same command on both,
-n0).cd0298429atomic_writesees Windows and the first replace onto a.jsonlraisesPermissionErroroncePermissionErrorfrom_login setup; file-wide 92 of 102 redassert 125.0 is None(maindb083f5d3, which carries #17507's pin, passes)_COLD_CATALOG_WAIT_SECS = 0.0netlink never published(and siblingtest_startup_warm_publishes_before_the_first_ip_literal, behind the silent 5 s join)assert False is Truepattern.sub, 1.05 s inside each reading of 2,000+ charactersNegative controls on the head.
assert ['deploy_web/artifact-deploy'] == []test_a_disabled_apps_skill_does_not_trigger,test_a_disabled_builtins_package_tree_skill_is_gated)MIN_RATE_GAP_SECS = 0assert 1000.0 is Nonehandling a 20000-unit pump cost 2.42s of CPU -- superlinear in the pump lengthhandling a 20000-unit pump cost 0.60s of CPU; the 4 green are reading shapes under 1.0 sd76f46e94, which used the shared 2.0 s budgetResource footprint. Settling a loader is what starts its
skill-catalog-refreshworker and opens its SQLite index, so the first revision of this change settled 53 loaders in tests that never enumerate. Those settles are gone, and the remaining ones are released by the fixture. Measured overtest_skills.pyat-n0, workers parked and descriptors open at the end of the file: parent 102 and 315, first revision 113 and 348, this head 0 and 10.Repeats.
-n0, every touched test (646 ids)-n 4 --dist loadgroup --max-worker-restart=2, the 5 files plustest_skill_catalog_snapshot.pyandtest_explain_for_skill.py(3,235 tests)KC_FORCE=coarse_clockandslow_thread(0.2 s), every touched test,-n0KC_FORCE=gil_hog(switch interval 5 ms), the 46 card-item testslocal-gate.py --dry-runselects 10 backend files and 325 frontend cross-surface files. The backend set ran at-n 4withtest_flake_pattern_ratchet.py(4,438 passed, 1 skipped); the frontend set was not run because the diff touches nowebsite/**path. Also run:mypy src/kiro_crew(clean), flake8, isort, the black gate, the flake-pattern ratchet (no file's count rises;test_markdown_link_parentheses.pydrops its one duration bound to 0), docs-lint and the diff-scoped comment-history, brand, harness-parity and subprocess-encoding gates (all pass). Semgrep 1.78 with CI's rule set finds the same 7 pre-existing results in the base and head versions of the five files, so 0 new.Manual verification
N/A: the tests are the subject. The exact-SHA Linux (3.12, 1-8), Windows (1-8) and macOS lanes are the cross-platform evidence and are recorded on this PR.
Related Issues
Part of #16536
Fixes #17782
Fixes #17447
Fixes #15503
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)