Skip to content

test: set or wait for state in five Windows/macOS-flaky test files - #17828

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
refactor/df-b3c-windows-macos-timing
Oct 8, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
refactor/df-b3c-windows-macos-timing

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: Five test families that went red on Windows and macOS runners for reasons unrelated to the PR under test now give the same verdict on every runner, because each test sets or waits for the state it asserts instead of betting on runner speed.

These failed on PRs that touched none of their code, and a rerun went green:

Test CI symptom
test_telegram_sessions.py::TestTelegramSessionPicker::test_a_press_serializes_against_the_next_message PermissionError: [WinError 5] Access is denied: '...\sessions\tmp….tmp' -> '...\sessions\dashboard_chat-1.jsonl' in setup (Windows shard 3, #17782)
test_diag_procs.py::test_two_reads_too_close_together_decline_the_rate assert 410.17 is None (Windows shard 4, #17447)
test_skills.py::TestDisabledAppSkillsAreNotTriggered::test_only_the_disabled_apps_skill_is_withheld assert [] == ['ops_mc/ops-mission-control'] (Windows shard 8, #15503)
test_denied_commands_security.py::TestSandboxEscapeSshSelf::test_slow_dns_does_not_hold_the_netlink_publish a 5 s stub hold racing a 5 s publish poll (de-flake catalog B6)
test_markdown_link_parentheses.py linear-time guards one wall-clock sample against an absolute budget (de-flake catalog B7)

Why it matters

Each red lands on whichever PR shares the shard and costs a rerun, which spends shared runner time and trains people to ignore red. Some of these tests also passed for the wrong reason when they lost the race: an empty skills catalog makes "the disabled app's skill is withheld" true, so on a slow runner they could not catch the regression they exist for.

Not a goal

  • No production change: every fix is in the five test files.
  • No new shared helper: the fixes use kiro_crew.testing.wait.wait_until, test/off_loop_helpers.off_loop and conftest.assert_rejected_without_backtracking, which already exist.
  • The other wall-clock bounds in test_denied_commands_security.py (the elapsed < sites outside TestSandboxEscapeSshSelf) are left for the planned complexity-guard batch.

What changed (motivation → approach → change)

Symptom → cause. Each test bet on runner speed in one of five ways. The Telegram fixture wrote its seed transcript on the event-loop thread, where replace_with_retry re-raises a Windows sharing violation on the first attempt instead of sleeping the loop, so a scanner holding the fresh temp file for an instant failed the setup. The diag test measured the gap between two real-clock scans, so runner load decided whether it crossed the 1 s rate floor. A fresh skills root, or one just mutated, waits at most _COLD_CATALOG_WAIT_SECS (2 s) for its background walk and then serves [] marked "building", which a slow walk turned into an empty catalog under test. The DNS test's stubbed lookup held for 5 s and its publish poll gave up after 5 s, so a slow publish let DNS return before the "still in flight" check. The markdown guards compared one perf_counter reading to an absolute budget, which charges the time a worker spends descheduled.

Approach. Set the state the assertion reads, or wait for it under a named lost-run bound that fails by name, and leave every assertion as strict or stricter.

Test file Fix
test_telegram_sessions.py _log() seeds through off_loop, so the bounded retry applies; this covers the 92 tests that build a log through it
test_diag_procs.py main already pins both scans to SCAN_MONOTONIC and ages the window to 0.5 s by hand (#17507); this change adds that the decline note must quote reads 0.5s apart, so the declined gap is the one the test set
test_skills.py _settled(loader) polls catalog_status() to "complete" under wait_until (_CATALOG_SETTLE_SECS = 30); every loader whose test enumerates goes through it, and so does every read that follows a mutation. A module autouse fixture requests close_skills_loaders so the settle's catalog worker and SQLite index are released
test_denied_commands_security.py the DNS hold (_DNS_HOLD_SECS = 60, released in finally) outlasts the wait_until publish poll (30 s); _join_resolver joins the resolver this test started — identified against the set _open_window records — and asserts it exited within 30 s
test_markdown_link_parentheses.py the 30 opener and reading guards go through assert_rejected_without_backtracking (thread CPU, ramp, then 10x pumps with a ratio bound), each keeping its original absolute ceiling: budget_seconds=0.5 for the two opener guards, 1.0 for the reading guard
test/conftest.py assert_rejected_without_backtracking gains one keyword-only budget_seconds (default REDOS_LARGE_BUDGET_SECONDS, unchanged), used where it applies the long pumps' absolute budget; every other caller behaves exactly as before

Backwards compatibility

Compatible: the diff changes test files only; no production module, fixture, configuration or pinned list changes.

Removes nothing: every assertion is kept or made stricter (test_iter_result_is_cached tightens calls["n"] <= 1 to == 1). The five test_each_reading_stays_linear parameter ids change from the raw 5,000-character payloads to names (openers, brackets, ...); nothing references them.

The absolute ceilings stay at their original constants, now read on thread CPU instead of wall time: 0.5 s at the 20,000-unit pump for the two opener guards, 1.0 s for the reading guard. One bound is wider and the measurement says why: the old ratio check was large <= max(8 * small@5k, 0.05), and the shared helper's is max(25 * cost@2k, 0.25). With the shipped cost at 0.0017 s of thread CPU the binding bound was that 0.05 s floor. A floor that low on a 15.6 ms Windows clock tick is the shape class 5 says reads a linear cost as unbounded, and the helper's floor is shared, not per call. The ratio exponent itself tightens (25x per 10x = 1.40 against 8x per 4x = 1.50), and the helper adds a 1-to-24 ramp the old shape did not have, which is the only thing that catches the exponential class.

Tests

All proofs ran on Linux CPython 3.12.13; host 1-minute load 4-6. The probes are out-of-tree pytest plugins and are not committed. The commit is rebased onto db083f5d3. On that base #17507 had already applied the same pin to the #17447 test, so its forced condition (+2 s per clock read) no longer reds the new parent; it reds the original parent cd0298429, and this head adds only the note assertion there. After the rebase: test_diag_procs.py 20x -n0 (54/54 each) and 10x -n 4 beside test_diag_threads.py and test_skills.py (308/308 each), the 646 touched tests once, and the flake-pattern ratchet all pass. With the ceilings restored: test_markdown_link_parentheses.py 20x -n0 (1,418/1,418 each) and 10x -n 4 beside two other ramp-helper files (1,503/1,503 each), every one of the 12 test files that call assert_rejected_without_backtracking once at the default budget (2,500 passed), the sleep probe still red on cd0298429 (30 of 31) and green on this head, and semgrep 0 new on conftest.py and the markdown file.

Forced condition: parent red, head green (same command on both, -n0).

Item Forced condition Parent cd0298429 Head
#17782 atomic_write sees Windows and the first replace onto a .jsonl raises PermissionError once red: PermissionError from _log in setup; file-wide 92 of 102 red 1 passed; file-wide 102/102, also with one refusal per distinct target
#17447 the procs clock advances 2 s per read red: assert 125.0 is None (main db083f5d3, which carries #17507's pin, passes) passed; file 54/54
#15503 _COLD_CATALOG_WAIT_SECS = 0.0 red: 9 of 11 in the class, 84 of 181 in the file 181/181
B6 netlink dump takes 5.1 s red: netlink never published (and sibling test_startup_warm_publishes_before_the_first_ip_literal, behind the silent 5 s join) class 277 passed
B6 the main thread stalls 5.2 s between the publish and the in-flight check red: assert False is True passed
B7 0.1 s sleep inside each large-pass pattern.sub, 1.05 s inside each reading of 2,000+ characters red: 30 of 30 30 passed

Negative controls on the head.

Mutation Head Parent
the disabled-app skill gate hides nothing, cold catalog 7 of 11 red by name, e.g. assert ['deploy_web/artifact-deploy'] == [] 2 of the 7 still pass (test_a_disabled_apps_skill_does_not_trigger, test_a_disabled_builtins_package_tree_skill_is_gated)
MIN_RATE_GAP_SECS = 0 red: assert 1000.0 is None n/a
every renderer link pattern becomes quadratic 5 of 5 red: handling a 20000-unit pump cost 2.42s of CPU -- superlinear in the pump length n/a
a LINEAR slowdown of 1.5e-5 s of thread CPU per character (0.6 s at the 20,000-opener pump, ratio still 10x) 27 of 31 red at the restored ceilings, e.g. handling a 20000-unit pump cost 0.60s of CPU; the 4 green are reading shapes under 1.0 s 31 of 31 pass on the previous head d76f46e94, which used the shared 2.0 s budget

Resource footprint. Settling a loader is what starts its skill-catalog-refresh worker and opens its SQLite index, so the first revision of this change settled 53 loaders in tests that never enumerate. Those settles are gone, and the remaining ones are released by the fixture. Measured over test_skills.py at -n0, workers parked and descriptors open at the end of the file: parent 102 and 315, first revision 113 and 348, this head 0 and 10.

Repeats.

Run Result
20x -n0, every touched test (646 ids) 20/20 green, 15.4-16.3 s each
10x -n 4 --dist loadgroup --max-worker-restart=2, the 5 files plus test_skill_catalog_snapshot.py and test_explain_for_skill.py (3,235 tests) 10/10 green, 19.0-21.5 s
3 shuffled orders (seeds 17782, 15503, 2026) 3/3 green
KC_FORCE=coarse_clock and slow_thread (0.2 s), every touched test, -n0 646/646 green each
KC_FORCE=gil_hog (switch interval 5 ms), the 46 card-item tests 46/46 green; over all 646 it did not finish inside 900 s on this host

local-gate.py --dry-run selects 10 backend files and 325 frontend cross-surface files. The backend set ran at -n 4 with test_flake_pattern_ratchet.py (4,438 passed, 1 skipped); the frontend set was not run because the diff touches no website/** path. Also run: mypy src/kiro_crew (clean), flake8, isort, the black gate, the flake-pattern ratchet (no file's count rises; test_markdown_link_parentheses.py drops its one duration bound to 0), docs-lint and the diff-scoped comment-history, brand, harness-parity and subprocess-encoding gates (all pass). Semgrep 1.78 with CI's rule set finds the same 7 pre-existing results in the base and head versions of the five files, so 0 new.

Manual verification

N/A: the tests are the subject. The exact-SHA Linux (3.12, 1-8), Windows (1-8) and macOS lanes are the cross-platform evidence and are recorded on this PR.

Related Issues

Part of #16536
Fixes #17782
Fixes #17447
Fixes #15503

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 7, 2026 19:37
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 6d98123f0cc784af82be485bb5a47f8c7afd4275 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Check that the markdown guards' ratio floor, which rises from 0.05 s to the shared helper's 0.25 s, still catches the regression class those guards were written for.

[DESIGN-REVIEWED] 6d98123

Description read

[DESCRIPTION-READ] e916553954beed082d875fa0d3922fe71d76fe49671939105bf3b0c404b0dc34

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 6d98123f0cc784af82be485bb5a47f8c7afd4275 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 6d98123

Verdict parsed from the review's SHA-scoped output markers for commit 6d98123f0cc784af82be485bb5a47f8c7afd4275.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 6d98123f0cc784af82be485bb5a47f8c7afd4275: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 6d98123f0cc784af82be485bb5a47f8c7afd4275 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 6d98123

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 6d98123f0cc784af82be485bb5a47f8c7afd4275: <one-sentence reason>

@bolichen97

Copy link
Copy Markdown
Contributor Author

Design Review note on 3e1eaa66b0e1 ("check that maintainers accept the wider constant-factor budget in the three markdown linear-time guards: it moves from a 0.05 s floor to 0.25 s"): recorded on purpose, with the measurement, under Backwards compatibility in the description. In short:

  • The old binding bound at the 20,000-unit pump was the max(8 * small, 0.05) floor, read on wall clock; the shipped cost there is 0.0017 s of thread CPU.
  • assert_rejected_without_backtracking is what testing-conventions class 5 prescribes for a regex, and commit 8cc031d converted the sibling guard in this same file the same way. Its budgets are shared, not per call.
  • What it buys: a 1-to-24 one-unit ramp, the only shape that catches the exponential class (the old 20,000-character probe would not return under it), and thread CPU instead of wall time, which is the flake being fixed.
  • What it costs: a pure constant-factor slowdown landing between 0.05 s and 0.25 s at that one size no longer reds.
  • The shape regression these guards exist for still does: the quadratic label-class mutant costs 2.5-3.2 s of thread CPU and fails superlinear in the pump length on all five renderers, and the ratio exponent tightens (25x per 10x = 1.40 vs 8x per 4x = 1.50).

No code change proposed; a maintainer who wants the 0.05 s constant kept can ask for one absolute assertion at the shipped size beside the helper call.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

The fixes go after real causes rather than adding retries. Seeding off the event loop fixes the WinError 5 setup failure, a pinned monotonic clock fixes the diag rate test, the skills catalog now waits for complete, the DNS stub's hold now outlasts the publish wait, and the markdown guards measure thread CPU with a ramp. The forced-condition and mutant evidence is strong. I'm holding for a maintainer for two reasons. First, the PR edits security-gating tests: TestSandboxEscapeSshSelf in test/test_denied_commands_security.py and the redact_for_display ReDoS guards in test/test_markdown_link_parentheses.py. Second, it loosens one bound in a test-only change: the constant-factor floor moves from 0.05 s to 0.25 s. You disclosed that bound and justified it, and Design Review flagged it for maintainer acceptance, so it needs a human sign-off rather than an automated approval.

@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Oct 8, 2026
@bolichen97
bolichen97 force-pushed the refactor/df-b3c-windows-macos-timing branch from 3e1eaa6 to d76f46e Compare October 8, 2026 00:57
@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed merge conflict Branch has merge conflicts with its base — author must resolve before merge readiness: passed Eligible automated validation passed for the current revision labels Oct 8, 2026
@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ⏭️ skipped

Revision 6d98123f0cc784af82be485bb5a47f8c7afd4275 ships no reviewable capability (docs, tests or generated files only), so there is nothing to inventory. Advisory — does not block merge.

@github-actions

github-actions Bot commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ⏭️ skipped

Revision 6d98123f0cc784af82be485bb5a47f8c7afd4275 touches no user-facing surface (no changes under website/ or committed screenshots), so the UX review was skipped. Advisory — does not block merge.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 8, 2026
Five test families went red on loaded Windows and macOS runners for
reasons unrelated to the change under test, and a rerun went green.
Each bet on runner speed:

- test_telegram_sessions: _log() seeded its transcript on the event
  loop, where replace_with_retry re-raises a Windows sharing violation
  instead of retrying (WinError 5). It now seeds through off_loop.
- test_diag_procs: main already pins both scans of the too-close rate
  test and ages the window to 0.5 s by hand; the decline note is now
  pinned to that hand-aged gap as well.
- test_skills: a fresh or just-mutated skills root serves [] marked
  "building" once its 2 s cold budget runs out. Every loader, and
  every read after a mutation, goes through _settled(), which waits
  for catalog_status() == "complete" under a named lost-run bound.
- test_denied_commands_security: the stubbed DNS hold and the publish
  poll were both 5 s. The hold now outlasts the wait_until poll, and
  the resolver join asserts the thread exited.
- test_markdown_link_parentheses: the linear-time guards read one
  wall-clock sample. They use assert_rejected_without_backtracking,
  which gains a keyword-only budget_seconds (default unchanged), so
  each guard keeps its original ceiling on thread CPU: 0.5 s for the
  two opener guards, 1.0 s for the reading guard.

Part of #16536
Fixes #17782
Fixes #17447
Fixes #15503
@bolichen97
bolichen97 force-pushed the refactor/df-b3c-windows-macos-timing branch from d76f46e to 6d98123 Compare October 8, 2026 01:45
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=a230bb1d3fe6 Preserve the existing complexity-guard ceilings (accepted and fixed in 6d98123)

fixed: assert_rejected_without_backtracking gains one keyword-only budget_seconds (default REDOS_LARGE_BUDGET_SECONDS, so the 12 other caller files run unchanged: 2,500 passed).
The two opener guards pass budget_seconds=0.5 and the reading guard 1.0: their original ceilings, now read on thread CPU.
Negative control: a linear slowdown of 1.5e-5 s CPU per character (0.6 s at the 20,000-opener pump, ratio still 10x) reds 27 of 31 guards on 6d98123 and passes all 31 on d76f46e.
The quadratic mutant still reds 5 of 5; the descheduling sleep probe reds the original parent cd02984 and stays green here.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

E2E (stub ACP backend, offline) on 6d98123f0cc7 (run 37714471308, job 113108622440) failed only the zero-flaky ceiling: chat.spec.ts:62 › displays streaming response passed on its retry (322 passed, 1 flaky). That is the open flake-ledger entry #17857 (5+ sightings today on unrelated PRs); this PR changes backend test files only, no website/** or chat code, and the same job passed on both earlier heads. Rerun once with gh run rerun 37714471308 --failed.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 8, 2026
@bolichen97

Copy link
Copy Markdown
Contributor Author

Design Review note on 6d98123f0cc7 ("check that the markdown guards' ratio floor, which rises from 0.05 s to the shared helper's 0.25 s, still catches the regression class those guards were written for"): it does, measured on this head.

  • The class these guards name is a label class that admits [ and so rescans to the end of the text from every opener: quadratic. With every renderer link pattern swapped for that mutant, all 5 renderer guards red: handling a 20000-unit pump cost 2.42s of CPU -- superlinear in the pump length. That cost is about 10x the 0.25 s floor and about 5x the restored 0.5 s absolute ceiling, so the mutant fails twice over.
  • The absolute ceilings are back at their original constants (0.5 s for the opener guards, 1.0 s for the reading guard), now on thread CPU. A LINEAR slowdown that keeps the 10x ratio but costs 0.6 s at the 20,000-opener pump reds 27 of 31 guards on this head; it passes all 31 on the previous head, which used the shared 2.0 s budget.
  • What the wider floor gives up: a ratio violation whose 20,000-unit cost lands between 0.05 s and 0.25 s. Such a cost is also below the 0.5 s absolute ceiling, so neither bound catches it. On a 15.6 ms Windows clock tick, a 0.05 s floor can read a linear cost as an unbounded ratio, which is why the helper's floor is shared rather than set per call.

No code change.

@KiroCrewCaptain
KiroCrewCaptain enabled auto-merge (squash) October 8, 2026 04:23

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test-only PR removes five runner-speed bets that turned Windows and macOS shards red. Each test now sets the state it asserts, or waits for it under a named lost-run bound, so the root cause is fixed rather than papered over with retries. I read the whole diff. Here is what each fix does. The Telegram _log() now seeds its log off the event loop, so replace_with_retry's bounded retry applies. The skills tests settle catalog_status() to "complete" via wait_until, so a slow walk can no longer turn an empty catalog into a false pass on the negative test, and the new autouse close_skills_loaders fixture releases the workers. The DNS hold (60 s, released in finally) now outlasts the 30 s publish poll, and _join_resolver joins only the resolver this test started, then asserts it exited. The markdown guards move to assert_rejected_without_backtracking with their original absolute ceilings (0.5 s / 1.0 s) kept through the new keyword-only budget_seconds, whose default leaves every other caller unchanged. On the Design lane's question about the ratio floor: it rises from 0.05 s to 0.25 s, but the shared helper now ramps over 10x steps on thread CPU with a 25x per-step bound, which still catches a quadratic regression (100x per step), and the absolute ceiling is unchanged. The GPT ceiling finding was fixed in 6d98123. No production code or policy changes, so touching test_denied_commands_security.py is not a security-policy change. All checks are green, and Opus and GPT report no findings. — crew-pr-reviewer

@KiroCrewCaptain
KiroCrewCaptain merged commit fc70925 into main Oct 8, 2026
152 of 153 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the refactor/df-b3c-windows-macos-timing branch October 8, 2026 06:17
@github-actions github-actions Bot removed the readiness: passed Eligible automated validation passed for the current revision label Oct 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants