Repository navigation
test: guard frontend test determinism (needs maintainer approval) - #17673
Conversation
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS Before merge, a maintainer should approve [DESIGN-REVIEWED] 0ef6e04 Description read[DESCRIPTION-READ] 220b3d310233a95e8053475300b027cbc60bea9b2246711e6009bff2d2501657 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. [UX-REVIEWED] 0ef6e04 Description read[DESCRIPTION-READ] 45ce70e413a259dcf3fd256cb0ef5d9a8b24a6a537e80f13b378c8e51a131e93 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
88ac77e to
20caacd
Compare
|
A vitest file that leaves fake timers installed or writes storage without undoing it hands that state to every later test in the file; vite.config.ts dropped every "not wrapped in act()" warning, so a test whose state lands after its last barrier never showed it; and Playwright retries let a flaky spec pass silently while a dead gateway turned every remaining spec into a retried timeout. integration/setup.ts now restores, when each test finishes, what it inherited: real timers when the test installed fakes, and localStorage / sessionStorage as they were when it started (two files that read an earlier test's writes are exempt until fixed). vite.config.ts sets unstubEnvs. setup.ts counts act() warnings per file against integration/act-warning-baseline.json, the 475 files that still emit them: another file's warnings are printed and reported on stderr, a listed file whose full run emits none is reported for removal, and KIROCREW_ACT_STRICT=1 fails the test (or the file) instead. A self-test file pins the guard and keeps both lists from growing. test_playwright_e2e.py fails a run with more flaky specs than MAX_FLAKY_SPECS (0), writes the counts and flaky titles to the job summary where it is writable, bounds the whole run inside the CI job, and stops it when the gateway exits: SIGINT first so the report is written, then the whole process group. That ceiling is the one flaky policy; playwright.config.ts keeps retries as the detector. The frontend testing doc and the e2e gate doc describe both. Part of #16536
20caacd to
0ef6e04
Compare
|
|
Reviewed, not approving (crew-pr-reviewer). The title says it needs maintainer approval, and it adds a new repo-wide test policy. That is a maintainer's decision. |
iamwhatever
left a comment
There was a problem hiding this comment.
Approving after a deeper pass (crew-pr-reviewer). It fixes three real harness gaps: fake timers and storage leaking between tests, act() warnings being hidden, and Playwright retries masking flaky specs or a dead gateway. The act-warning baseline lets the current 475 files keep passing while new ones are counted, so it does not turn every PR red on day one. Design and UX lanes PASS. CI is green and the base is main.
Problem / Motivation
Goal: A frontend test can no longer leak fake timers or storage into the tests after it, an
act()warning is counted instead of hidden, and the Playwright gate fails on a flaky spec or a dead gateway instead of retrying past it.Three harness gaps let a frontend test pass or fail by order or by luck:
localStoragewithout undoing it hands that state to every later test in the file. Two files already depend on such a leak and fail when run alone or shuffled.vite.config.ts'sonConsoleLogdrops every "not wrapped in act(...)" warning, so a test whose state update lands after its last barrier never shows it. A census of two full runs on this base found 475 files that emit them.Why it matters
Each of these is a flaky red waiting for an unrelated PR, or a real defect a green run hides. The determinism contract and the review rules already forbid the shapes; the harness should stop the leaks and surface the rest.
Not a goal
KIROCREW_ACT_STRICT=1opts in).restoreMocksormockReset.playwright/andintegration/, a new devDependency,.gitattributes, or any CI workflow change (proposals only, below).What changed (motivation → approach → change)
The harness now restores what a test leaves behind, counts the warning it used to hide, and treats a retried pass as the failure it is.
website/integration/setup.tsonTestFinished, after its own hooks), real timers come back if the test installed fake ones, andlocalStorage/sessionStoragereturn to what the test inherited (abeforeAllseed survives). Two files that read an earlier test's storage keep it until fixed (STORAGE_CARRYOVER_FILES, shrink-only). act() warnings are counted per file: a file inintegration/act-warning-baseline.json(475, from the census) is kept quiet; any other file's warning prints as usual plus an[act-warnings]line on stderr; a listed file whose full run emits none is reported for removal;KIROCREW_ACT_STRICT=1fails the test (or the file).website/integration/setup.refactor.determinism.test.ts(new)console.error), the restore, and both shrink-only lists.website/vite.config.tsunstubEnvs: true; the act() line leavesonConsoleLog.test/test_playwright_e2e.py,website/playwright.config.tsMAX_FLAKY_SPECS = 0is the one flaky policy, withretrieskept as the detector (three local full passes, the last on this base: 323 specs, 0 flaky); counts and flaky titles to the job summary where writable; the run is bounded inside the CI job (PLAYWRIGHT_RUN_CEILING_SECS) and stops when the gateway exits, SIGINT first so the report survives, then the whole process group.website/docs/testing.mdanddocs/ci/e2e-gate.mddescribe the restores, the act() report, and what to do when someone else's flaky spec reds your PR.Needs maintainer approval
MAX_FLAKY_SPECS = 0makes a spec that passes only on a retry fail the E2E gate. Three local passes found none, and so did both CI E2E runs on this PR (runs 37600164324 and 37603472762, each passing under that ceiling). A contributor hit by someone else's flaky spec records it in the flake ledger and reruns once (e2e-gate.md)..gitattributeswebsite/** text=auto eol=lf(above the imagebinarylines), a vitest fork-crash forensics step, and the CI lanes in the campaign's proposal list.Backwards compatibility
Compatible: no product code changes. A test that passes today keeps passing:
KIROCREW_ACT_STRICT=1is set;Removes nothing: the act() filter in
onConsoleLoghid output only, and its warnings now print or are counted.Tests
website/integration/setup.refactor.determinism.test.ts(new, 6 tests):String()cannot convert, and recognises the warning by a string argument only;test/test_playwright_e2e.py(16 unit tests, 11 new):Manual verification
0d885e681): two full vitest runs (2,656 files, 45,124 tests each, all passing).src/test/CrewChatWindow.test.tsx(added 2026-10-05)src/test/MarkdownRenderer.lightboxCopy.test.tsx(added 2026-10-05)src/test/ComposerSendDuringUpload.test.tsx(added 2026-10-06)src/test/ChatSidebar.refactor.touchRowMenu.test.tsx(added 2026-10-07)src/test/App.metricsPopover.test.tsx(0 to 4 warnings after fix(website): collapse the desktop top bar instead of clipping it #15656 changed it on 2026-10-06)src/test/DeveloperPanel.test.tsxandsrc/pages/settings/decisionsProviderPicker.test.tsx.env -iwith an isolatedHOME/KIROCREW_HOMEand the harness's own ephemeral port.voice-recovery.spec.tsspecs because the servedsrc/kiro_crew/static/distwas a stale local build from before fix(chat): keep Copy a visible row button on every message #16977. After rebuilding and stagingwebsite/dist(asmake builddoes), all specs pass.--sequence.shuffle;-n0, 10/10 at-n 4beside two neighbour files, and 5/5 shuffled seeds.tsc -p tsconfig.app.json, the flake-pattern ratchet againstorigin/main, subprocess-encoding, black (the file is on the black baseline), brand, comment-history,scripts/docs_lint.py, and the CI semgrep command in baseline mode: 0 findings.eslintdoes not read these files today (its config coverssrc; see the proposals).Related Issues
Part of #16536
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)