Skip to content

test: guard frontend test determinism (needs maintainer approval) - #17673

Merged
iamwhatever merged 1 commit into
mainfrom
refactor/df-g4-frontend-guards
Oct 7, 2026
Merged

iamwhatever merged 1 commit into
mainfrom
refactor/df-g4-frontend-guards

Conversation

@bolichen97

@bolichen97 bolichen97 commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A frontend test can no longer leak fake timers or storage into the tests after it, an act() warning is counted instead of hidden, and the Playwright gate fails on a flaky spec or a dead gateway instead of retrying past it.

Three harness gaps let a frontend test pass or fail by order or by luck:

  • A vitest file that installs fake timers or writes localStorage without undoing it hands that state to every later test in the file. Two files already depend on such a leak and fail when run alone or shuffled.
  • vite.config.ts's onConsoleLog drops every "not wrapped in act(...)" warning, so a test whose state update lands after its last barrier never shows it. A census of two full runs on this base found 475 files that emit them.
  • Playwright's CI retries let a flaky spec pass silently, the flaky count is never shown, and a gateway that dies mid-run turns every remaining spec and retry into a timeout.

Why it matters

Each of these is a flaky red waiting for an unrelated PR, or a real defect a green run hides. The determinism contract and the review rules already forbid the shapes; the harness should stop the leaks and surface the rest.

Not a goal

  • Fixing the files that emit act() warnings, or the two files that depend on a storage leak (their owners' follow-ups).
  • Failing a run on an act() warning by default. That needs a CI measurement first (KIROCREW_ACT_STRICT=1 opts in).
  • Restoring globals after each test, restoreMocks or mockReset.
  • Linting playwright/ and integration/, a new devDependency, .gitattributes, or any CI workflow change (proposals only, below).
  • Any production code change.

What changed (motivation → approach → change)

The harness now restores what a test leaves behind, counts the warning it used to hide, and treats a retried pass as the failure it is.

Area Change
website/integration/setup.ts When each test finishes (from onTestFinished, after its own hooks), real timers come back if the test installed fake ones, and localStorage / sessionStorage return to what the test inherited (a beforeAll seed survives). Two files that read an earlier test's storage keep it until fixed (STORAGE_CARRYOVER_FILES, shrink-only). act() warnings are counted per file: a file in integration/act-warning-baseline.json (475, from the census) is kept quiet; any other file's warning prints as usual plus an [act-warnings] line on stderr; a listed file whose full run emits none is reported for removal; KIROCREW_ACT_STRICT=1 fails the test (or the file).
website/integration/setup.refactor.determinism.test.ts (new) Pins the guard (a null-prototype, revoked-proxy or throwing argument never breaks console.error), the restore, and both shrink-only lists.
website/vite.config.ts unstubEnvs: true; the act() line leaves onConsoleLog.
test/test_playwright_e2e.py, website/playwright.config.ts MAX_FLAKY_SPECS = 0 is the one flaky policy, with retries kept as the detector (three local full passes, the last on this base: 323 specs, 0 flaky); counts and flaky titles to the job summary where writable; the run is bounded inside the CI job (PLAYWRIGHT_RUN_CEILING_SECS) and stops when the gateway exits, SIGINT first so the report survives, then the whole process group.
docs website/docs/testing.md and docs/ci/e2e-gate.md describe the restores, the act() report, and what to do when someone else's flaky spec reds your PR.

Needs maintainer approval

  • MAX_FLAKY_SPECS = 0 makes a spec that passes only on a retry fail the E2E gate. Three local passes found none, and so did both CI E2E runs on this PR (runs 37600164324 and 37603472762, each passing under that ceiling). A contributor hit by someone else's flaky spec records it in the flake ledger and reruns once (e2e-gate.md).
  • Proposals, not in this diff: eslint-plugin-playwright, .gitattributes website/** text=auto eol=lf (above the image binary lines), a vitest fork-crash forensics step, and the CI lanes in the campaign's proposal list.

Backwards compatibility

Compatible: no product code changes. A test that passes today keeps passing:

  • the restores undo only what a test itself left behind;
  • an act() warning prints and is reported but does not fail a run unless KIROCREW_ACT_STRICT=1 is set;
  • the E2E gate fails only on a spec that passed on a retry (0 in two local passes) or a dead gateway, which failed it already.

Removes nothing: the act() filter in onConsoleLog hid output only, and its warnings now print or are counted.

Tests

  • website/integration/setup.refactor.determinism.test.ts (new, 6 tests):
    • the act() guard never throws on an argument String() cannot convert, and recognises the warning by a string argument only;
    • the act baseline stays sorted, unique, existing and within its census (475), and the storage carry-over list stays at two files;
    • a test that leaves fake timers and a storage write behind hands neither to the next test.
  • test/test_playwright_e2e.py (16 unit tests, 11 new):
    • the flaky ceiling fails the run, and flaky titles are read from nested suites with their describe blocks;
    • the counts and flaky titles reach the job summary, and an unwritable summary does not fail the run;
    • a run is stopped when its gateway exits or past its bound, and Playwright is reaped when the poll raises;
    • a partial or missing report salvages nothing and never raises.

Manual verification

  • act census on this base (0d885e681): two full vitest runs (2,656 files, 45,124 tests each, all passing).
    • Added, reported outside the census in both runs, all from upstream since the first census:
      • src/test/CrewChatWindow.test.tsx (added 2026-10-05)
      • src/test/MarkdownRenderer.lightboxCopy.test.tsx (added 2026-10-05)
      • src/test/ComposerSendDuringUpload.test.tsx (added 2026-10-06)
      • src/test/ChatSidebar.refactor.touchRowMenu.test.tsx (added 2026-10-07)
      • src/test/App.metricsPopover.test.tsx (0 to 4 warnings after fix(website): collapse the desktop top bar instead of clipping it #15656 changed it on 2026-10-06)
    • Removed, no warning in either run: src/test/DeveloperPanel.test.tsx and src/pages/settings/decisionsProviderPicker.test.tsx.
    • The list is 475 files and shrinks only from the day it lands.
  • Playwright: one full pass of the whole E2E suite on a freshly built frontend: 323 passed, 0 flaky, 0 unexpected, 9.0 min. It ran under env -i with an isolated HOME / KIROCREW_HOME and the harness's own ephemeral port.
  • Repeats:
    • the self-test file 10/10 plain and 10/10 with --sequence.shuffle;
    • the harness tests 20/20 at -n0, 10/10 at -n 4 beside two neighbour files, and 5/5 shuffled seeds.
  • Gates: tsc -p tsconfig.app.json, the flake-pattern ratchet against origin/main, subprocess-encoding, black (the file is on the black baseline), brand, comment-history, scripts/docs_lint.py, and the CI semgrep command in baseline mode: 0 findings. eslint does not read these files today (its config covers src; see the proposals).

Related Issues

Part of #16536

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested a review from a team as a code owner October 7, 2026 09:22
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Oct 7, 2026
@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 0ef6e047bcda420f82e5f0ee27e1cca07a793a36 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

Design-Verdict: PASS

Before merge, a maintainer should approve MAX_FLAKY_SPECS = 0: from then on, a spec that passes only on a retry fails the E2E gate on every PR, including PRs that didn't touch it.

[DESIGN-REVIEWED] 0ef6e04

Description read

[DESCRIPTION-READ] 220b3d310233a95e8053475300b027cbc60bea9b2246711e6009bff2d2501657

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 0ef6e047bcda420f82e5f0ee27e1cca07a793a36 — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS/CONCERNS are advisory.

UX-Verdict: PASS

Nothing to check.

[UX-REVIEWED] 0ef6e04

Description read

[DESCRIPTION-READ] 45ce70e413a259dcf3fd256cb0ef5d9a8b24a6a537e80f13b378c8e51a131e93

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 0ef6e047bcda420f82e5f0ee27e1cca07a793a36 — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0ef6e04

Verdict parsed from the review's SHA-scoped output markers for commit 0ef6e047bcda420f82e5f0ee27e1cca07a793a36.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 0ef6e047bcda420f82e5f0ee27e1cca07a793a36: <one-sentence reason>

@github-actions

github-actions Bot commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 0ef6e047bcda420f82e5f0ee27e1cca07a793a36 and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0ef6e04

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 0ef6e047bcda420f82e5f0ee27e1cca07a793a36: <one-sentence reason>

@bolichen97
bolichen97 force-pushed the refactor/df-g4-frontend-guards branch from 88ac77e to 20caacd Compare October 7, 2026 09:51
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=9d3080f05c8b: fixed (wait for PID contents before terminating the child)

Accepted and fixed in 20caacd. The window is closed at its source: the child writes child.pid.tmp and os.replace()s it onto child.pid, so the file only ever appears complete. The reader is robust as well: poll() waits until the text parses as a positive int, caches that pid, then raises, and the assertion checks the cached pid.
Forced proof (scratch, not committed): a 0.5 s delay between the child creating and writing its pid file reds 88ac77e's shape with "ValueError: invalid literal for int() with base 10: ''" in 5/5 runs. The fixed writer and reader pass, and the fixed reader alone survives the old writer.
Repeats on the fix: 20/20 at -n0 and 10/10 at -n 4. G4 holds no other exists-then-read shape on a live writer; the report is read only after the run is reaped.

A vitest file that leaves fake timers installed or writes storage
without undoing it hands that state to every later test in the file;
vite.config.ts dropped every "not wrapped in act()" warning, so a test
whose state lands after its last barrier never showed it; and
Playwright retries let a flaky spec pass silently while a dead gateway
turned every remaining spec into a retried timeout.

integration/setup.ts now restores, when each test finishes, what it
inherited: real timers when the test installed fakes, and localStorage
/ sessionStorage as they were when it started (two files that read an
earlier test's writes are exempt until fixed). vite.config.ts sets
unstubEnvs. setup.ts counts act() warnings per file against
integration/act-warning-baseline.json, the 475 files that still emit
them: another file's warnings are printed and reported on stderr, a
listed file whose full run emits none is reported for removal, and
KIROCREW_ACT_STRICT=1 fails the test (or the file) instead. A self-test
file pins the guard and keeps both lists from growing.

test_playwright_e2e.py fails a run with more flaky specs than
MAX_FLAKY_SPECS (0), writes the counts and flaky titles to the job
summary where it is writable, bounds the whole run inside the CI job,
and stops it when the gateway exits: SIGINT first so the report is
written, then the whole process group. That ceiling is the one flaky
policy; playwright.config.ts keeps retries as the detector. The
frontend testing doc and the e2e gate doc describe both.

Part of #16536
@bolichen97
bolichen97 force-pushed the refactor/df-g4-frontend-guards branch from 20caacd to 0ef6e04 Compare October 7, 2026 10:30
@bolichen97

Copy link
Copy Markdown
Contributor Author
  • span=974c06618799: rebutted (Watch, the flaky-spec rule rests on local runs only)

Main's e2e logs hold no flaky count. pytest captures the harness's stdout on a passing run, so the [test:e2e:playwright] ... stats= line never reaches the job log: run 37603664378, job 112735122722 on main has no such line. Main's harness also counted a flaky spec as executed and passed it, so main's flaky rate cannot be read back from its runs.
The measurements that exist are 0: both CI E2E runs on this PR passed under the enforcing ceiling (runs 37600164324 and 37603472762, flaky = 0), and so did three local full passes (323 specs each).
The ceiling names every flaky title in the failure and the job summary, so a red points at one spec to fix. e2e-gate.md tells a contributor hit by someone else's flake to record it in the flake ledger and rerun once.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 7, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

Reviewed, not approving (crew-pr-reviewer). The title says it needs maintainer approval, and it adds a new repo-wide test policy. That is a maintainer's decision.

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving after a deeper pass (crew-pr-reviewer). It fixes three real harness gaps: fake timers and storage leaking between tests, act() warnings being hidden, and Playwright retries masking flaky specs or a dead gateway. The act-warning baseline lets the current 475 files keep passing while new ones are counted, so it does not turn every PR red on day one. Design and UX lanes PASS. CI is green and the base is main.

This was referenced Nov 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants