Repository navigation
test(e2e): keep a trace on 17b's first failure, wait for row B - #18529
iamwhatever wants to merge 2 commits into
Conversation
First Principles Review (Opus 5.5) — ⏭️ skippedRevision |
UX Review (Opus 5.5) — ✅ PASSUX-level review of UX-Verdict: PASS Nothing to check. [UX-REVIEWED] e4bb54c Description read[DESCRIPTION-READ] ac1788727ef257716e5746d915085971413525d212ebb809336db0f9174351c7 That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
GPT 6.1 Review — ✅ no blocking findingsGPT 6.1 completed its review of This comment is updated in place on each push. Review detailsNo findings. False positive or not applicable? A repository writer can comment: |
Opus 5.5 Review — ✅ no blocking findingsReviewed Verdict parsed from the review's SHA-scoped output markers for commit False positive or not applicable? A repository writer can comment: |
Design Review (Opus 5.5) — ✅ PASSDesign-level review of Design-Verdict: PASS The title says "keep a trace on 17b's first failure", but this diff keeps no trace and the description says trace retention was removed. Retitle before merge. Suggestions
[DESIGN-REVIEWED] e4bb54c Description read[DESCRIPTION-READ] 4e6dd4aa721906385901aefad3d0edb75610824dcb053f78877bc8349b98ab29 That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
37ad0a1 to
adf6b0d
Compare
|
|
|
|
|
|
|
adf6b0d to
d52132f
Compare
|
|
|
|
|
|
buluoray
left a comment
There was a problem hiding this comment.
Thanks, the fresh-page-per-attempt shape and the stable-box wait are good, and keeping a trace when every attempt fails is exactly what #17941 needs. One blocking problem with the masked-miss path:
A recovered miss leaves no evidence anywhere on CI. Inside toPass, a miss that a later attempt recovers makes 17b pass as expected, not flaky, so the MAX_FLAKY_SPECS = 0 ceiling (docs/ci/e2e-gate.md: retries detect a flake, they never fix one) stops seeing it. The PR offers the [flaky-gesture] line as the replacement evidence, but on CI it never reaches the job log:
- The harness runs Playwright with
--reporter=html,json(test/test_playwright_e2e.py:435). Neither reporter echoes test stdout to the terminal, soconsole.loggoes only into the report files. playwright-report/andtest-results/are uploaded onlyif: failure()(ci.yml, "Upload Playwright failure artifacts"). On a green run the annotation and the stdout are discarded with the runner.
So an intermittent regression in backdrop-forwarded right-click, the very race this issue is about, would now pass with no signal at all. A deterministic break still goes red, which is good.
Any one of these would fix it:
- Have the harness read
flaky-gestureannotations from the JSON report it already parses, print them, and write them to the step summary (ideally counting them against the flaky ceiling). - Emit the line on a channel the job log shows, and upload the report whenever such an annotation exists.
- Drop the
toPasswrapper and keep only the stable-box wait plusretain-on-failure. A remaining miss would then still trip the flaky gate, and it would now come with a trace.
Non-blocking:
- The catch block's
page.evaluatecan throw on a crashed page and replace the original error. Consider guarding it so the assertion error is always the one rethrown. - Please note in #17941 that the wrapper should come out once the root cause is fixed.
d52132f to
6735b55
Compare
|
@buluoray Thanks, agreed: the Blocking: a recovered miss now shows on CI, green runs included.
Not counted against the flaky ceiling, on purpose. Counting it would turn every recovered miss red again, which is the same as today's red on unrelated PRs, and option 3 does that with less code. If you would rather have the ceiling count it, it is a small change in Non-blocking:
|
6735b55 to
84ed380
Compare
|
|
84ed380 to
9605e6d
Compare
|
@buluoray Switched to your option 3 in 9605e6d. CI made the case: at 84ed380, all 6 in-test
17b can still flake on this PR's own E2E run until the race is fixed. That is the known #17941 flake, not a new one. |
9605e6d to
29e56c0
Compare
|
The new hit-test probe caught 17b's CI miss at 9605e6d: at row B's click point the first element under the backdrop was the sidebar search input (the floating ListDock), not row B. In 29e56c0, 17b centres row B in its scroller and checks that the point reaches row B before right-clicking. Evidence: #17941 (comment). The other flaky spec on that run, 18, crashed in browser.newContext (chrome-headless-shell SEGV), which this PR does not touch. |
29e56c0 to
ae8d7d8
Compare
The sidebar's search dock floats over the top of the list, so row B can be visible yet covered at the click point, and the backdrop then forwards the right-click to the search input. Centre row B in its scroller and check the point reaches row B before right-clicking. Tag a miss with the page state and hit tests at the click point. Fixes #17941
A spec can tag a failing attempt with a flaky-gesture annotation holding the page state at the failure. The reporters keep test stdout out of the job log and the report is uploaded only from a red job, so the harness now reads those annotations from the JSON report and prints each one to the log and the step summary. The tag is evidence only: the attempt still fails, and a passing retry still counts against MAX_FLAKY_SPECS.
ae8d7d8 to
e4bb54c
Compare
|
Another recurrence of 17b (session-tags-e2e.spec.ts, right-click on another row) passing only on retry, on unrelated backend-only PR #18412 head 47caa58: https://github.com/kirodotdev/KiroCrew/actions/runs/38004325522/job/114070347259 . Same failure: expect(menu).toBeVisible() at line 783 times out at 5000ms on the first attempt. |
|
Closing: #18644 landed the same 17b fix on main (keep row B clear of the search dock, then check the point beneath the backdrop is row B) and closed #17941. It was built from the trace and hit-test probe this PR captured on CI, so nothing here is still needed for 17b. Thanks @buluoray for the review that pushed this toward evidence instead of a retry. |
Problem / Motivation
Goal: Stop 17b aiming its right-click at a point the sidebar's search dock covers, and make any later CI failure leave the evidence to explain it.
Spec
session-tags-e2e.spec.ts› 17b (right-click on another row while the Tags picker is open) fails on CI's first attempt and passes on the retry.MAX_FLAKY_SPECS=0then fails the whole E2E job. It has hit at least 10 unrelated PRs since 2026-09-30 (see #17941). The failing attempt stops atexpect(menu).toBeVisible(): picker closed,__ctxtrue, no menu open, focus on body.Why it matters
Every PR that hits it goes red for a spec it never touched.
Not a goal
toPass) was tried in an earlier revision and removed: on CI all 6 in-test reloads missed in the same browser context (run 37991140068, evidence in Flaky: session-tags-e2e.spec.ts 17b right-click on another row passes only on a retry in E2E (stub ACP backend, offline) #17941), so it masked nothing and only weakened the gate.MAX_FLAKY_SPECS.test.use({ trace: 'retain-on-failure', screenshot: 'only-on-failure' })at file level; it caught the first CI trace of 17b, but while it was on,browser.newContextcrashed Chrome (SEGV) in other specs of this file on three CI runs (specs 18; 16b and 26; 2), and on no main or other-PR run checked. Playwright refuses these options inside adescribe, so it cannot be narrowed to 17b. It is removed; theflaky-gestureannotation and hit test carry a miss's evidence.What changed (motivation → approach → change)
Test-only, three files:
website/playwright/session-tags-e2e.spec.ts, 17b: before aiming at row B, centre it in its scroller (scrollIntoView({ block: 'center' })), wait until its box is the same across two animation frames, and poll that the first element under the backdrop at the click point is inside row B. The sidebar'sListDockfloats the search capsule over the top of the list, and CI's miss had the sidebar search input there (see Evidence). If the point is still covered, 17b fails with a geometry message instead of a missing menu.flaky-gestureannotation with the step that failed, the page state (menu count, picker, focus, row B live/stub, itsdata-state,__ctx), the click point, and anelementsFromPointhit test at that point taken before the right-click (what the backdrop'selementBeneath()walks) and after the miss. Then rethrow, so the attempt still fails and the flaky ceiling still sees it. The page reads are guarded, so a crashed page never replaces the assertion error.test/test_playwright_e2e.py: read everyflaky-gestureannotation (GESTURE_MISS_ANNOTATION) from the JSON report the harness already parses, and print each one to the job log and$GITHUB_STEP_SUMMARY. The reporters keep test stdout out of the job log, so without this the annotation is only inside the uploaded report.docs/ci/e2e-gate.mdnames the channel.Evidence
At 9605e6d the hit-test probe caught a CI miss (run 37995411780). At row B's click point
(269, 105),elementsFromPointreturned the backdrop, theninputinsidesearch-field-glass, then row B. SoelementBeneath()forwarded the right-click to the search input. Details in #17941.With the fix (centre row B, check the point reaches row B), 17b passed on CI's first attempt at 29e56c0 (run 37997670466): not in the flaky list, no
flaky-gestureline. In the same window, 17b was the flaky spec on 4 of 4 other PRs' E2E runs (runs 37998093717, 37998079541, 37997794520, 37997518889).Backwards compatibility
Compatible: test-only change; no product code or contract is touched.
Removes nothing: the spec keeps every assertion it had; it only adds a scroll, a hit-target check and a failure annotation.
Tests
tests[].annotationsandresults[].annotations, read oncescrollIntoView(false), reaches row B after it (true)["div[role=button].fixed inset-0","span[role=button][in row B].inner","div[in row B].","html."][flaky-gesture]lines reached the job logThe spec file runs
mode: 'serial', so a shuffled order does not apply.Manual verification
Reproduction attempts on origin/main before the change, all 0 failures of 17b: 30x alone, 40x under full CPU load, 20x with 250 seeded sessions, 20x at 3x CDP CPU throttle, whole spec 8x, about 33 full-suite passes. An event-order logger on 13 passes showed the same order every time: trusted
contextmenuon the backdrop → forwardedcontextmenuon row B → menu focus → backdrop removed.Related Issues
Fixes #17941
Pattern harvest
A Playwright
toBeVisible()row can still be covered by a floating dock; a spec that drives the raw mouse at a row's box should check the hit target, as 17b now does.Checklist