Repository navigation
[Douze] E2E: Playwright alternatives - #2156
Draft
Sébastien Ros (sebastienros) wants to merge 105 commits into
Draft
Sébastien Ros (sebastienros) wants to merge 105 commits into
Sébastien Ros (sebastienros) wants to merge 105 commits into
Conversation
… root, step order, outputs, dependencies
…he site's utilities
…d, libvips; drop UnoCSS
…ipted converter, 647 files, 35 in the manual queue)
…slash as a Douze worker step
…tyles, client scripts, sidebar and i18n data, output templates
…rkdown, default layout, root pages as HTML
…laywright config for the Douze build
…ndex, Liquid partials of the API components, Markdown twins
…only for locales that have them
…h an installed Chrome build
…t-id meta from build info, redirect template permalink guard, sidebar buttons styled
…ale home pages; home pages without heading anchors; root page in the collections
…ollers keep typed text in the url; no fade on targeted members
…yles on API pages, API pages see the collections
…codes (live videos, gallery, catalog, sessions, journeys, runtime choices)
…h files reached layout twice)
…and script, readable titles, file tree icons, tab selection on server-rendered lists, dropped component styles restored, icons passthrough
This was referenced Nov 15, 2026
Closed
…-alternatives-3e7f32 # Conflicts: # .github/workflows/frontend-build.yml
…wse page can exceed 5 s on CI
…Unit port Init script and routing for the consent state (as emulateConsentNotRequiredRegion), role locators for getByRole, and native URL, focus and attribute waits in place of JavaScript predicates.
The merge with main brought the fix to HomePage.astro, but the Douze build compiles its own copy of that rule in src/site/assets/css/components.css, so the homepage reduced-motion tests failed in shard 2/2 of every project.
This was referenced Nov 15, 2026
…s layout regressions and axe-core's selector costs 2503 carries sebastienros/jint#4654, #4655, #4661 and #4662. Locally, the xUnit port passes 34/34 in 13 s (22 s on 2469), and the CDP slice against the 2503 native AOT tool passes 55/74 in 4.9 min, the axe test included; the 19 failures all need real layout.
…-alternatives-3e7f32 # Conflicts: # .github/workflows/ci.yml # .github/workflows/frontend-build.yml
This was referenced Nov 16, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft. This PR evaluates alternatives to Playwright and its Chromium for the E2E suite of the Douze build (#2151), aiming at a faster CI. It adds prototype jobs only. The existing E2E jobs and the frontend gate are unchanged, and the prototype jobs do not gate.
Recommendation
Keep Playwright with Chromium, and stop recording trace and video for every test. The configuration records both for every test (
retain-on-failure) and deletes them for passing tests. On the 74-test slice, that recording costs 1.6 to 2 times the test time: 168 to 213 s with recording, 104 to 106 s without, over two CI runs on the same runner type. Recording on the first retry instead (trace: 'on-first-retry',video: 'on-first-retry'; CI already retries twice) keeps a trace and a video for every failure. If the full suite gains as much as the slice, each E2E leg drops from 8 to 18 min to roughly 4 to 9 min. That is a one-line change inplaywright.config.mjs, now onmain(#2159). In run 43748981211, which has it, the slice took 117 s as configured and 126 s with recording turned off entirely.None of the alternative browsers can run this suite. They pass the tests that only make requests or read the DOM, and fail most tests that read layout, computed styles or visibility. Their speed does not make up for it. Lightpanda is about as fast as Chromium per test. Over CDP, Jint.Browser 5.0.0-preview.2457 (the build with the performance update) is 4 times slower than Chromium without recording, against 7 times for preview.2441. Preview.2469, which fixes the five issues filed from this suite, passes the same 48 of 74 tests and is 6 times slower than Chromium over CDP, because its axe scan now runs until the 180 s test timeout instead of failing after 5 s. Preview.2503, with the fixes for the axe-core costs and two layout regressions found on the way, passes 51 of 74 and is 3.3 times slower than Chromium over CDP (478 s against 145 s); the axe test still runs into its 180 s timeout on CI. Kitesurf is 11 times slower. In process, Jint.Browser 5.0.0-preview.2503 runs the subset's 8 behavior tests 2 to 3.4 times slower than Chromium (40 s against 12 to 20 s; 58 s with preview.2469), with the port on its
PageAPI: init scripts, routing, role locators and native URL and focus waits. Not downloading Chromium saves about 20 s per leg (18 to 23 s measured), 2 to 4 % of a leg.Candidates
wrangler dev(workerd).Accessibility.*implementedconnectOverCDPjint-browser serve)--load-resources stylesheetPuppeteer was not prototyped: it drives the same Chromium over the same CDP, so it cannot be faster per test, and the suite would lose Playwright test's runner, auto-waiting
expect, sharding, retries and reports. Thechromium-cdpentry measures what a CDP client costs on its own.What the suite needs
Across the 49 spec files:
getBoundingClientRectorboundingBoxin 23 files (293 uses),getComputedStyleortoHaveCSSin 23 (397),page.routein 20,addInitScriptin 26,AxeBuilderin 13, the keyboard in 18, the mouse or hover in 12,setViewportSizein 18,emulateMediain 13, Document Picture-in-Picture in 6. Almost every spec also usestoBeVisibleortoBeHidden, which need real boxes. Only 6 spec files need neither layout nor styles.Input.*)Emulation.setDeviceMetricsOverridepage.routeFetch.*Fetch.*(CDP) orRouteAsync(API)Fetch.*channel: 'chromium')Measurements
Runner
ubuntu-latest(4 vCPU). Thefrontend-distartifact is served by StaticHost. desktop-chromium project, 2 workers, no retries. Prototype jobs from runs 43624160556, 43627901380 (xUnit), 43630006685 (both Jint.Browser jobs on preview.2441 from the preview feed) 43673861979 (every job again, with Jint.Browser preview.2457) 43748981211 (every job again, with Jint.Browser preview.2469 and trace and video on the first retry onmain) and 43929464693 (every job again, with Jint.Browser preview.2503; Kitesurf off).Slice:
analytics-scripts,dev-browse,cookie-dialog-layout,og-metadata(74 tests)connectOverCDPto Chromium headless shellconnectOverCDPto Lightpanda 1.0.4connectOverCDPto Jint.Browser (jint-browser serve, 5.0.0-preview.2441 from the preview feed)dotnet tool install²)connectOverCDPto Jint.Browser (jint-browser serve, 5.0.0-preview.2457 from the preview feed)dotnet tool install²)connectOverCDPto Jint.Browser (jint-browser serve, 5.0.0-preview.2469 from the preview feed)dotnet tool install²)connectOverCDPto Jint.Browser (jint-browser serve, 5.0.0-preview.2503 from the preview feed)dotnet tool install²)connectOverCDPto Kitesurf¹ A reduced-motion
scroll-behaviorassertion indev-browsethat passed in the local run.² The native AOT
Jint.Browser.Toolfrom the Jint preview feed (f.feedz.io/sebastienros/jint), the same package version as the xUnit port. A first run that built the tool from the same commit took 53.9 s to provision and 660 s for the tests, with the same 40 passes.Per spec, every engine passes
analytics-scripts(3) andog-metadata(23): they only make requests, and run in under 2 s anywhere.cookie-dialog-layout, which reads geometry and colors, fails on every alternative (0 or 1 of 9), and Jint.Browser and Kitesurf time out on it. Of the 39dev-browsetests, Lightpanda passes 15, Jint.Browser 14 (19 with preview.2457) and Kitesurf 13. The failures are intoHaveCSS, geometry, visibility, focus and timeouts. With preview.2457, Jint.Browser also passes 3 of the 9cookie-dialog-layouttests. Of the 20dev-browsetests that still fail on preview.2457, 9 need real layout. The other 11 come from five engine gaps, filed with minimal repros: unresolved lengths in computed style (sebastienros/jint#4615), themaskshorthand (sebastienros/jint#4616),animationlonghands (sebastienros/jint#4617), Tab order onto hidden controls (sebastienros/jint#4618), and the 5 s task ceiling reported as "Execution context was destroyed", which every axe scan of the browse page hits; axe takes 234 s there against 3.5 s in Chromium (sebastienros/jint#4619).In run 43673861979, the Chromium rows on the same runners took 178 s (as configured), 114 s (no recording) and 138 s (over CDP). Jint.Browser preview.2457 over CDP (463 s) is 4.1 times the time of Chromium without recording; preview.2441 was 7.1 times.
In run 43748981211, the Chromium rows took 117 s (as configured, now recording on the first retry), 126 s (no recording) and 112 s (over CDP), all 74 passed. Jint.Browser preview.2469 took 719 s, 6.4 times Chromium over CDP. Lightpanda took 213 s with 42 passes, and Kitesurf 1,164 s with 39.
With preview.2469. The symptoms of sebastienros/jint#4615 to #4619 are gone: computed lengths resolve,
maskexpands,animationlonghands have their initial values, Tab skips hidden controls, and the axe scan is no longer stopped after 5 s. Jint.Browser passes 20 of the 39dev-browsetests (19 with preview.2457) and 2 of the 9cookie-dialog-layouttests. Most of the tests those fixes unblocked now fail later, on layout:dev-browsetests need real layout. These are the 9 from before, plus the breadcrumbs ×3 (the image header is not 16:9), every dropdown shows responsive options (an option is 80 px high, at most 36 expected), and pagination with both motion settings (the breadcrumb ends 88 px from the top after the scroll).history.lengthgoes from 3 to 4; Chromium keeps it), sogoBackreturns to the reloaded URL.:focus-visibledoesn't match after a script focus (element.focus()orlocator.focus()). Chromium matches it unless the last focus came from the mouse.forcedColors: 'active',matchMediamatches, but a gradientbackground-imagekeeps its value. Chromium computes it asnonein forced-colors mode, and keepsurl()images.cookie-dialog-layout: 2 tests fail on layout (an inset of 15.8 px, at least 16 expected). The other 5 time out after 60 s inpage.addStyleTagon CI only, as they already did with preview.2457. Locally the spec takes 2.8 s with two workers, with no timeout.The history,
:focus-visibleand forced-colors gaps reproduce on minimal pages.With preview.2503 (commit b0dd1a2). It carries the fixes for the three gaps above (sebastienros/jint#4642, sebastienros/jint#4643, sebastienros/jint#4644), the axe-core costs (sebastienros/jint#4647, sebastienros/jint#4649, sebastienros/jint#4661, sebastienros/jint#4662) and two layout regressions found on the way (sebastienros/jint#4654, sebastienros/jint#4655). In run 43929464693, Jint.Browser over CDP passes 51 of 74 in 478 s, 3.3 times Chromium over CDP (145 s) and 4.5 times Chromium without recording (105 s); preview.2469 took 719 s for 48. The 23 failures:
dev-browseand 4cookie-dialog-layouttests that need real layout;cookie-dialog-layoutcolor tests that time out after 60 s, on CI only, as with every earlier preview. Locally, a new page in a fresh context sometimes never reachesloadover CDP (0 to 2 times in 60 tries, on every build since before preview.2469), which is the likely cause.Locally (M-series Mac, 2 workers), against the same shipped native AOT tool, the slice passes 55 of 74 in 4.9 min, the axe test and the three color tests included.
Locally (M-series Mac, 2 workers), the same slice took 67.1 s as configured, 32.1 s without recording, 35.6 s over CDP to Chromium, and 395 s over CDP to Jint.Browser (40 passed).
Jint.Browser in process: 34 tests ported to xUnit
tests/E2E.Jintportsanalytics-scripts,og-metadataand the 8dev-browsetests that check behavior rather than layout (URL state, filters, pagination, history, blog coverage) to xUnit on Jint.Browser's .NET API (Jint.Browser5.0.0-preview.2441), in process, without CDP. The request-only tests useHttpClientand Jint.HtmlParser in place of rehype. The Playwright rows run the same 34 tests (SUBSET=1).Run 43627901380. In each row, StaticHost is already built.
dev-browsetestsJint.Browser5.0.0-preview.2441 restored from the preview feed)Run 43673861979, with Jint.Browser 5.0.0-preview.2457, and the Playwright rows run again on the same runners:
dev-browsetestsFrom preview.2441 to preview.2457, the in-process
dotnet testdrops from 43 s to 36 s and the 8dev-browsetests from 76.5 s to 61 s (−20 %). Chromium without recording is still about 6 times faster on those 8 tests in the same run, and its leg is 23 s shorter.With Jint.Browser's native waits. The port above checked each assertion with a JavaScript predicate polled through
WaitForAsync. Commits 310bbe1 and c049fd0 switch it to the native queries and waits (WaitForCountAsync,WaitForVisibleAsync,WaitForTextAsync,WaitForSelectorAsync,CountAsync) and to the site's data-attribute selectors. Only the URL, focus and list-equality checks stay in JavaScript, because no native API exists for them yet (sebastienros/jint#4626). The tasks also get a 30 s budget (MaxTaskDuration): a click on the browse page scrolls its target into view through the synthetic layout, and that ran past the 5 s default on CI. Run 43685884521:dev-browsetestsThat is no faster than the JavaScript-predicate port (36.4 s and 61 s in the previous run; locally 25–28 s against 24–29 s). On the browse page a native count takes 1 ms against 140–300 ms in JavaScript, but a navigation takes 0.9–2.8 s and a pagination click 1.8–2.8 s. Both run the site's own scripts, which is where the time goes. The in-process APIs the suite would need beyond these are filed as sebastienros/jint#4620 to #4626.
With preview.2469 and the
PageAPI. Commit 3bc2147 moves the port to the members preview.2469 adds:context.AddInitScriptAsyncandcontext.RouteAsync(...).AbortAsync()reproduceemulateConsentNotRequiredRegionas the Playwright helper does: they set the consent flag before the site's scripts run and fail WCP requests. They replace thelocalStorageseed page and the same-originUrlFilter.role=button[name="Next"s],.browse-empty >> role=button[...]) replace the attribute-and-text helpers, asgetByRole(..., { exact: true })does.WaitForUrlAsync(aRegexor a predicate),WaitForFocusAsyncandWaitForAttributeAsyncreplace the last JavaScript predicates. A list comparison now pollsTextsAsync, because there is no native list wait.Run 43748981211:
dev-browsetestsPageAPIdotnet testtook 32.5 s (40.6 s in the previous run) and the 8dev-browsetests 58 s (69 s). Two earlier runs with the same engine differed by as much (36.4 s and 40.6 s), so this is not clearly a gain. Locally, served bypython -m http.server, every combination took 19 to 21 s: the previous port on preview.2457 or on preview.2469, the new port, the new port with the same-originUrlFilterrestored, and the new port withTieredPGOoff, as the docs' performance page suggests. ThePageAPI makes the port read like the Playwright spec, but doesn't change its speed. Chromium still runs the 8dev-browsetests about 4 times faster.Locally the same 34 tests took 7.3 s (Playwright as configured), 4.2 s (no recording) and 20 s (Jint.Browser in process), with StaticHost already running.
With preview.2503. Run 43929464693:
dev-browsetestsPageAPIdotnet testtook 23.5 s (32.5 s with preview.2469) and the 8dev-browsetests 40 s (58 s). Locally the port takes 13 s, against 22 to 24 s with preview.2469 on the same machine.The in-process API removes the CDP round trips: on CI, these
dev-browsetests take 2 to 15 s each, against 4 to 45 s throughjint-browser serve. Chromium still runs them about 4.6 times faster, because Jint interprets the site's JavaScript. No browser download (about 20 s) does not make up for that, even on this subset, which leaves out every layout test. With preview.2441 to 2457, the port had to replace parts of the suite:getByRolewith selector and text helpers,addInitScript(then exposed only over CDP) with alocalStorageseed, andpage.routewith a contextUrlFilter. Preview.2469 has a native equivalent for each, and the port uses them.Where the time goes in a real leg
From the same run, the six E2E legs spend 18 to 23 s installing Chromium and 419 to 988 s in
playwright test. The download is 2 to 4 % of a leg. The shards are also unbalanced (desktop 905 s against 614 s, tablet 988 s against 537 s); balancing them is separate work.Migration effort
playwright.config.mjs, done in #2159tests/cdp), but more than half of the layout specs fail, permanently (no layout by design)expect; the layout, style, axe and PiP specs cannot be portedKITESURF_ALLOW_PRIVATE_NETWORKfor loopback, and most layout specs failWhat this PR adds
src/frontend/playwright.cdp.config.mjs,tsconfig.cdp.jsonandtests/cdp/playwright-test-shim.ts: the specs'@playwright/testimport resolves to a shim that connects each worker toCDP_ENDPOINToverconnectOverCDP, so the specs run unchanged against any CDP browser. WithoutCDP_ENDPOINT, Playwright launches Chromium as usual, without trace or video.src/frontend/tests/cdp/run-engine.shandsummarize.mjs: provision a browser, run the slice, and write per-spec results and timings to the job summary.tests/E2E.Jint: the xUnit port on Jint.Browser.src/site/assets/css/components.css: the reduced-motion reveal fix of Fix three flaky E2E tests and the reduced-motion reveal offset behind one of them #2162. The merge frommainbrought it only toHomePage.astro, which the Douze build doesn't compile, so the homepage reduced-motion tests failed in shard 2 of every project. With the port, all six E2E jobs pass (43752967437).sebros/douze-migrationneeds the same port.frontend-build.yml:E2E alternative (…)for each engine,E2E alternative (Jint.Browser in-process xUnit)andE2E alternative (kitesurf). They needbuildand nothing depends on them. The Kitesurf job is disabled (if: false) since 1eb5948: it takes about 30 min per run, and its numbers above come from the earlier runs.