Skip to content

[Douze] E2E: Playwright alternatives - #2156

Draft
Sébastien Ros (sebastienros) wants to merge 105 commits into
mainfrom
sebros/e2e-playwright-alternatives-3e7f32
Draft

Sébastien Ros (sebastienros) wants to merge 105 commits into
mainfrom
sebros/e2e-playwright-alternatives-3e7f32

Conversation

@sebastienros

@sebastienros Sébastien Ros (sebastienros) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Draft. This PR evaluates alternatives to Playwright and its Chromium for the E2E suite of the Douze build (#2151), aiming at a faster CI. It adds prototype jobs only. The existing E2E jobs and the frontend gate are unchanged, and the prototype jobs do not gate.

Recommendation

Keep Playwright with Chromium, and stop recording trace and video for every test. The configuration records both for every test (retain-on-failure) and deletes them for passing tests. On the 74-test slice, that recording costs 1.6 to 2 times the test time: 168 to 213 s with recording, 104 to 106 s without, over two CI runs on the same runner type. Recording on the first retry instead (trace: 'on-first-retry', video: 'on-first-retry'; CI already retries twice) keeps a trace and a video for every failure. If the full suite gains as much as the slice, each E2E leg drops from 8 to 18 min to roughly 4 to 9 min. That is a one-line change in playwright.config.mjs, now on main (#2159). In run 43748981211, which has it, the slice took 117 s as configured and 126 s with recording turned off entirely.

None of the alternative browsers can run this suite. They pass the tests that only make requests or read the DOM, and fail most tests that read layout, computed styles or visibility. Their speed does not make up for it. Lightpanda is about as fast as Chromium per test. Over CDP, Jint.Browser 5.0.0-preview.2457 (the build with the performance update) is 4 times slower than Chromium without recording, against 7 times for preview.2441. Preview.2469, which fixes the five issues filed from this suite, passes the same 48 of 74 tests and is 6 times slower than Chromium over CDP, because its axe scan now runs until the 180 s test timeout instead of failing after 5 s. Preview.2503, with the fixes for the axe-core costs and two layout regressions found on the way, passes 51 of 74 and is 3.3 times slower than Chromium over CDP (478 s against 145 s); the axe test still runs into its 180 s timeout on CI. Kitesurf is 11 times slower. In process, Jint.Browser 5.0.0-preview.2503 runs the subset's 8 behavior tests 2 to 3.4 times slower than Chromium (40 s against 12 to 20 s; 58 s with preview.2469), with the port on its Page API: init scripts, routing, role locators and native URL and focus waits. Not downloading Chromium saves about 20 s per leg (18 to 23 s measured), 2 to 4 % of a leg.

Candidates

Candidate What it is License, maturity Layout and CSS JavaScript Accessibility tree Driven by
Playwright + Chromium (today) Chromium headless shell, launched by Playwright Apache 2.0, mature Blink: real layout and computed styles V8 Full Playwright test
Puppeteer Another CDP client for the same Chromium (Google) Apache 2.0, mature Same as Chromium Same Same Its own API, with no test runner, fixtures, sharding or retries
Kitesurf (Cloudflare, open-sourced on 13 November) A browser engine that runs in Cloudflare Workers (V8 isolates). Locally it runs under wrangler dev (workerd). Apache 2.0, "experimental", no releases; a Bazel build of about 14 min on a 4-core runner Blitz (layout), Stylo (Firefox's CSS engine), Parley and Vello. Real layout, "not pixel-perfect". Boa, compiled to WebAssembly Accessibility.* implemented CDP: Puppeteer, Playwright connectOverCDP
Jint.Browser A .NET browser in one process: Jint.HtmlParser (DOM, CSSOM) and the Jint interpreter BSD-2-Clause, Jint 5 preview (preview feed only) No layout: a synthetic flat box model ("text does not wrap and elements are never truly side by side"), cascade without rendering, no screenshots Jint (interpreted ECMAScript) Yes Its own .NET API in process, or CDP (jint-browser serve)
Lightpanda (other option) A headless browser in Zig with a CDP server AGPL-3.0, 1.0.4, release binaries No visual layout (geometry approximated); stylesheets only with --load-resources stylesheet V8 Yes CDP

Puppeteer was not prototyped: it drives the same Chromium over the same CDP, so it cannot be faster per test, and the suite would lose Playwright test's runner, auto-waiting expect, sharding, retries and reports. The chromium-cdp entry measures what a CDP client costs on its own.

What the suite needs

Across the 49 spec files: getBoundingClientRect or boundingBox in 23 files (293 uses), getComputedStyle or toHaveCSS in 23 (397), page.route in 20, addInitScript in 26, AxeBuilder in 13, the keyboard in 18, the mouse or hover in 12, setViewportSize in 18, emulateMedia in 13, Document Picture-in-Picture in 6. Almost every spec also uses toBeVisible or toBeHidden, which need real boxes. Only 6 spec files need neither layout nor styles.

Need Chromium (Playwright or Puppeteer) Kitesurf Jint.Browser Lightpanda
Geometry, computed styles Yes Partial (real layout, gaps) No (synthetic boxes, cascade only) No
Keyboard, mouse Yes Yes (Input.*) Yes (synthetic hit testing) Yes
Viewport and device emulation Yes Emulation.setDeviceMetricsOverride Viewport per context and resize, touch, media emulation Partial
page.route Yes Fetch.* Fetch.* (CDP) or RouteAsync (API) Fetch.*
axe-core Yes Runs, but needs layout for contrast and visibility Runs, but no layout Runs, but no layout
Document Picture-in-Picture Yes (channel: 'chromium') No No No
Screenshots Yes Yes No No
Traces, video Yes No No No

Measurements

Runner ubuntu-latest (4 vCPU). The frontend-dist artifact is served by StaticHost. desktop-chromium project, 2 workers, no retries. Prototype jobs from runs 43624160556, 43627901380 (xUnit), 43630006685 (both Jint.Browser jobs on preview.2441 from the preview feed) 43673861979 (every job again, with Jint.Browser preview.2457) 43748981211 (every job again, with Jint.Browser preview.2469 and trace and video on the first retry on main) and 43929464693 (every job again, with Jint.Browser preview.2503; Kitesurf off).

Slice: analytics-scripts, dev-browse, cookie-dialog-layout, og-metadata (74 tests)

Browser Passed Browser provisioning Tests (wall) Sum of test durations
Playwright + Chromium, as configured (trace and video) 74 16.7 s (download + install) 213 s (168 s in a second run) 337 s
Playwright + Chromium, no trace or video 74 18.0 s (download + install) 104 s (106 s in a second run) 129 s
Playwright connectOverCDP to Chromium headless shell 73 (1 timing failure¹) 17.5 s (download + install) 115 s 146 s
Playwright connectOverCDP to Lightpanda 1.0.4 42 1.8 s (binary download) 210 s 312 s
Playwright connectOverCDP to Jint.Browser (jint-browser serve, 5.0.0-preview.2441 from the preview feed) 40 7.8 s (dotnet tool install²) 743 s 1,342 s
Playwright connectOverCDP to Jint.Browser (jint-browser serve, 5.0.0-preview.2457 from the preview feed) 48 5.2 s (dotnet tool install²) 463 s 822 s
Playwright connectOverCDP to Jint.Browser (jint-browser serve, 5.0.0-preview.2469 from the preview feed) 48 5.5 s (dotnet tool install²) 719 s 1,179 s
Playwright connectOverCDP to Jint.Browser (jint-browser serve, 5.0.0-preview.2503 from the preview feed) 51 6.9 s (dotnet tool install²) 478 s 696 s
Playwright connectOverCDP to Kitesurf 39 844 s (clone and Bazel build) 1,174 s 1,948 s

¹ A reduced-motion scroll-behavior assertion in dev-browse that passed in the local run.
² The native AOT Jint.Browser.Tool from the Jint preview feed (f.feedz.io/sebastienros/jint), the same package version as the xUnit port. A first run that built the tool from the same commit took 53.9 s to provision and 660 s for the tests, with the same 40 passes.

Per spec, every engine passes analytics-scripts (3) and og-metadata (23): they only make requests, and run in under 2 s anywhere. cookie-dialog-layout, which reads geometry and colors, fails on every alternative (0 or 1 of 9), and Jint.Browser and Kitesurf time out on it. Of the 39 dev-browse tests, Lightpanda passes 15, Jint.Browser 14 (19 with preview.2457) and Kitesurf 13. The failures are in toHaveCSS, geometry, visibility, focus and timeouts. With preview.2457, Jint.Browser also passes 3 of the 9 cookie-dialog-layout tests. Of the 20 dev-browse tests that still fail on preview.2457, 9 need real layout. The other 11 come from five engine gaps, filed with minimal repros: unresolved lengths in computed style (sebastienros/jint#4615), the mask shorthand (sebastienros/jint#4616), animation longhands (sebastienros/jint#4617), Tab order onto hidden controls (sebastienros/jint#4618), and the 5 s task ceiling reported as "Execution context was destroyed", which every axe scan of the browse page hits; axe takes 234 s there against 3.5 s in Chromium (sebastienros/jint#4619).

In run 43673861979, the Chromium rows on the same runners took 178 s (as configured), 114 s (no recording) and 138 s (over CDP). Jint.Browser preview.2457 over CDP (463 s) is 4.1 times the time of Chromium without recording; preview.2441 was 7.1 times.

In run 43748981211, the Chromium rows took 117 s (as configured, now recording on the first retry), 126 s (no recording) and 112 s (over CDP), all 74 passed. Jint.Browser preview.2469 took 719 s, 6.4 times Chromium over CDP. Lightpanda took 213 s with 42 passes, and Kitesurf 1,164 s with 39.

With preview.2469. The symptoms of sebastienros/jint#4615 to #4619 are gone: computed lengths resolve, mask expands, animation longhands have their initial values, Tab skips hidden controls, and the axe scan is no longer stopped after 5 s. Jint.Browser passes 20 of the 39 dev-browse tests (19 with preview.2457) and 2 of the 9 cookie-dialog-layout tests. Most of the tests those fixes unblocked now fail later, on layout:

  • 15 dev-browse tests need real layout. These are the 9 from before, plus the breadcrumbs ×3 (the image header is not 16:9), every dropdown shows responsive options (an option is 80 px high, at most 36 expected), and pagination with both motion settings (the breadcrumb ends 88 px from the top after the scroll).
  • date and title directions persist: a reload over CDP adds a history entry (history.length goes from 3 to 4; Chromium keeps it), so goBack returns to the reloaded URL.
  • card title underlines animate: :focus-visible doesn't match after a script focus (element.focus() or locator.focus()). Chromium matches it unless the last focus came from the mouse.
  • CSS artwork texture: with forcedColors: 'active', matchMedia matches, but a gradient background-image keeps its value. Chromium computes it as none in forced-colors mode, and keeps url() images.
  • filters and cards remain accessible: axe now runs without the ceiling, and the test's four scans exceed its 180 s timeout.
  • cookie-dialog-layout: 2 tests fail on layout (an inset of 15.8 px, at least 16 expected). The other 5 time out after 60 s in page.addStyleTag on CI only, as they already did with preview.2457. Locally the spec takes 2.8 s with two workers, with no timeout.

The history, :focus-visible and forced-colors gaps reproduce on minimal pages.

With preview.2503 (commit b0dd1a2). It carries the fixes for the three gaps above (sebastienros/jint#4642, sebastienros/jint#4643, sebastienros/jint#4644), the axe-core costs (sebastienros/jint#4647, sebastienros/jint#4649, sebastienros/jint#4661, sebastienros/jint#4662) and two layout regressions found on the way (sebastienros/jint#4654, sebastienros/jint#4655). In run 43929464693, Jint.Browser over CDP passes 51 of 74 in 478 s, 3.3 times Chromium over CDP (145 s) and 4.5 times Chromium without recording (105 s); preview.2469 took 719 s for 48. The 23 failures:

Locally (M-series Mac, 2 workers), against the same shipped native AOT tool, the slice passes 55 of 74 in 4.9 min, the axe test and the three color tests included.

Locally (M-series Mac, 2 workers), the same slice took 67.1 s as configured, 32.1 s without recording, 35.6 s over CDP to Chromium, and 395 s over CDP to Jint.Browser (40 passed).

Jint.Browser in process: 34 tests ported to xUnit

tests/E2E.Jint ports analytics-scripts, og-metadata and the 8 dev-browse tests that check behavior rather than layout (URL state, filters, pagination, history, blog coverage) to xUnit on Jint.Browser's .NET API (Jint.Browser 5.0.0-preview.2441), in process, without CDP. The request-only tests use HttpClient and Jint.HtmlParser in place of rehype. The Playwright rows run the same 34 tests (SUBSET=1).

Run 43627901380. In each row, StaticHost is already built.

Runner (34 tests, 2 in parallel) Passed Browser provisioning StaticHost start + tests Leg total Sum of test durations The 8 dev-browse tests
Playwright + Chromium, as configured 34 20.5 s (Chromium download + install) 46.4 s 66.9 s 35.2 s —
Playwright + Chromium, no trace or video 34 18.3 s (Chromium download + install) 41.2 s 59.5 s 19.5 s 16.6 s
xUnit + Jint.Browser in process (Jint.Browser 5.0.0-preview.2441 restored from the preview feed) 34 9 s (NuGet restore + build, no browser; 7 s in a second run) 28 s + 43 s (41 s in a second run) 80 s 78.5 s 76.5 s

Run 43673861979, with Jint.Browser 5.0.0-preview.2457, and the Playwright rows run again on the same runners:

Runner (34 tests, 2 in parallel) Passed Browser provisioning StaticHost start + tests Leg total Sum of test durations The 8 dev-browse tests
Playwright + Chromium, as configured 34 16.0 s 44.5 s 60.5 s 33.1 s 31.4 s
Playwright + Chromium, no trace or video 34 14.0 s 31.4 s 45.4 s 12.1 s 10.4 s
xUnit + Jint.Browser in process, preview.2457 34 6 s (NuGet restore + build) 26 s + 36.4 s 68 s 62.8 s 61 s

From preview.2441 to preview.2457, the in-process dotnet test drops from 43 s to 36 s and the 8 dev-browse tests from 76.5 s to 61 s (−20 %). Chromium without recording is still about 6 times faster on those 8 tests in the same run, and its leg is 23 s shorter.

With Jint.Browser's native waits. The port above checked each assertion with a JavaScript predicate polled through WaitForAsync. Commits 310bbe1 and c049fd0 switch it to the native queries and waits (WaitForCountAsync, WaitForVisibleAsync, WaitForTextAsync, WaitForSelectorAsync, CountAsync) and to the site's data-attribute selectors. Only the URL, focus and list-equality checks stay in JavaScript, because no native API exists for them yet (sebastienros/jint#4626). The tasks also get a 30 s budget (MaxTaskDuration): a click on the browse page scrolls its target into view through the synthetic layout, and that ran past the 5 s default on CI. Run 43685884521:

Runner (34 tests, 2 in parallel) Passed Browser provisioning Tests The 8 dev-browse tests
Playwright + Chromium, as configured 34 15.6 s 32.6 s (StaticHost start included) 11.9 s
Playwright + Chromium, no trace or video 34 21.5 s 45.1 s (StaticHost start included) 17.0 s
xUnit + Jint.Browser 2457, native waits 34 8 s (NuGet restore + build) 27 s StaticHost start + 40.6 s 69 s

That is no faster than the JavaScript-predicate port (36.4 s and 61 s in the previous run; locally 25–28 s against 24–29 s). On the browse page a native count takes 1 ms against 140–300 ms in JavaScript, but a navigation takes 0.9–2.8 s and a pagination click 1.8–2.8 s. Both run the site's own scripts, which is where the time goes. The in-process APIs the suite would need beyond these are filed as sebastienros/jint#4620 to #4626.

With preview.2469 and the Page API. Commit 3bc2147 moves the port to the members preview.2469 adds:

  • context.AddInitScriptAsync and context.RouteAsync(...).AbortAsync() reproduce emulateConsentNotRequiredRegion as the Playwright helper does: they set the consent flag before the site's scripts run and fail WCP requests. They replace the localStorage seed page and the same-origin UrlFilter.
  • Role locators (role=button[name="Next"s], .browse-empty >> role=button[...]) replace the attribute-and-text helpers, as getByRole(..., { exact: true }) does.
  • WaitForUrlAsync (a Regex or a predicate), WaitForFocusAsync and WaitForAttributeAsync replace the last JavaScript predicates. A list comparison now polls TextsAsync, because there is no native list wait.

Run 43748981211:

Runner (34 tests, 2 in parallel) Passed Browser provisioning StaticHost start + tests Leg total Sum of test durations The 8 dev-browse tests
Playwright + Chromium, as configured (trace and video on the first retry, #2159) 34 16.2 s 34.8 s 51 s 16.0 s 14.0 s
Playwright + Chromium, no trace or video 34 22.2 s 42.8 s 65 s 19.5 s 16.8 s
xUnit + Jint.Browser 5.0.0-preview.2469 in process, Page API 34 6 s (NuGet restore + build) 22 s + 32.5 s 61 s 59.1 s 58 s

dotnet test took 32.5 s (40.6 s in the previous run) and the 8 dev-browse tests 58 s (69 s). Two earlier runs with the same engine differed by as much (36.4 s and 40.6 s), so this is not clearly a gain. Locally, served by python -m http.server, every combination took 19 to 21 s: the previous port on preview.2457 or on preview.2469, the new port, the new port with the same-origin UrlFilter restored, and the new port with TieredPGO off, as the docs' performance page suggests. The Page API makes the port read like the Playwright spec, but doesn't change its speed. Chromium still runs the 8 dev-browse tests about 4 times faster.

Locally the same 34 tests took 7.3 s (Playwright as configured), 4.2 s (no recording) and 20 s (Jint.Browser in process), with StaticHost already running.

With preview.2503. Run 43929464693:

Runner (34 tests, 2 in parallel) Passed Browser provisioning StaticHost start + tests Leg total Sum of test durations The 8 dev-browse tests
Playwright + Chromium, as configured 34 19.2 s 45.7 s 65 s 23.3 s 20.1 s
Playwright + Chromium, no trace or video 34 16.0 s 32.7 s 49 s 13.6 s 11.8 s
xUnit + Jint.Browser 5.0.0-preview.2503 in process, Page API 34 6.7 s (NuGet restore + build) 26 s + 23.5 s 56 s 41.7 s 39.9 s

dotnet test took 23.5 s (32.5 s with preview.2469) and the 8 dev-browse tests 40 s (58 s). Locally the port takes 13 s, against 22 to 24 s with preview.2469 on the same machine.

The in-process API removes the CDP round trips: on CI, these dev-browse tests take 2 to 15 s each, against 4 to 45 s through jint-browser serve. Chromium still runs them about 4.6 times faster, because Jint interprets the site's JavaScript. No browser download (about 20 s) does not make up for that, even on this subset, which leaves out every layout test. With preview.2441 to 2457, the port had to replace parts of the suite: getByRole with selector and text helpers, addInitScript (then exposed only over CDP) with a localStorage seed, and page.route with a context UrlFilter. Preview.2469 has a native equivalent for each, and the port uses them.

Where the time goes in a real leg

From the same run, the six E2E legs spend 18 to 23 s installing Chromium and 419 to 988 s in playwright test. The download is 2 to 4 % of a leg. The shards are also unbalanced (desktop 905 s against 614 s, tablet 988 s against 537 s); balancing them is separate work.

Migration effort

Option Effort Expected gain
Record trace and video on the first retry One line in playwright.config.mjs, done in #2159 About 2× on test time (measured on the slice)
Puppeteer Rewrite the 49 specs and rebuild runner, fixtures, sharding, retries, reports None: same browser, same protocol
Lightpanda over CDP Config only (tests/cdp), but more than half of the layout specs fail, permanently (no layout by design) Saves about 15 s of download per leg. Tests are no faster than Chromium without recording.
Jint.Browser over CDP Config only (the tool installs from the preview feed in 5 to 8 s), same failures; 3.3× slower with preview.2503 (4× with preview.2457, 7× with preview.2441, 6× with preview.2469 as the axe scan runs to its timeout) Negative
Jint.Browser in process (xUnit) Rewrite the specs in C#; since preview.2469 it has role locators, init scripts, routing and native waits, but no auto-waiting expect; the layout, style, axe and PiP specs cannot be ported Negative on wall time today; no browser download
Kitesurf Config only, a 14-minute Bazel build per job (or a cached build of about 6.6 GB), KITESURF_ALLOW_PRIVATE_NETWORK for loopback, and most layout specs fail Negative: 5× slower than Chromium

What this PR adds

  • src/frontend/playwright.cdp.config.mjs, tsconfig.cdp.json and tests/cdp/playwright-test-shim.ts: the specs' @playwright/test import resolves to a shim that connects each worker to CDP_ENDPOINT over connectOverCDP, so the specs run unchanged against any CDP browser. Without CDP_ENDPOINT, Playwright launches Chromium as usual, without trace or video.
  • src/frontend/tests/cdp/run-engine.sh and summarize.mjs: provision a browser, run the slice, and write per-spec results and timings to the job summary.
  • tests/E2E.Jint: the xUnit port on Jint.Browser.
  • src/site/assets/css/components.css: the reduced-motion reveal fix of Fix three flaky E2E tests and the reduced-motion reveal offset behind one of them #2162. The merge from main brought it only to HomePage.astro, which the Douze build doesn't compile, so the homepage reduced-motion tests failed in shard 2 of every project. With the port, all six E2E jobs pass (43752967437). sebros/douze-migration needs the same port.
  • Jobs in frontend-build.yml: E2E alternative (…) for each engine, E2E alternative (Jint.Browser in-process xUnit) and E2E alternative (kitesurf). They need build and nothing depends on them. The Kitesurf job is disabled (if: false) since 1eb5948: it takes about 30 min per run, and its numbers above come from the earlier runs.

…ipted converter, 647 files, 35 in the manual queue)
…tyles, client scripts, sidebar and i18n data, output templates
…ndex, Liquid partials of the API components, Markdown twins
…t-id meta from build info, redirect template permalink guard, sidebar buttons styled
…ale home pages; home pages without heading anchors; root page in the collections
…ollers keep typed text in the url; no fade on targeted members
…yles on API pages, API pages see the collections
…codes (live videos, gallery, catalog, sessions, journeys, runtime choices)
…and script, readable titles, file tree icons, tab selection on server-rendered lists, dropped component styles restored, icons passthrough
…-alternatives-3e7f32

# Conflicts:
#	.github/workflows/frontend-build.yml
…Unit port

Init script and routing for the consent state (as emulateConsentNotRequiredRegion), role locators
for getByRole, and native URL, focus and attribute waits in place of JavaScript predicates.
The merge with main brought the fix to HomePage.astro, but the Douze build compiles its own copy of
that rule in src/site/assets/css/components.css, so the homepage reduced-motion tests failed in
shard 2/2 of every project.
…s layout regressions and axe-core's selector costs

2503 carries sebastienros/jint#4654, #4655, #4661 and #4662. Locally, the xUnit port passes 34/34
in 13 s (22 s on 2469), and the CDP slice against the 2503 native AOT tool passes 55/74 in 4.9 min,
the axe test included; the 19 failures all need real layout.
…-alternatives-3e7f32

# Conflicts:
#	.github/workflows/ci.yml
#	.github/workflows/frontend-build.yml
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants