Repository navigation
[Investigation] Why CDP alternatives to Chromium fail the E2E slice - #2158
Draft
Sébastien Ros (sebastienros) wants to merge 101 commits into
Draft
Sébastien Ros (sebastienros) wants to merge 101 commits into
Sébastien Ros (sebastienros) wants to merge 101 commits into
Conversation
… root, step order, outputs, dependencies
…he site's utilities
…d, libvips; drop UnoCSS
…ipted converter, 647 files, 35 in the manual queue)
…slash as a Douze worker step
…tyles, client scripts, sidebar and i18n data, output templates
…rkdown, default layout, root pages as HTML
…laywright config for the Douze build
…ndex, Liquid partials of the API components, Markdown twins
…only for locales that have them
…h an installed Chrome build
…t-id meta from build info, redirect template permalink guard, sidebar buttons styled
…ale home pages; home pages without heading anchors; root page in the collections
…ollers keep typed text in the url; no fade on targeted members
…yles on API pages, API pages see the collections
…codes (live videos, gallery, catalog, sessions, journeys, runtime choices)
…h files reached layout twice)
…and script, readable titles, file tree icons, tab selection on server-rendered lists, dropped component styles restored, icons passthrough
- Escape the json attributes of the version filter and the asciinema player, so the API version dropdowns work. - Point the Markdown actions of fallback pages at the shown content's Markdown. - Give the locale homes the home Markdown twin, with links in their locale. - Keep the AspireConf clock and timezone choices for the tab's session. - Load ResourceBrowser.js as an async module, so the chrome's scripts don't wait for it.
The specs no longer expect Astro's client router (astro:page-load, swapped documents, state on window surviving a navigation): they check each page after its load, and the native transition and swap fallback variants are one test.
…ere()
The hub layout's `.dev-center :where(button, input, select) { font: inherit }` beat `.search-action` once bumped,
so the glossary's Clear filters button inherited 16px and pushed the search input on narrower fonts.
- site-search: the held Pagefind UI bundle can delay the load event, so wait for the parsed document. - route-style-navigation: name the properties that differ when a route doesn't match its fresh rendering. - dev-browse: the axe scans of the browse page need more than the default timeout on CI runners.
…undles ordered them HubLayout's global `.dev-center :where(button)` default came first there, so `.search-action` won their tie (the glossary's Clear filters button is 14px, not 16px).
The same 74 desktop tests run with Playwright as configured, Playwright without trace and video recording, and Playwright connected over CDP to Chromium, Lightpanda, Jint.Browser and Kitesurf. The jobs report to the summary and do not gate the frontend checks.
analytics-scripts, og-metadata and the behavioral dev-browse tests run in one .NET process, without CDP or a browser download. The prototype jobs run the same 34 tests with Playwright, with and without trace and video recording, so the CI times compare.
The CDP job now restores Jint.Browser.Tool 5.0.0-preview.2441, the native AOT tool of the same commit and the same package version as the xUnit port, so both Jint.Browser runs use the published packages.
…scope Lightpanda drops the connection on CDP messages over 1 MB, which the axe-core source exceeds, and allows one browser context per connection. Jint.Browser never completes page.setContent or page.addStyleTag. None of the three engines applies CSS @scope, where Douze puts half of the site's rules. The shim gains opt-in workarounds for each, and the jobs run each engine with them.
|
|
||
| function rootElement(source) { | ||
| let text = source.replace(/^---[\s\S]*?\n---/, ''); | ||
| text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, ''); |
|
|
||
| function rootElement(source) { | ||
| let text = source.replace(/^---[\s\S]*?\n---/, ''); | ||
| text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, ''); |
|
|
||
| function styleBlocks(source) { | ||
| let text = source.replace(/^---[\s\S]*?\n---/, (m) => ' '.repeat(m.length)); | ||
| text = text.replace(/<script\b[\s\S]*?<\/script>/g, (m) => ' '.repeat(m.length)); |
| if (!width || !height) return; | ||
|
|
||
| const theme = getCurrentTheme(); | ||
| placeholder.src = getPlaceholderUrl(width, height, theme); |
| while (video.firstChild) video.removeChild(video.firstChild); | ||
| themeSources.forEach(({ src, type, title }) => { | ||
| const source = document.createElement('source'); | ||
| source.src = src; |
| // the class goes on it and on any other top-level element. | ||
| function rootElement(source) { | ||
| let text = source.replace(/^---[\s\S]*?\n---/, ''); | ||
| text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, ''); |
| // the class goes on it and on any other top-level element. | ||
| function rootElement(source) { | ||
| let text = source.replace(/^---[\s\S]*?\n---/, ''); | ||
| text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, ''); |
| function styleBlocks(source) { | ||
| // Drop the front matter and the scripts, then find the <style> tags. | ||
| let text = source.replace(/^---[\s\S]*?\n---/, (m) => ' '.repeat(m.length)); | ||
| text = text.replace(/<script\b[\s\S]*?<\/script>/g, (m) => ' '.repeat(m.length)); |
| } | ||
| function parse(html) { | ||
| const root = { tag: '#root', attrs: {}, children: [] }; const stack = [root]; | ||
| const re = /<(\/?)([a-zA-Z][\w-]*)((?:\s+[^\s=>\/]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s>]+))?)*)\s*(\/?)>|([^<]+)|(<)/g; let m; |
| deepMerge(target[k], v, `${path}${k}.`); | ||
| } else { | ||
| if (t && typeof t === 'object') conflicts.push(`${path}${k}: an object in a lower source is replaced by a string`); | ||
| target[k] = v; |
Contributor
Frontend HTML artifact readyThe latest frontend build uploaded the This comment updates automatically when a new frontend build artifact is uploaded. |
…crashes its layout
This was referenced Nov 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft, investigation only, not for merge. It builds on #2156 (alternatives to Playwright's Chromium for the E2E suite) and answers why Lightpanda, Jint.Browser and Kitesurf pass only about 40 of the 74 tests in its slice (
analytics-scripts,dev-browse,cookie-dialog-layout,og-metadata, desktop project) when Chromium passes all 74. It fixes the harness where the harness was at fault. The branch is #2156's branch plus four small commits totests/cdp/and the prototype jobs. Nothing else changes.Summary
@scope. Douze compiles component styles to@scope (.cid-…) to (…)blocks: 128 blocks, holding 2,166 of the 4,443 rules insite.css. None of the three engines applies them, so half of the site's CSS is missing for them. The Astro build used attribute selectors, which all three handle. A test-only polyfill that flattens@scopekeeps Chromium at 73/74, so it is faithful.@scopewas the first blocker in 5 Lightpanda tests and 9 Jint.Browser tests. With the polyfill, 5 of those 14 pass and the other 9 get further, then fail on a layout or computed-style gap. On Kitesurf, turning on Stylo's own@scopesupport (one pref) gives the same results as the polyfill, and then triggers a Blitz layout panic that brings the engine down.page.setContentandpage.addStyleTag({ content })never complete. That caused 9 of Jint.Browser's failures and 9 of Kitesurf's. Jint.Browser also stops any page task that runs longer than 5 s.Harness fixes in this PR
--cdp-max-message-size 16777216AxeBuilder.analyze()sends the axe-core source (1.3 MB) in oneRuntime.evaluate; Lightpanda's 1 MB default closes the socket (CDP message too bigin its log). The test fails with "Target page, context or browser has been closed", its context leaks, and the next test fails withTarget.createBrowserContext: Cannot have more than one browser context at a timerun-engine.shextra-connection:browser.newContextopens a second CDP connection when the engine refuses a second contextset-content-route:page.setContentserves the HTML from a same-origin route andgotos itsetContentwrites withdocument.open()/write()and waits for aconsole.debugtag. Jint.Browser makesdocument.writeafter load a no-op and never reports page console messages; Kitesurf's own Playwright expectations marksetContentas failing. All 9cookie-dialog-layouttests timed out at 60 sstyle-tag-no-load:page.addStyleTag({ content })appends the<style>without waiting for itsloadeventloadon an inline<style>; Playwright waits forever ("Execution context was destroyed" on Jint.Browser after 30 s)--max-task-duration 120srun-engine.shscope-polyfill: every stylesheet is fetched in Node and@scope (X) to (Y) { S }is rewritten to:where(X) S(:scope→:where(X))@scopebuys. Uses Node'sfetch, becauseroute.fetch()hangs on Lightpanda (it reads cookies over CDP first)unscope-css.ts, shimAll workarounds are opt-in through
CDP_WORKAROUNDS;run-engine.shsets each engine's default list andWORKAROUNDS=noneturns them off. They change how a test reaches the page, never what it asserts. The Chromium-over-CDP row runs once without and once with the polyfill. Both fail onlypagination scrolls … with reduce motion, on CI only (6/6 locally). Playwright-launched Chromium passes it, so it is a CDP-attached-Chromium timing difference, not an engine gap.Results
CI run 43635051252 and 43638574961 (Kitesurf harness-only; it also reran Lightpanda 44, Jint.Browser 44, and Chromium with the polyfill at 74/74),
ubuntu-latest, 2 workers, no retries. The "before" columns are #2156's run 43624160556. Local runs on an M-series Mac gave the same pass counts for Lightpanda and Jint.Browser.@scopeFor Lightpanda and Jint.Browser, the
@scopepolyfill gives the gains ininitial paint … zzzz,controller fails to loadand (Jint)without JavaScript; the harness fixes give the cookie-dialog gain. Beyond the gains, 20 failures now happen later in their tests (5 on Lightpanda, 15 on Jint.Browser). Examples: the 8 Jint cookie tests reach their assertions instead of timing out,initial paintfails at line 174 instead of 157, and the Lightpanda axe test completes its scan and fails at the focus check after it. The fix removed one blocker, and a layout or computed-style assertion further on fails instead.Per-test matrix (the 47 tests that fail in at least one run)
27 tests pass in every run and are not listed (analytics-scripts, og-metadata and one dev-browse test). Ten of the rows fail only in the "KS + scope" run, after the layout panic.
"KS + scope" is Kitesurf built with Stylo's
@scopepref; the polyfill run gave the same 29. "Remaining cause" codes are each engine's first blocking cause after the fixes (Kitesurf: from the harness-only run).Root causes
Counts are tests whose first remaining failure is that cause (LP / Jint / KS, after the harness fixes).
@scopenot applied@scope<style>@scope (.r) { .a { display: flex } }</style><div class=r><span class=a></span></div>→getComputedStyle(span).display: Chromiumflex, all threeinlinegetComputedStyle<style>.a{display:flex;background-color:rgb(81,43,212)}</style><span class=a>→inline,rgba(0, 0, 0, 0). By design: stylesheet rules feed onlydisplay:none,visibility,opacity,pointer-events(lightpanda serve --help)getComputedStyle#c{background:rgb(81,43,212)}→backgroundColortransparent (shorthandsbackground,borderare not expanded into longhands);maskImage,objectFit,gridTemplateColumns,scrollPaddingTopareundefined;getComputedStyle(el, '::after')returns defaults (transform: none,opacity: 1). Kitesurf:getComputedStyle(el, '::after').transformisscaleX(0)instead ofmatrix(0, 0, 0, 1, 0, 0)(pseudo-elements skipresolved_transform), andscroll-padding-topis emptygetBoundingClientRect,boundingBox,scrollWidth,IntersectionObserver<div style="width:123px;height:45px;margin:7px">→ Chromium15,354,123,45; Lightpanda0,290,123,45(no margins or flow); Jint0,368,1440,16(synthetic full-width lines). Kitesurf lays out but differs by a few pixels (insets 14 vs 16, card 1214 vs 1219 wide, panel 773 vs ≤ 700 high) and materializes 1,119 lazy images where Chromium materializes < 75Emulation.setEmulatedMediafeaturespage.emulateMedia({ reducedMotion: 'reduce' })→matchMedia('(prefers-reduced-motion: reduce)').matchesstaysfalseon Lightpanda and Kitesurf (Kitesurf'smatchMediais a fixed shim and Stylo's servo media features stop atprefers-color-scheme);forced-colorslikewise. Jint.Browser handles it, across navigations tooInput.dispatchKeyEvent,Input.dispatchMouseEvent,:hover,:focus-visible<summary>doesn't open<details>. Lightpanda: Enter on<summary>doesn't open it and<summary>can't take focus. Kitesurf:mouseMovednever sets:hover,:focus-visibleis hard-coded false in Blitz. Jint and Lightpanda don't apply:hoverfrom the mouse eitherPage.navigateToHistoryEntry(page.goBack)Page.navigate, sopushStateentries lose their state and get nopopstate. Lightpanda and Jint.Browser behave like ChromiumFetch.requestPaused/continueRequestpage.routethat holds a script until the test releases it: Kitesurf continues the request on its own afterPARK_TIMEOUT_MS = 5000, so theinitial painttests never see the loading stateRuntime.evaluate(awaitPromise)axe.run()on the browse page exceeds a fixed 30 s evaluate ceiling (still with--max-task-duration 0); axe also misses the contrast violation on a small page (no shorthand or layout). Kitesurf: Boa in Wasm; the axe test took 360 s, the run's server stopped answering/json/versionafterwardsEmulation.setScriptExecutionDisablednaturalWidthHTMLImageElement.naturalWidth<img src=icon.svg>→ 0; Lightpanda reads dimensions from the first 16 KB of PNG, GIF, JPEG and WebP only@scopestyles apply@scopeapplied (polyfill or Stylo pref),blitz-dom-0.3.0-beta.4/src/layout/mod.rs:161panics "invalid SlotMap key used" about 75,000 times, workerd terminates scripts, and every later test fails inconnectOverCDP. Probably DioxusLabs/blitz#1560 (layout panic when removing pseudo-elements from a hidden fixed-position host), which matches the browse page's hidden fixed filter panels¹ First blocker before the polyfill. With it, these tests pass or fail on another row, where they are counted, so this row is not part of the totals for Lightpanda and Jint.Browser.
² Only in the
@scoperuns; not counted in the harness-only column.What to do about each cause
@scopeset_pref!("layout.css.at-scope.enabled", true)ininit_style_prefs(cssom.rs#L45-L49); blocked by the Blitz panic::before/::after#4594transformfor pseudo-elements like elements; otherwise keep on ChromiumfeaturesinEmulation.setEmulatedMedia) or keep on Chromium<summary>focus and activation):hover#4598:hover,:focus-visible)PARK_TIMEOUT_MS)setContent,addStyleTagdocument.open()after load #4595, console events #4589, inline<style>load#4596; the shim works around themJint.Browser issues filed from this investigation (each with a standalone repro, checked on 5.0.0-preview.2441): #4589 console events, #4590 30 s evaluate ceiling, #4591
@scope, #4592 shorthands, #4593 computed-style accessors, #4594 pseudo-elements, #4595document.open()after load, #4596 inline<style>load, #4597 keyboard activation, #4598:hover. No issues were filed for Lightpanda or Kitesurf.Is the hybrid split worth it?
No, not for speed. A hybrid would run DOM and behavior tests in process and keep layout, visual and axe tests on Chromium. On this slice, the tests that pass everywhere are the 26 request-only tests and about 11 URL-state and filter tests. On Chromium they take 1 to 2 s each, and Lightpanda is no faster per test (#2156). Every leg still downloads Chromium for the rest: across the 49 spec files, 23 read geometry, 23 read computed styles, and almost all use
toBeVisible, which needs real boxes. The split adds a second runner and a second set of failure modes, to save seconds per leg out of 8 to 18 minutes. The one-line change #2156 recommends (record trace and video on the first retry) saves about half of the test time.If an in-process engine is still wanted, for example to drop the browser download or run behavior tests inside the .NET test process, Jint.Browser is the engine to use. Its blockers for DOM and behavior tests are specific, fixable bugs:
@scope, shorthand expansion, unknown properties, pseudo-element styles, Space and Enter activation,setContentand inline-styleload, the 30 s evaluate ceiling. It needs no layout for those tests, and its media emulation and history already match Chromium. Lightpanda's cascade covers only visibility by design. Kitesurf has the most realistic layout but crashes on this site's styles, needs a 14-minute Bazel build, and is 5 to 11 times slower.