Skip to content

[Investigation] Why CDP alternatives to Chromium fail the E2E slice - #2158

Draft
Sébastien Ros (sebastienros) wants to merge 101 commits into
mainfrom
sebros/e2e-browser-gap
Draft

Sébastien Ros (sebastienros) wants to merge 101 commits into
mainfrom
sebros/e2e-browser-gap

Conversation

@sebastienros

@sebastienros Sébastien Ros (sebastienros) commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

Draft, investigation only, not for merge. It builds on #2156 (alternatives to Playwright's Chromium for the E2E suite) and answers why Lightpanda, Jint.Browser and Kitesurf pass only about 40 of the 74 tests in its slice (analytics-scripts, dev-browse, cookie-dialog-layout, og-metadata, desktop project) when Chromium passes all 74. It fixes the harness where the harness was at fault. The branch is #2156's branch plus four small commits to tests/cdp/ and the prototype jobs. Nothing else changes.

Summary

  • Most of the gap is the engines, not the harness. After the harness fixes, Lightpanda passes 44 tests (was 42), Jint.Browser 44 (was 40) and Kitesurf 40 (was 39). The remaining failures are tests that read layout, computed styles, focus or hover. Those are the specs the suite has most of.
  • One cause hits all three engines: CSS @scope. Douze compiles component styles to @scope (.cid-…) to (…) blocks: 128 blocks, holding 2,166 of the 4,443 rules in site.css. None of the three engines applies them, so half of the site's CSS is missing for them. The Astro build used attribute selectors, which all three handle. A test-only polyfill that flattens @scope keeps Chromium at 73/74, so it is faithful. @scope was the first blocker in 5 Lightpanda tests and 9 Jint.Browser tests. With the polyfill, 5 of those 14 pass and the other 9 get further, then fail on a layout or computed-style gap. On Kitesurf, turning on Stylo's own @scope support (one pref) gives the same results as the polyfill, and then triggers a Blitz layout panic that brings the engine down.
  • Harness bugs fixed: Lightpanda dropped the CDP connection on the axe-core payload (1.3 MB, over its 1 MB default limit), and allows a single browser context per connection. On Jint.Browser and Kitesurf, page.setContent and page.addStyleTag({ content }) never complete. That caused 9 of Jint.Browser's failures and 9 of Kitesurf's. Jint.Browser also stops any page task that runs longer than 5 s.
  • Recommendation: don't do the hybrid split. The tests that every engine passes (26 request-only tests and about 11 DOM and URL-state tests) already take under 2 s each on Chromium. Moving them saves seconds per leg, and each leg still needs Chromium for the other 90 % of the suite. If an in-process engine is wanted later anyway, Jint.Browser is the one to invest in, because the gaps blocking the DOM and behavior tests are concrete Jint.HtmlParser and CDP bugs (listed below). They are fixable upstream; layout is not, by design. Lightpanda's no-cascade design and Kitesurf's crash and speed rule them out for this site.

Harness fixes in this PR

Fix Engine Symptom before Where
--cdp-max-message-size 16777216 Lightpanda AxeBuilder.analyze() sends the axe-core source (1.3 MB) in one Runtime.evaluate; Lightpanda's 1 MB default closes the socket (CDP message too big in its log). The test fails with "Target page, context or browser has been closed", its context leaks, and the next test fails with Target.createBrowserContext: Cannot have more than one browser context at a time run-engine.sh
extra-connection: browser.newContext opens a second CDP connection when the engine refuses a second context Lightpanda The no-JavaScript test creates its own context next to the fixture's shim
set-content-route: page.setContent serves the HTML from a same-origin route and gotos it Jint.Browser, Kitesurf setContent writes with document.open()/write() and waits for a console.debug tag. Jint.Browser makes document.write after load a no-op and never reports page console messages; Kitesurf's own Playwright expectations mark setContent as failing. All 9 cookie-dialog-layout tests timed out at 60 s shim
style-tag-no-load: page.addStyleTag({ content }) appends the <style> without waiting for its load event Jint.Browser, Kitesurf Neither engine fires load on an inline <style>; Playwright waits forever ("Execution context was destroyed" on Jint.Browser after 30 s) shim
--max-task-duration 120s Jint.Browser Default 5 s per page turn; the page is torn down when the resource browser's first render or axe-core takes longer on a CI runner run-engine.sh
scope-polyfill: every stylesheet is fetched in Node and @scope (X) to (Y) { S } is rewritten to :where(X) S (:scope → :where(X)) all three Measures what an engine fix for @scope buys. Uses Node's fetch, because route.fetch() hangs on Lightpanda (it reads cookies over CDP first) unscope-css.ts, shim

All workarounds are opt-in through CDP_WORKAROUNDS; run-engine.sh sets each engine's default list and WORKAROUNDS=none turns them off. They change how a test reaches the page, never what it asserts. The Chromium-over-CDP row runs once without and once with the polyfill. Both fail only pagination scrolls … with reduce motion, on CI only (6/6 locally). Playwright-launched Chromium passes it, so it is a CDP-attached-Chromium timing difference, not an engine gap.

Results

CI run 43635051252 and 43638574961 (Kitesurf harness-only; it also reran Lightpanda 44, Jint.Browser 44, and Chromium with the polyfill at 74/74), ubuntu-latest, 2 workers, no retries. The "before" columns are #2156's run 43624160556. Local runs on an M-series Mac gave the same pass counts for Lightpanda and Jint.Browser.

Engine Before After harness fixes After harness fixes and @scope Test time after
Chromium over CDP 73 — 73 146 s / 154 s (polyfill)
Lightpanda 1.0.4 42 44 44 198 s
Jint.Browser 5.0.0-preview.2441 40 44 44 398 s (was 660 s: no more setContent timeouts)
Kitesurf @ 4a29038 39 40 29 (Blitz layout panic, then the engine stops answering) 890 s / 738 s

For Lightpanda and Jint.Browser, the @scope polyfill gives the gains in initial paint … zzzz, controller fails to load and (Jint) without JavaScript; the harness fixes give the cookie-dialog gain. Beyond the gains, 20 failures now happen later in their tests (5 on Lightpanda, 15 on Jint.Browser). Examples: the 8 Jint cookie tests reach their assertions instead of timing out, initial paint fails at line 174 instead of 157, and the Lightpanda axe test completes its scan and fails at the focus check after it. The fix removed one blocker, and a layout or computed-style assertion further on fails instead.

Per-test matrix (the 47 tests that fail in at least one run)

27 tests pass in every run and are not listed (analytics-scripts, og-metadata and one dev-browse test). Ten of the rows fail only in the "KS + scope" run, after the layout panic.

Test Chromium LP before LP after Jint before Jint after KS before KS after KS + scope Remaining cause (LP / Jint / KS)
cookie-dialog-layout › cookie banner uses accessible primary-action colors in light mode ✅ ❌ toHaveCSS :106 ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :115 ❌ toHaveCSS :115 cascade / CSSOM / input
cookie-dialog-layout › cookie dialog uses accessible primary-action colors in light mode ✅ ❌ toHaveCSS :106 ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :115 ❌ toHaveCSS :115 cascade / CSSOM / input
cookie-dialog-layout › cookie dialog keeps its close control and actions inset in light ltr ✅ ❌ toBeGreaterThanOrEqual :158 ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :159 ❌ toBeGreaterThanOrEqual :159 layout / layout / layout
cookie-dialog-layout › cookie dialog keeps its close control and actions inset in light rtl ✅ ❌ toBeGreaterThanOrEqual :158 ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :159 ❌ toBeGreaterThanOrEqual :159 layout / layout / layout
cookie-dialog-layout › cookie banner uses accessible primary-action colors in dark mode ✅ ❌ toHaveCSS :106 ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :115 ❌ toHaveCSS :115 cascade / CSSOM / input
cookie-dialog-layout › cookie dialog uses accessible primary-action colors in dark mode ✅ ❌ toHaveCSS :106 ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :106 ❌ timeout ❌ toHaveCSS :115 ❌ toHaveCSS :115 cascade / CSSOM / input
cookie-dialog-layout › cookie dialog keeps its close control and actions inset in dark ltr ✅ ❌ toBeGreaterThanOrEqual :158 ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :159 ❌ toBeGreaterThanOrEqual :159 layout / layout / layout
cookie-dialog-layout › cookie dialog keeps its close control and actions inset in dark rtl ✅ ❌ toBeGreaterThanOrEqual :158 ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :158 ❌ timeout ❌ toBeGreaterThanOrEqual :159 ❌ toBeGreaterThanOrEqual :159 layout / layout / layout
cookie-dialog-layout › cookie dialog remains usable with larger text ✅ ✅ ✅ ❌ timeout ✅ ❌ timeout ✅ ✅ — / — / —
dev-browse › card title underlines animate from the left for hover and keyboard focus ✅ ❌ toBe :32 ❌ toBe :32 ❌ toBe :32 ❌ toBe :32 ❌ toBe :32 ❌ toBe :32 ❌ toBe :32 cascade / CSSOM / CSSOM
dev-browse › AWS artwork switches to a legible logo for each theme ✅ ❌ toHaveCSS :52 ❌ toHaveCSS :52 ✅ ✅ ✅ ✅ ✅ cascade / — / —
dev-browse › CSS artwork texture stays subtle, edge-biased, and stable after reload ✅ ❌ toBe :112 ❌ toBe :112 ❌ toBe :112 ❌ toBe :112 ❌ toMatch :128 ❌ toMatch :128 ❌ toMatch :128 cascade / CSSOM / media
dev-browse › initial paint waits for the requested resource state: default ✅ ❌ toHaveCount :157 ❌ toBeLessThanOrEqual :174 ❌ toHaveCount :157 ❌ toBeLessThanOrEqual :174 ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 layout / layout / Fetch
dev-browse › initial paint waits for the requested resource state: ?type=blog&sort=oldest&title=desc&page=2 ✅ ❌ toHaveCount :157 ❌ toBeLessThanOrEqual :174 ❌ toHaveCount :157 ❌ toBeLessThanOrEqual :174 ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 layout / layout / Fetch
dev-browse › initial paint waits for the requested resource state: ?q=zzzz-no-matching-resource ✅ ❌ toHaveCount :157 ✅ ❌ toHaveCount :157 ✅ ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 ❌ toHaveAttribute :155 — / — / Fetch
dev-browse › resource links remain usable if the browser controller fails to load ✅ ❌ toBeHidden :195 ✅ ❌ toBeHidden :195 ✅ ❌ toBeHidden :195 ❌ toBeHidden :195 ❌ toBeGreaterThan :196 — / — / scope
dev-browse › only displayed resource artwork is materialized on first load ✅ ✅ ✅ ✅ ✅ ❌ toBeLessThan :210 ❌ toBeLessThan :210 ❌ toBeLessThan :210 — / — / layout
dev-browse › Dev Hub leads to the complete paginated resource directory ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ timeout ? / ? / ?
dev-browse › search, type and topic filters combine and survive reload and browser history ✅ ✅ ✅ ❌ not.toBeChecked :242 ❌ not.toBeChecked :242 ❌ not.toHaveURL :243 ❌ not.toHaveURL :243 ❌ TimeoutError: browserType.connectOverCDP — / input / history
dev-browse › search highlights card matches through typing, pagination, reload and history ✅ ❌ not.toBe :281 ❌ not.toBe :281 ❌ not.toBe :281 ❌ not.toBe :281 ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / —
dev-browse › tutorials and quickstarts keep their integration topic, while includes stay excluded ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › article language filters include AppHost and consuming-client examples ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › every ingested official blog post is reachable across resource pages ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › reference identities and release imagery stay distinct in both themes ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › empty results preserve useful filters and unknown URL state is normalized ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › multi-type filters and sorting reflect their URL state ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › legacy provider links do not leave an invisible filter active ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › multiple languages combine with other filters and survive reload, history, and clearing ✅ ✅ ✅ ❌ not.toBeChecked :441 ❌ not.toBeChecked :441 ❌ toBeVisible :425 ❌ toBeVisible :425 ❌ TimeoutError: browserType.connectOverCDP — / input / scope
dev-browse › all resource variants render, with bounded cards and no horizontal overflow ✅ ❌ toBe :464 ❌ toBe :464 ❌ locator.evaluate: Error: Cannot read pro :463 ❌ locator.evaluate: Error: Cannot read pro :463 ❌ toBe :465 ❌ toBe :465 ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / layout
dev-browse › shared breadcrumbs and uniform image headers at 390px ✅ ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toBeGreaterThan :503 ❌ toBeGreaterThan :503 ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / layout
dev-browse › shared breadcrumbs and uniform image headers at 768px ✅ ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ timeout ❌ timeout ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / speed
dev-browse › shared breadcrumbs and uniform image headers at 1440px ✅ ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ toHaveCSS :496 ❌ timeout ❌ timeout ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / speed
dev-browse › video platforms remain searchable metadata without a dedicated filter ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › top filters support keyboard dismissal and stay within the viewport ✅ ❌ toBeGreaterThanOrEqual :537 ❌ toBeGreaterThanOrEqual :537 ❌ toBeGreaterThanOrEqual :537 ❌ toBeGreaterThanOrEqual :537 ❌ toBeGreaterThanOrEqual :537 ❌ toBeGreaterThanOrEqual :537 ❌ TimeoutError: browserType.connectOverCDP layout / layout / layout
dev-browse › mobile filters open as readable fixed panels without shifting results ✅ ❌ toHaveCSS :587 ❌ toBeGreaterThanOrEqual :590 ❌ toHaveCSS :587 ❌ toBeLessThanOrEqual :592 ❌ toHaveCSS :587 ❌ toHaveCSS :587 ❌ TimeoutError: browserType.connectOverCDP layout / layout / scope
dev-browse › every dropdown shows responsive options without a separate search input ✅ ❌ toHaveCSS :631 ❌ toHaveCSS :631 ❌ toHaveCSS :631 ❌ toHaveCount :666 ❌ toHaveCSS :631 ❌ toHaveCSS :631 ❌ TimeoutError: browserType.connectOverCDP cascade / layout / scope
dev-browse › filter changes preserve control and result positions ✅ ❌ toEqual :685 ❌ toEqual :685 ❌ toEqual :685 ❌ toEqual :685 ❌ toEqual :685 ❌ toEqual :685 ❌ TimeoutError: browserType.connectOverCDP layout / layout / layout
dev-browse › type options use available height without unnecessary scrolling ✅ ✅ ✅ ❌ toBeLessThanOrEqual :711 ❌ toBeLessThanOrEqual :711 ❌ toBeLessThanOrEqual :711 ❌ toBeLessThanOrEqual :711 ❌ TimeoutError: browserType.connectOverCDP — / layout / layout
dev-browse › filtered selections persist alongside sort history and clearing ✅ ✅ ✅ ✅ ✅ ✅ ✅ ❌ TimeoutError: browserType.connectOverCDP ? / ? / ?
dev-browse › date and title directions persist across searches, reloads and history ✅ ❌ toHaveCSS :756 ❌ toHaveCSS :756 ❌ toHaveCSS :756 ❌ toBeGreaterThanOrEqual :759 ❌ toHaveCSS :756 ❌ toHaveCSS :756 ❌ TimeoutError: browserType.connectOverCDP cascade / layout / scope
dev-browse › history returns focus from hidden pagination and closed filter options without stealing it ✅ ❌ toBeFocused :821 ❌ toBeFocused :821 ✅ ✅ ❌ toBeFocused :815 ❌ toBeFocused :815 ❌ TimeoutError: browserType.connectOverCDP input / — / history
dev-browse › numbered pagination preserves combined sorting, keyboard focus, history, and page boundaries ✅ ❌ toContain :884 ❌ toContain :884 ❌ toContain :884 ❌ toContain :884 ❌ toBeFocused :854 ❌ toBeFocused :854 ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / input
dev-browse › pagination scrolls to the breadcrumb with no-preference motion without losing keyboard focus ✅ ❌ toBeLessThanOrEqual :918 ❌ toBeLessThanOrEqual :918 ❌ toBeLessThanOrEqual :918 ❌ toBeLessThanOrEqual :918 ❌ toBeLessThanOrEqual :918 ❌ toBeLessThanOrEqual :918 ❌ TimeoutError: browserType.connectOverCDP cascade / CSSOM / CSSOM
dev-browse › pagination scrolls to the breadcrumb with reduce motion without losing keyboard focus ✅ ❌ toHaveAttribute :913 ❌ toHaveAttribute :913 ❌ toHaveAttribute :913 ❌ toHaveAttribute :913 ❌ toHaveAttribute :913 ❌ toHaveAttribute :913 ❌ TimeoutError: browserType.connectOverCDP media / CSSOM / media
dev-browse › language icons show article coverage opposite the type badge in both themes ✅ ❌ toBeGreaterThan :960 ❌ toBeGreaterThan :960 ❌ toBeVisible :957 ❌ toBeGreaterThan :962 ❌ timeout ❌ timeout ❌ TimeoutError: browserType.connectOverCDP SVG img / layout / speed
dev-browse › filters and cards remain accessible in both themes ✅ ❌ page.evaluate: Target page, context or b :978 ❌ toBeFocused :981 ❌ frame.evaluate: Execution context was de :978 ❌ frame.evaluate: Execution context was de :978 ❌ timeout ❌ timeout ❌ TimeoutError: browserType.connectOverCDP input / speed / speed
dev-browse › all resource destinations remain available without JavaScript ✅ ❌ browser.newContext: Protocol error (Targ :987 ❌ browserContext.newPage: Protocol error ( :988 ❌ toBeHidden :994 ✅ ❌ TimeoutError: browserType.connectOverCDP ❌ TimeoutError: browserType.connectOverCDP ❌ TimeoutError: browserType.connectOverCDP no-JS / — / speed
Passed (of 74) 74 42 44 40 44 39 40 29

"KS + scope" is Kitesurf built with Stylo's @scope pref; the polyfill run gave the same 29. "Remaining cause" codes are each engine's first blocking cause after the fixes (Kitesurf: from the harness-only run).

Root causes

Counts are tests whose first remaining failure is that cause (LP / Jint / KS, after the harness fixes).

Cause LP Jint KS API or CDP method Minimal repro
CSS @scope not applied 5¹ 9¹ 5 CSS Cascading 6 @scope <style>@scope (.r) { .a { display: flex } }</style><div class=r><span class=a></span></div> → getComputedStyle(span).display: Chromium flex, all three inline
Cascade limited to visibility properties 16 — — getComputedStyle <style>.a{display:flex;background-color:rgb(81,43,212)}</style><span class=a> → inline, rgba(0, 0, 0, 0). By design: stylesheet rules feed only display:none, visibility, opacity, pointer-events (lightpanda serve --help)
CSSOM gaps — 14 2 getComputedStyle Jint: #c{background:rgb(81,43,212)} → backgroundColor transparent (shorthands background, border are not expanded into longhands); maskImage, objectFit, gridTemplateColumns, scrollPaddingTop are undefined; getComputedStyle(el, '::after') returns defaults (transform: none, opacity: 1). Kitesurf: getComputedStyle(el, '::after').transform is scaleX(0) instead of matrix(0, 0, 0, 1, 0, 0) (pseudo-elements skip resolved_transform), and scroll-padding-top is empty
No or approximate layout 9 13 10 getBoundingClientRect, boundingBox, scrollWidth, IntersectionObserver <div style="width:123px;height:45px;margin:7px"> → Chromium 15,354,123,45; Lightpanda 0,290,123,45 (no margins or flow); Jint 0,368,1440,16 (synthetic full-width lines). Kitesurf lays out but differs by a few pixels (insets 14 vs 16, card 1214 vs 1219 wide, panel 773 vs ≤ 700 high) and materializes 1,119 lazy images where Chromium materializes < 75
Media emulation ignored 1 — 2 Emulation.setEmulatedMedia features page.emulateMedia({ reducedMotion: 'reduce' }) → matchMedia('(prefers-reduced-motion: reduce)').matches stays false on Lightpanda and Kitesurf (Kitesurf's matchMedia is a fixed shim and Stylo's servo media features stop at prefers-color-scheme); forced-colors likewise. Jint.Browser handles it, across navigations too
Keyboard, focus, hover 2 2 5 Input.dispatchKeyEvent, Input.dispatchMouseEvent, :hover, :focus-visible Jint: Space on a focused checkbox doesn't toggle it, Enter on <summary> doesn't open <details>. Lightpanda: Enter on <summary> doesn't open it and <summary> can't take focus. Kitesurf: mouseMoved never sets :hover, :focus-visible is hard-coded false in Blitz. Jint and Lightpanda don't apply :hover from the mouse either
History navigation reloads — — 2 Page.navigateToHistoryEntry (page.goBack) Kitesurf always does a full Page.navigate, so pushState entries lose their state and get no popstate. Lightpanda and Jint.Browser behave like Chromium
Fetch interception auto-continues after 5 s — — 3 Fetch.requestPaused / continueRequest page.route that holds a script until the test releases it: Kitesurf continues the request on its own after PARK_TIMEOUT_MS = 5000, so the initial paint tests never see the loading state
Speed and engine limits — 1 5 Runtime.evaluate (awaitPromise) Jint: axe.run() on the browse page exceeds a fixed 30 s evaluate ceiling (still with --max-task-duration 0); axe also misses the contrast violation on a small page (no shorthand or layout). Kitesurf: Boa in Wasm; the axe test took 360 s, the run's server stopped answering /json/version afterwards
Script disabling 1 — (1) Emulation.setScriptExecutionDisabled Lightpanda: "Disabling script execution is not supported". Kitesurf ignores it
SVG naturalWidth 1 — — HTMLImageElement.naturalWidth <img src=icon.svg> → 0; Lightpanda reads dimensions from the first 16 KB of PNG, GIF, JPEG and WebP only
Layout panic once @scope styles apply — — (11)² Blitz layout With @scope applied (polyfill or Stylo pref), blitz-dom-0.3.0-beta.4/src/layout/mod.rs:161 panics "invalid SlotMap key used" about 75,000 times, workerd terminates scripts, and every later test fails in connectOverCDP. Probably DioxusLabs/blitz#1560 (layout panic when removing pseudo-elements from a hidden fixed-position host), which matches the browse page's hidden fixed filter panels

¹ First blocker before the polyfill. With it, these tests pass or fail on another row, where they are counted, so this row is not part of the totals for Lightpanda and Jint.Browser.
² Only in the @scope runs; not counted in the harness-only column.

What to do about each cause

Cause Lightpanda Jint.Browser Kitesurf
@scope Moot (no cascade) Fix upstream: sebastienros/jint#4591 Fix upstream: set_pref!("layout.css.at-scope.enabled", true) in init_style_prefs (cssom.rs#L45-L49); blocked by the Blitz panic
Cascade / CSSOM Keep on Chromium (by design) Fix upstream: shorthands #4592, camelCase accessors and computed values #4593, ::before/::after #4594 Fix upstream: resolve transform for pseudo-elements like elements; otherwise keep on Chromium
Layout Keep on Chromium Keep on Chromium (by design) Keep on Chromium (close, not exact)
Media emulation Fix upstream (read features in Emulation.setEmulatedMedia) or keep on Chromium — Fix upstream (dispatch.ts#L278-L286)
Keyboard, focus, hover Fix upstream (<summary> focus and activation) Fix upstream: Space and Enter activation #4597, :hover #4598 Fix upstream (:hover, :focus-visible)
History — — Fix upstream (page.ts#L588-L610)
Fetch 5 s park — — Fix upstream (configurable PARK_TIMEOUT_MS)
Speed / limits — Fix upstream: 30 s evaluate ceiling #4590; axe tests stay on Chromium anyway (contrast needs layout) Drop for axe tests; concurrency is cloudflare/kitesurf#129
Script disabling Fix upstream or keep that test on Chromium — Fix upstream
setContent, addStyleTag — Fix upstream: document.open() after load #4595, console events #4589, inline <style> load #4596; the shim works around them Same; the shim works around it
Layout panic — — Fix upstream (DioxusLabs/blitz#1560)

Jint.Browser issues filed from this investigation (each with a standalone repro, checked on 5.0.0-preview.2441): #4589 console events, #4590 30 s evaluate ceiling, #4591 @scope, #4592 shorthands, #4593 computed-style accessors, #4594 pseudo-elements, #4595 document.open() after load, #4596 inline <style> load, #4597 keyboard activation, #4598 :hover. No issues were filed for Lightpanda or Kitesurf.

Is the hybrid split worth it?

No, not for speed. A hybrid would run DOM and behavior tests in process and keep layout, visual and axe tests on Chromium. On this slice, the tests that pass everywhere are the 26 request-only tests and about 11 URL-state and filter tests. On Chromium they take 1 to 2 s each, and Lightpanda is no faster per test (#2156). Every leg still downloads Chromium for the rest: across the 49 spec files, 23 read geometry, 23 read computed styles, and almost all use toBeVisible, which needs real boxes. The split adds a second runner and a second set of failure modes, to save seconds per leg out of 8 to 18 minutes. The one-line change #2156 recommends (record trace and video on the first retry) saves about half of the test time.

If an in-process engine is still wanted, for example to drop the browser download or run behavior tests inside the .NET test process, Jint.Browser is the engine to use. Its blockers for DOM and behavior tests are specific, fixable bugs: @scope, shorthand expansion, unknown properties, pseudo-element styles, Space and Enter activation, setContent and inline-style load, the 30 s evaluate ceiling. It needs no layout for those tests, and its media emulation and history already match Chromium. Lightpanda's cascade covers only visibility by design. Kitesurf has the most realistic layout but crashes on this site's styles, needs a 14-minute Bazel build, and is 5 to 11 times slower.

…ipted converter, 647 files, 35 in the manual queue)
…tyles, client scripts, sidebar and i18n data, output templates
…ndex, Liquid partials of the API components, Markdown twins
…t-id meta from build info, redirect template permalink guard, sidebar buttons styled
…ale home pages; home pages without heading anchors; root page in the collections
…ollers keep typed text in the url; no fade on targeted members
…yles on API pages, API pages see the collections
…codes (live videos, gallery, catalog, sessions, journeys, runtime choices)
…and script, readable titles, file tree icons, tab selection on server-rendered lists, dropped component styles restored, icons passthrough
- Escape the json attributes of the version filter and the asciinema player, so the API version dropdowns work.
- Point the Markdown actions of fallback pages at the shown content's Markdown.
- Give the locale homes the home Markdown twin, with links in their locale.
- Keep the AspireConf clock and timezone choices for the tab's session.
- Load ResourceBrowser.js as an async module, so the chrome's scripts don't wait for it.
The specs no longer expect Astro's client router (astro:page-load, swapped documents, state on window surviving a
navigation): they check each page after its load, and the native transition and swap fallback variants are one test.
…ere()

The hub layout's `.dev-center :where(button, input, select) { font: inherit }` beat `.search-action` once bumped,
so the glossary's Clear filters button inherited 16px and pushed the search input on narrower fonts.
- site-search: the held Pagefind UI bundle can delay the load event, so wait for the parsed document.
- route-style-navigation: name the properties that differ when a route doesn't match its fresh rendering.
- dev-browse: the axe scans of the browse page need more than the default timeout on CI runners.
…undles ordered them

HubLayout's global `.dev-center :where(button)` default came first there, so `.search-action` won their tie (the
glossary's Clear filters button is 14px, not 16px).
The same 74 desktop tests run with Playwright as configured, Playwright without trace and
video recording, and Playwright connected over CDP to Chromium, Lightpanda, Jint.Browser
and Kitesurf. The jobs report to the summary and do not gate the frontend checks.
analytics-scripts, og-metadata and the behavioral dev-browse tests run in one .NET process,
without CDP or a browser download. The prototype jobs run the same 34 tests with Playwright,
with and without trace and video recording, so the CI times compare.
The CDP job now restores Jint.Browser.Tool 5.0.0-preview.2441, the native AOT tool of the
same commit and the same package version as the xUnit port, so both Jint.Browser runs use
the published packages.
…scope

Lightpanda drops the connection on CDP messages over 1 MB, which the axe-core
source exceeds, and allows one browser context per connection. Jint.Browser
never completes page.setContent or page.addStyleTag. None of the three engines
applies CSS @scope, where Douze puts half of the site's rules. The shim gains
opt-in workarounds for each, and the jobs run each engine with them.

function rootElement(source) {
let text = source.replace(/^---[\s\S]*?\n---/, '');
text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, '');

function rootElement(source) {
let text = source.replace(/^---[\s\S]*?\n---/, '');
text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, '');

function styleBlocks(source) {
let text = source.replace(/^---[\s\S]*?\n---/, (m) => ' '.repeat(m.length));
text = text.replace(/<script\b[\s\S]*?<\/script>/g, (m) => ' '.repeat(m.length));
if (!width || !height) return;

const theme = getCurrentTheme();
placeholder.src = getPlaceholderUrl(width, height, theme);
while (video.firstChild) video.removeChild(video.firstChild);
themeSources.forEach(({ src, type, title }) => {
const source = document.createElement('source');
source.src = src;
// the class goes on it and on any other top-level element.
function rootElement(source) {
let text = source.replace(/^---[\s\S]*?\n---/, '');
text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, '');
// the class goes on it and on any other top-level element.
function rootElement(source) {
let text = source.replace(/^---[\s\S]*?\n---/, '');
text = text.replace(/<(script|style)\b[\s\S]*?<\/\1>/g, '').replace(/<!--[\s\S]*?-->/g, '');
function styleBlocks(source) {
// Drop the front matter and the scripts, then find the <style> tags.
let text = source.replace(/^---[\s\S]*?\n---/, (m) => ' '.repeat(m.length));
text = text.replace(/<script\b[\s\S]*?<\/script>/g, (m) => ' '.repeat(m.length));
}
function parse(html) {
const root = { tag: '#root', attrs: {}, children: [] }; const stack = [root];
const re = /<(\/?)([a-zA-Z][\w-]*)((?:\s+[^\s=>\/]+(?:\s*=\s*(?:"[^"]*"|'[^']*'|[^\s>]+))?)*)\s*(\/?)>|([^<]+)|(<)/g; let m;
deepMerge(target[k], v, `${path}${k}.`);
} else {
if (t && typeof t === 'object') conflicts.push(`${path}${k}: an object in a lower source is replaced by a string`);
target[k] = v;
@aspire-repo-bot

Copy link
Copy Markdown
Contributor

Frontend HTML artifact ready

The latest frontend build uploaded the frontend-dist artifact for PR #2158. Use the VS Code button below to open this PR with GitHub Artifacts Explorer and browse the built HTML locally.

VS Code: Open PR #2158 artifacts

This comment updates automatically when a new frontend build artifact is uploaded.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants