diff --git a/.changeset/build-stylex-cross-file-consts.md b/.changeset/build-stylex-cross-file-consts.md new file mode 100644 index 0000000000000..2d4cb0278765a --- /dev/null +++ b/.changeset/build-stylex-cross-file-consts.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/build': patch +--- + +[fix] `astryxStylex` resolves `defineConsts` values read from another file. With the `{stylexOptions}` form (the one Storybook uses), a file that read only constants from a `.stylex.ts` module lost that import once StyleX compiled it, so the constants never reached the CSS: a constant used as a media-query key failed the build ("Invalid empty selector") and one used as a value shipped as a custom property nothing declares. A constant defined on the library side of the layer split and read on the product side stayed unresolved in either form. A production build now also warns, naming the variable and the rule that reads it, when its stylesheet reads a StyleX-generated custom property that nothing declares; set `undefinedVariables: 'error'` to fail the build instead. + +@vjeux diff --git a/.changeset/gesture-browser-negotiation.md b/.changeset/gesture-browser-negotiation.md new file mode 100644 index 0000000000000..1be0500883217 --- /dev/null +++ b/.changeset/gesture-browser-negotiation.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: touch-action derivation, the native scroll probe and haptics for the gesture system. Not exported from the package entry point; no component uses them yet. + +@vjeux diff --git a/.changeset/gesture-delegates.md b/.changeset/gesture-delegates.md new file mode 100644 index 0000000000000..16f7a41d38557 --- /dev/null +++ b/.changeset/gesture-delegates.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: code outside a component relates to the component's own gesture recognizers by name, the way a UIKit app uses a recognizer's name and its own delegate (`gestureNamed`, `addGestureDelegate`, `useGestureDelegate`). Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/gesture-host.md b/.changeset/gesture-host.md new file mode 100644 index 0000000000000..d4f793b36e7f1 --- /dev/null +++ b/.changeset/gesture-host.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the gesture system's DOM host (claims, click suppression, the browser's takeover, the pointer behind a contact). Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/gesture-pan.md b/.changeset/gesture-pan.md new file mode 100644 index 0000000000000..12eb02fc42391 --- /dev/null +++ b/.changeset/gesture-pan.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the pan gesture recognizer, with UIKit's measured hysteresis and velocity filter. Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/gesture-press-tracking.md b/.changeset/gesture-press-tracking.md new file mode 100644 index 0000000000000..b0cce680324e4 --- /dev/null +++ b/.changeset/gesture-press-tracking.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: UIControl press tracking with control and cell profiles, the shared press paint, delegated press tracking and the gesture hooks. Not exported from the package entry point; no component uses them yet. + +@vjeux diff --git a/.changeset/gesture-readme.md b/.changeset/gesture-readme.md new file mode 100644 index 0000000000000..9a6293b811b68 --- /dev/null +++ b/.changeset/gesture-readme.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the gesture system's README for component authors, with its browser proof in Chromium and WebKit. + +@vjeux diff --git a/.changeset/gesture-recognizer-core.md b/.changeset/gesture-recognizer-core.md new file mode 100644 index 0000000000000..0b0bfc389235f --- /dev/null +++ b/.changeset/gesture-recognizer-core.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the gesture recognizer core for touch presentations: UIKit's recognizer state machine and the arena that settles which recognizer owns a touch. Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/gesture-swipe-pinch-rotation-edge-hover.md b/.changeset/gesture-swipe-pinch-rotation-edge-hover.md new file mode 100644 index 0000000000000..e8370c0709687 --- /dev/null +++ b/.changeset/gesture-swipe-pinch-rotation-edge-hover.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: swipe, pinch, rotation, screen-edge pan and hover gesture recognizers. Not exported from the package entry point; no component uses them yet. + +@vjeux diff --git a/.changeset/gesture-tap-long-press.md b/.changeset/gesture-tap-long-press.md new file mode 100644 index 0000000000000..9ef786da65842 --- /dev/null +++ b/.changeset/gesture-tap-long-press.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: tap and long-press gesture recognizers with UIKit's measured thresholds. Not exported from the package entry point; no component uses them yet. + +@vjeux diff --git a/.changeset/reduced-motion-record.md b/.changeset/reduced-motion-record.md new file mode 100644 index 0000000000000..17ebb3fdc326d --- /dev/null +++ b/.changeset/reduced-motion-record.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[fix] The experimental touch animator follows design:reduced-motion. Under Reduce Motion every `animate` and `play` (travel or opacity, on a spring or any curve) becomes an opacity-only cross-fade of 200 ms on a coarse primary pointer, or for a call marked `touchRelease`, and completes at once on any other pointer; nothing moves, scales or springs, and travel lands with a plain style write while the element is transparent, never as an animated transform. `timing.reducedMotionCrossFadeMs` is now 200 ms and is no longer a theme role. The `reducedMotion` and `crossfade` call options are removed; `crossfadeAs` stays. `animator.reducedMotion()` reports the mode for feedback with no fade of its own, `reducedMotionFade()` gives the same fade to code beside the animator, and `reducedMotion.stylex.ts` holds its constants for StyleX transitions. + +@vjeux diff --git a/.changeset/touch-motion-theme-roles.md b/.changeset/touch-motion-theme-roles.md new file mode 100644 index 0000000000000..feeef3c8207aa --- /dev/null +++ b/.changeset/touch-motion-theme-roles.md @@ -0,0 +1,8 @@ +--- +'@astryxdesign/core': patch +'@astryxdesign/cli': patch +--- + +[feat] Themes can retune touch motion roles. `defineTheme({motion})` takes `springs`, `durations` and `curves` keyed by motion role, the table path of a measured touch motion value (`springs.sheetPresent`, `navigation.transitionMs`), at the root or in an adaptation rule, and writes one `--motion-*` variable per role it sets. A role the theme leaves out keeps its measured default, an unknown role id is refused, and the duration scale (`fast`, `medium`, `ratio`) is optional when a theme sets only roles. No shipped component reads the roles yet. `astryx docs motion` lists every motion role with its CSS variable and default, and the theme authoring guide and template describe the new fields. + +@vjeux diff --git a/.changeset/touch-presentation.md b/.changeset/touch-presentation.md new file mode 100644 index 0000000000000..ad0ae3ff44944 --- /dev/null +++ b/.changeset/touch-presentation.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[feat] `presentation="adaptive"` on DropdownMenu, ContextMenu, MoreMenu, Selector and MultiSelector, and a DropdownMenu's or ContextMenu's default sub-menu drill-in, now use the theme's compact definition with a coarse primary pointer: a viewport narrower than the theme's `md` point (768px by default) or shorter than 500px. This is a behavior change. The old default was a bottom sheet at 768px wide and below on a coarse pointer. Now a coarse-pointer screen exactly 768px wide keeps the anchored popover, and a phone in landscape (wider than 768px but shorter than 500px) gets the bottom sheet and the drilled-in sub-menu. A theme that moves `md` moves the edge with it. To pin a component's presentation regardless of the rule, pass `presentation="popover"` or `presentation="bottom-sheet"`; to keep a sub-menu's flyout, pass `presentation="flyout"` on `DropdownMenuSubMenu`. Mouse and keyboard users on desktop see no change, because every case requires a coarse pointer. + +@vjeux diff --git a/.changeset/uikit-constants-table.md b/.changeset/uikit-constants-table.md new file mode 100644 index 0000000000000..01fd41a182b0b --- /dev/null +++ b/.changeset/uikit-constants-table.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the UIKit constants table for touch presentations (springs, scroll physics, timing curves and navigation values), each value with its source and a checked list of the simulator measurements still owed. Not exported from the package entry point; no component reads it yet. + +@vjeux diff --git a/.changeset/uikit-motion-animator-curves.md b/.changeset/uikit-motion-animator-curves.md new file mode 100644 index 0000000000000..22929c1cc797c --- /dev/null +++ b/.changeset/uikit-motion-animator-curves.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the touch presentations' animator plays any closed-form curve (such as a spring chasing a moving target), drives a pseudo-element such as a dialog's ::backdrop, follows design:reduced-motion (under Reduce Motion, an opacity-only cross-fade of 200 ms on a coarse pointer or after a touch release and an immediate change on any other pointer, with no option or theme value that keeps travel or lengthens the fade; an entrance or exit role orients a slide that changes no opacity), and starts a call from rest when the browser commits it, as Core Animation does. Not exported from the package entry point. + +@vjeux diff --git a/.changeset/uikit-motion-animator.md b/.changeset/uikit-motion-animator.md new file mode 100644 index 0000000000000..cfae66b64a112 --- /dev/null +++ b/.changeset/uikit-motion-animator.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: a compositor animator for touch presentations. It plays UIKit's springs, scroll coasts and any closed-form curve as Web Animations on translate, rotate, scale and opacity, so they keep running while JavaScript is busy, and reads values and velocities back from the curves for interruption, retargeting and Reduce Motion cross-fades. Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/uikit-motion-math.md b/.changeset/uikit-motion-math.md new file mode 100644 index 0000000000000..7af3d73a92a98 --- /dev/null +++ b/.changeset/uikit-motion-math.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: the spring, decay, rubber-band and projection math of UIKit for touch presentations, as pure functions checked against an independent integration and against values read from UIKit. Not exported from the package entry point; no component uses it yet. + +@vjeux diff --git a/.changeset/uikit-motion-use-animator.md b/.changeset/uikit-motion-use-animator.md new file mode 100644 index 0000000000000..b234a4e987752 --- /dev/null +++ b/.changeset/uikit-motion-use-animator.md @@ -0,0 +1,7 @@ +--- +'@astryxdesign/core': patch +--- + +[chore] Internal: useAnimator, a React wrapper around the touch presentations' animator that never re-renders per frame, and browser proofs on the interaction harness that the animator's motion keeps running through a blocked main thread in Chromium and WebKit. Not exported from the package entry point. + +@vjeux diff --git a/.github/scripts/ci-test-routing.test.mjs b/.github/scripts/ci-test-routing.test.mjs index d5cc531118122..df91e3838a882 100644 --- a/.github/scripts/ci-test-routing.test.mjs +++ b/.github/scripts/ci-test-routing.test.mjs @@ -612,3 +612,52 @@ describe('deploy.yml push gating', () => { } }); }); + +describe('ci.yml interaction proofs', () => { + const workflow = load('ci.yml'); + const job = workflow.jobs['pr-interaction']; + const step = name => job.steps.find(candidate => candidate.name === name); + + it('runs the interaction config in Chromium against the exact PR head', () => { + expect(KNOWN_RUNNERS.has(job['runs-on'])).toBe(true); + expect(job.needs).toEqual( + expect.arrayContaining(['check-scope', 'build-storybook']), + ); + const checkout = job.steps.find(candidate => + candidate.uses?.startsWith('actions/checkout@'), + ); + expect(checkout?.with?.ref).toBe( + '${{ github.event.pull_request.head.sha }}', + ); + const run = step('Run interaction proofs').run; + expect(run).toContain('--config playwright.interaction.config.ts'); + expect(run).toContain('--project chromium'); + const packageJson = JSON.parse(read('package.json')); + expect(packageJson.scripts['test:interaction']).toBe( + 'playwright test --config playwright.interaction.config.ts', + ); + }); + + it('runs whenever the harness, its config, a component or a story changes', () => { + const scope = step('Check interaction scope').run; + expect(scope).toContain('.github/scripts/pr-scope-touches.sh'); + for (const path of [ + 'internal/interaction-harness/', + 'playwright\\.interaction\\.config\\.ts', + 'packages/[^/]+/src/', + 'apps/storybook/', + ]) { + expect(scope).toContain(path); + } + // Unknown scope (exit 2) runs the proofs; only "out of scope" skips them. + expect(scope).toContain('if [ "$scope" = 1 ]; then'); + }); + + it('typechecks the harness in both workflows', () => { + for (const file of ['ci.yml', 'deploy.yml']) { + expect(runLines(load(file).jobs.typecheck), file).toContain( + '@astryxdesign/interaction-harness typecheck', + ); + } + }); +}); diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 55d437a675c85..0b4e913e72e80 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1270,6 +1270,12 @@ jobs: if: needs.check-scope.outputs.docsite_only != 'true' && needs.check-scope.outputs.spec_only != 'true' && needs.check-scope.outputs.tooling_only != 'true' run: pnpm -F @astryxdesign/native typecheck:types + # The interaction harness is consumed as TypeScript source by Vitest and + # Playwright, which do not typecheck; this is the only gate over it. + - name: Typecheck interaction harness + if: needs.check-scope.outputs.docsite_only != 'true' && needs.check-scope.outputs.spec_only != 'true' && needs.check-scope.outputs.tooling_only != 'true' + run: pnpm -F @astryxdesign/interaction-harness typecheck + # Sandbox preview build — the single slowest piece of PR CI (~4min), run as # its own job so it overlaps build-storybook instead of serializing after it. # PR-only: merge_group runs never deploy previews, and docsite-only PRs skip @@ -3803,6 +3809,88 @@ jobs: retention-days: 30 delete-merged: true + # Touch interaction and animation proofs: the harness in + # internal/interaction-harness and every *.interaction.browser.spec.ts. + # Real touch input, composited frames, rendering traces and presented + # animation values only exist in a real engine, and the specs read stories + # from this PR's own Storybook build. The harness's self-test runs with + # them, so a change to an instrument is proven by its negative controls + # before a component spec relies on it. Chromium only on these runners; + # see the harness README for running WebKit. + pr-interaction: + needs: [check-scope, build-storybook] + if: github.event_name == 'pull_request' && needs.check-scope.outputs.docsite_only != 'true' && needs.check-scope.outputs.spec_only != 'true' && needs.check-scope.outputs.tooling_only != 'true' + # Frame evidence measures timing: four cores, one Playwright worker. + runs-on: 4-core-ubuntu + permissions: + contents: read + steps: + - name: Checkout + uses: actions/checkout@v7 + with: + ref: ${{ github.event.pull_request.head.sha }} + + - name: Verify evidence checkout + env: + EXPECTED_HEAD_SHA: ${{ github.event.pull_request.head.sha }} + run: test "$(git rev-parse HEAD)" = "$EXPECTED_HEAD_SHA" + + # The helper comes from the base, since an older PR head lacks it. + # Exit 1 means out of scope; 0 in scope; 2 unknown, so run. + - name: Check interaction scope + id: scope + env: + BASE_SHA: ${{ github.event.pull_request.base.sha }} + run: | + scope=2 + helper="$(mktemp)" + if git fetch --quiet --no-tags --depth=1 origin "$BASE_SHA" \ + && git show "$BASE_SHA:.github/scripts/pr-scope-touches.sh" > "$helper"; then + scope=0 + bash "$helper" "$BASE_SHA" '^(internal/interaction-harness/|playwright\.interaction\.config\.ts$|packages/[^/]+/src/|apps/storybook/)' || scope=$? + fi + if [ "$scope" = 1 ]; then + echo "Neither the interaction harness nor a component changed; skipping the interaction proofs." + echo 'run=false' >> "$GITHUB_OUTPUT" + else + echo 'run=true' >> "$GITHUB_OUTPUT" + fi + + - name: Setup Node and pnpm + if: steps.scope.outputs.run == 'true' + uses: ./.github/actions/setup + with: + install: 'false' + + - name: Download Storybook artifact + if: steps.scope.outputs.run == 'true' + uses: actions/download-artifact@v8 + with: + name: storybook-${{ needs.build-storybook.outputs.short_hash }} + path: apps/storybook/dist + + - name: Install Playwright + if: steps.scope.outputs.run == 'true' + run: pnpm install --frozen-lockfile && npx playwright install chromium + + - name: Run interaction proofs + if: steps.scope.outputs.run == 'true' + run: | + pnpm exec playwright test \ + --config playwright.interaction.config.ts \ + --project chromium \ + --output interaction-proof-output + + # Storyboards (PNG contact sheets) and curve traces, for review. + - name: Upload interaction evidence + if: always() && steps.scope.outputs.run == 'true' + uses: actions/upload-artifact@v7 + with: + name: interaction-evidence + path: test-results/interaction-evidence + if-no-files-found: ignore + retention-days: 7 + # Build package test owner. The plugin's production CSS partition is a Build # integration contract: source inspection cannot prove which declaration # paints after Vite emits the final bundle. The `test-build` name makes that diff --git a/.github/workflows/deploy.yml b/.github/workflows/deploy.yml index 698440caafd9d..8badd15edbf40 100644 --- a/.github/workflows/deploy.yml +++ b/.github/workflows/deploy.yml @@ -146,6 +146,9 @@ jobs: - name: Typecheck accessibility spec-test contracts run: pnpm -F @astryxdesign/a11y-spec typecheck + - name: Typecheck interaction harness + run: pnpm -F @astryxdesign/interaction-harness typecheck + # Stable join for the three post-merge validation lanes. test: needs: [test-ui, test-node, typecheck] diff --git a/apps/sandbox/vite.config.mts b/apps/sandbox/vite.config.mts index f91d6d4d024bd..a904a908b8a23 100644 --- a/apps/sandbox/vite.config.mts +++ b/apps/sandbox/vite.config.mts @@ -192,6 +192,7 @@ export default defineConfig({ unstable_moduleResolution: stylexResolution.stylexModuleResolution, }, stylexPrefix: 'x', + undefinedVariables: 'error', }), react({ babel: { diff --git a/apps/storybook/.storybook/main.test.ts b/apps/storybook/.storybook/main.test.ts index c63b484479205..8e3c402f74123 100644 --- a/apps/storybook/.storybook/main.test.ts +++ b/apps/storybook/.storybook/main.test.ts @@ -57,6 +57,7 @@ const runtimeImports = [ describe('storybook main.ts workspace source aliases', () => { let viteAliases: Record; let stylexAliases: Record; + let undefinedVariables: unknown; const stylexSpy = vi.spyOn(build, 'astryxStylex'); beforeAll(async () => { @@ -69,6 +70,11 @@ describe('storybook main.ts workspace source aliases', () => { throw new Error('Expected the Storybook StyleX options'); } stylexAliases = options.stylexOptions!.aliases as Record; + undefinedVariables = options.undefinedVariables; + }); + + it('fails the build on a generated variable nothing declares', () => { + expect(undefinedVariables).toBe('error'); }); afterAll(() => stylexSpy.mockRestore()); diff --git a/apps/storybook/.storybook/main.ts b/apps/storybook/.storybook/main.ts index 243cb3468d30c..b9370dadfbc1c 100644 --- a/apps/storybook/.storybook/main.ts +++ b/apps/storybook/.storybook/main.ts @@ -198,6 +198,9 @@ const config: StorybookConfig = { }, }, libraryPattern: 'packages/', + // CI catches a style that reads a generated variable nothing + // declares (a dropped declaration) instead of shipping it. + undefinedVariables: 'error', }), ], resolve: { diff --git a/apps/storybook/stories/ContextMenu.stories.tsx b/apps/storybook/stories/ContextMenu.stories.tsx index 9ee6b0bbc5fae..5c144b8bf45a9 100644 --- a/apps/storybook/stories/ContextMenu.stories.tsx +++ b/apps/storybook/stories/ContextMenu.stories.tsx @@ -494,7 +494,7 @@ export const AdaptivePresentation: Story = { docs: { description: { story: - 'Uses the built-in adaptive policy: BottomSheet at 768px and below with a coarse primary pointer, cursor-positioned popover otherwise. Important actions still need a visible MoreMenu or equivalent entry point.', + 'Uses the built-in adaptive policy: BottomSheet on a compact touch screen (a coarse primary pointer, and a viewport narrower than the theme `md` breakpoint (768px by default) or shorter than 500px), cursor-positioned popover otherwise. Important actions still need a visible MoreMenu or equivalent entry point.', }, }, }, diff --git a/apps/storybook/stories/MoreMenu.stories.tsx b/apps/storybook/stories/MoreMenu.stories.tsx index ad736177aedc0..a922c4b2e7489 100644 --- a/apps/storybook/stories/MoreMenu.stories.tsx +++ b/apps/storybook/stories/MoreMenu.stories.tsx @@ -316,7 +316,7 @@ export const AdaptivePresentation: Story = { docs: { description: { story: - 'Uses the built-in adaptive policy: BottomSheet at 768px and below with a coarse primary pointer, anchored popover otherwise. Use device emulation to exercise the touch branch.', + 'Uses the built-in adaptive policy: BottomSheet on a compact touch screen (a coarse primary pointer, and a viewport narrower than the theme `md` breakpoint (768px by default) or shorter than 500px), anchored popover otherwise. Use device emulation to exercise the touch branch.', }, }, }, diff --git a/internal/README.md b/internal/README.md index 101fd8b07c28f..650fbd245d1a6 100644 --- a/internal/README.md +++ b/internal/README.md @@ -4,9 +4,10 @@ Internal tooling packages not published to npm. -| Directory | Role | Purpose | -| ------------- | ------- | ----------------------------------------------------------------------- | -| `a11y-spec/` | Testing | Reusable accessibility spec-test contracts and their evidence harnesses | -| `scripts/` | Tooling | One-off automation scripts (Crowdin screenshot upload, etc.) | -| `test-utils/` | Testing | Shared testing utilities and setup for Vitest | -| `vibe-tests/` | Testing | LLM vibeability test harness for component system | +| Directory | Role | Purpose | +| ---------------------- | ------- | --------------------------------------------------------------------------------- | +| `a11y-spec/` | Testing | Reusable accessibility spec-test contracts and their evidence harnesses | +| `interaction-harness/` | Testing | Playwright instruments for touch interaction and animation proofs and storyboards | +| `scripts/` | Tooling | One-off automation scripts (Crowdin screenshot upload, etc.) | +| `test-utils/` | Testing | Shared testing utilities and setup for Vitest | +| `vibe-tests/` | Testing | LLM vibeability test harness for component system | diff --git a/internal/interaction-harness/README.md b/internal/interaction-harness/README.md new file mode 100644 index 0000000000000..3a68a04634782 --- /dev/null +++ b/internal/interaction-harness/README.md @@ -0,0 +1,139 @@ +# @astryxdesign/interaction-harness + +Playwright instruments for touch interaction and animation proofs: real touch +input, frame evidence, curve sampling and storyboards. **Internal and +unpublished.** Specs live next to their component as +`*.interaction.browser.spec.ts` and run under +[`playwright.interaction.config.ts`](../../playwright.interaction.config.ts) on +a phone-sized touch screen (402 × 714 CSS px page, 402 × 874 screen, 3 device +pixels per CSS px, touch, mobile viewport rules), in Chromium and WebKit. + +The exemplar to copy is +[`BottomSheetDetent.interaction.browser.spec.ts`](../../packages/core/src/BottomSheet/__tests__/BottomSheetDetent.interaction.browser.spec.ts). +The harness proves its own instruments in +[`src/harness.interaction.browser.spec.ts`](src/harness.interaction.browser.spec.ts): +every positive check has a negative control beside it (a rAF animation that +freezes, a `top` animation that lays out every frame). + +## Running + +```sh +pnpm storybook:build # the specs read stories from the built Storybook +pnpm exec playwright install chromium webkit +pnpm test:interaction --project chromium # everything, Chromium +pnpm test:interaction BottomSheetDetent # one spec, both engines +pnpm storyboards # only the tests tagged @storyboard +``` + +On Linux distributions whose system libraries are older than Playwright's +WebKit build needs, run the WebKit project inside the official Playwright +image (podman or docker; the repository is mounted at the same path): + +```sh +node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…] +``` + +Evidence (storyboards, curve traces, busy-main-thread frames) is written to +`test-results/interaction-evidence/` (override with +`ASTRYX_INTERACTION_EVIDENCE`) and attached to the test report. It is never +committed: a pull request links or attaches it. + +## Instruments + +| Module | What it gives a spec | +| --------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `scripts.ts` | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact release velocity, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`. | +| `touch.ts` | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received. | +| `frames.ts` | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `sampleRects` (per-rAF rects), `frameStats`. | +| `busy.ts` | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, and a verdict. | +| `trace.ts` | `traceRendering(page, body)`: main-thread Layout / Paint / UpdateLayoutTree / PrePaint counts, `countBetween` for a sub-window, long tasks. `installLongTaskLog` / `readLongTasks`. | +| `curves.ts` | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `waitForRest`, `releaseAnimations`. | +| `math.ts` | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`. | +| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event), `castStoryboard` (real screencast frames): PNG contact sheet + JSON trace. | +| `output.ts` | `evidenceDir`, `writeEvidence`: where evidence goes and how it is attached to the report. | +| `story.ts` | `phone`, `withStorybook`, `openStory`, `frames`, and the timer gate (`installTimerGate`, `freezeTimers`, `thawTimers`). | + +## The proofs + +**Real touch.** Untrusted DOM events are not input: they start no native +scroll, claim no gesture, and read `isTrusted: false`. Chromium input goes +through CDP `Input.dispatchTouchEvent` with an explicit timestamp per event: the +page's `event.timeStamp` is the scripted time to the microsecond, whatever the +dispatch latency, and the events go through the compositor's gesture detection +(native scrolling, flings with velocity from those timestamps, `touch-action`, +`pointercancel`). WebKit input goes through the engine's own input command; see +the limits below. Read positions a frame after a move: Chromium delivers +`pointermove` aligned to animation frames. + +**Curve sampling.** Pause every animation (CSS transitions and animations are +Web Animations too), seek to a time, read `getComputedStyle`: the presented +value at that exact time, independent of frame timing. Compare it with an +oracle from `math.ts` through `compareCurve`; a tolerance states the error +budget (for a spring baked into `linear()` stops of `h` ms, chords err by up to +`h² / 8 · |x''|`). A seek past an animation's end lands half a millisecond +inside it, the last frame it presents. An animation whose keyframes end in a run of +identical values (an end value held so a later interruption can read it) +ends where that run starts: `curveEndMs`, `waitForAnimation`'s interval and +the default storyboard times stop there, and `waitForRest`, which judges the +presented `transform`, `translate`, `rotate`, `scale` and `opacity` rather +than `playState`, does not wait the hold out. + +**Busy main thread.** Mark the element, start the screencast, start the +animation, wait until it runs (`waitForAnimation`: an animation cannot start +while the thread is held), block the main thread 500–1000 ms, stop. Always +start the gesture or animation first and block mid-flight, never before: +in headless Chromium a touch that starts while the main thread is busy does +not scroll or start a gesture even with only passive listeners (the input +waits for the main thread). Tracking a finger under a busy main thread is +therefore proven on the iOS simulator, not here. The +screencast is captured off the main thread, so a compositor animation shows a +new position every frame of the block, and a main-thread animation shows none +(nothing changed, so no frame is sent at all: `busyWindowProof` counts the +whole window as one stall). `settledWithin` (2 px) stops judging at the easing +tail: the screencast rounds to whole pixels, so a tail that still moves shows +plateaus within a pixel or two of rest. The `verdict` tolerates what a +loaded machine does to a screencast and nothing a frozen animation does: a +median frame interval of at most 20 ms, no stall longer than three screencast +intervals (two dropped frames), and at least 90% of the window's whole frame +intervals presenting a new position (one miss is always allowed). +`verdict.failures` names each miss with its numbers, and `summary` carries the +raw numbers for the report. Frames are judged in presentation order: a loaded +machine can deliver them out of order. `castStoryboard` turns the frames of the +block into a filmstrip. + +**No Layout / Paint.** `traceRendering` records a Chromium trace around the +interaction; `countBetween` counts the steady middle of an animation (its +first and last frames may commit). A transform/opacity animation does no +Layout or Paint there; a layout-property animation does one per frame. + +**Storyboards.** Deterministic: frames come from seeking held animations or +from stepping scripted input, never from wall-clock capture. The JSON trace +carries each frame's measurements and the spec asserts on it; the sheet is +evidence. A component that finishes an animation from a timer (a transitionend +backstop, a cleanup render) would do so behind a held storyboard's back: +`installTimerGate` before the animation starts and `freezeTimers` while +photographing. + +## Engine limits (measured with Playwright 1.61) + +| | Chromium 149 | WebKit 26.5 (Linux build) | +| ------------------------------------------------------ | ------------ | ----------------------------------------------------------------------------------------------------- | +| Trusted touch + pointer events, multi-touch | yes | yes (a `touchstart` lists only the fingers that landed in it; pointer events exact) | +| Page sees the scripted timestamps | yes, exactly | no: real dispatch time; real-time pacing keeps velocities within ~1 ms | +| Sub-pixel touch positions | yes | no: integer CSS px | +| Native scroll, fling, pinch-zoom from injected touches | yes | no | +| Lifting one finger while others stay down | yes | pointer events right, but that `touchend` lists no remaining `touches`: skip with `PARTIAL_LIFT_SKIP` | +| Click synthesised from a touch | yes | only from `driver.tap` (WebKit's tap gesture), not from a raw touch sequence | +| Curve sampling, storyboards, input log | yes | yes | +| Screencast frame evidence (busy main thread) | yes | yes (WebKit's own screencast; frames keep coming while the main thread is blocked) | +| Rendering trace, long tasks | yes | no (Playwright exposes neither) | + +What WebKit cannot show here — native scroll hand-off, flings, a finger +lifting while another stays down, finger tracking under a busy main thread, and +anything iOS-only — needs Mobile Safari itself, on a device or the iOS +simulator. Skip such a proof on the driver's flag, citing the reason: +`test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP)`. + +The WebKit input command is reached through Playwright's in-process server +object, so it needs a locally launched browser (the default), not `connect()`. +Firefox has no touch input path and no project. diff --git a/internal/interaction-harness/package.json b/internal/interaction-harness/package.json new file mode 100644 index 0000000000000..942eccb9ed8d5 --- /dev/null +++ b/internal/interaction-harness/package.json @@ -0,0 +1,22 @@ +{ + "name": "@astryxdesign/interaction-harness", + "version": "0.0.0", + "private": true, + "type": "module", + "description": "Internal: Playwright helpers for touch interaction and animation proofs (real touch input, frame evidence, curve sampling, storyboards). Never published.", + "main": "./src/index.ts", + "exports": { + ".": "./src/index.ts", + "./math": "./src/math.ts", + "./scripts": "./src/scripts.ts" + }, + "scripts": { + "typecheck": "tsc --project tsconfig.json --noEmit" + }, + "devDependencies": { + "@astryxdesign/a11y-spec": "workspace:*", + "@playwright/test": "^1.61.1", + "typescript": "^6.0.3", + "vitest": "^4.1.11" + } +} diff --git a/internal/interaction-harness/scripts/webkit-container.mjs b/internal/interaction-harness/scripts/webkit-container.mjs new file mode 100644 index 0000000000000..32cc0452396df --- /dev/null +++ b/internal/interaction-harness/scripts/webkit-container.mjs @@ -0,0 +1,72 @@ +#!/usr/bin/env node +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * Runs the interaction proofs' WebKit project inside the official Playwright + * container image, for hosts where WebKit cannot launch (Playwright's Linux + * WebKit build needs newer system libraries than some distributions ship). + * + * node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…] + * + * The repository is mounted at the same path, so node_modules, the built + * Storybook and the evidence directory are the host's own. The image tag + * matches the installed playwright, so its browsers match the client. + * Uses podman when present, else docker. `--network=host` lets the specs' + * local Storybook server and the browser reach each other. + */ +import {spawnSync} from 'node:child_process'; +import fs from 'node:fs'; +import {createRequire} from 'node:module'; +import path from 'node:path'; +import {fileURLToPath} from 'node:url'; + +const root = path.resolve( + path.dirname(fileURLToPath(import.meta.url)), + '../../..', +); +const require = createRequire(path.join(root, 'package.json')); +const {version} = JSON.parse( + fs.readFileSync(require.resolve('playwright/package.json'), 'utf8'), +); +const image = + process.env.PLAYWRIGHT_IMAGE ?? + `mcr.microsoft.com/playwright:v${version}-noble`; +const engine = ['podman', 'docker'].find( + cmd => spawnSync(cmd, ['--version'], {stdio: 'ignore'}).status === 0, +); +if (engine == null) { + console.error('webkit-container: neither podman nor docker is installed'); + process.exit(1); +} +const args = process.argv.slice(2); +const env = [ + 'ASTRYX_STORYBOOK_DIR', + 'ASTRYX_INTERACTION_EVIDENCE', + 'INTERACTION_WORKERS', + 'CI', +] + .filter(name => process.env[name] != null) + .flatMap(name => ['-e', `${name}=${process.env[name]}`]); +const command = [ + 'run', + '--rm', + '--network=host', + '--ipc=host', + '-v', + `${root}:${root}`, + '-w', + root, + ...env, + image, + 'node', + path.join(root, 'node_modules/@playwright/test/cli.js'), + 'test', + '--config', + 'playwright.interaction.config.ts', + '--project', + 'webkit', + ...args, +]; +console.log(`${engine} ${command.join(' ')}`); +const result = spawnSync(engine, command, {stdio: 'inherit'}); +process.exit(result.status ?? 1); diff --git a/internal/interaction-harness/src/busy.test.ts b/internal/interaction-harness/src/busy.test.ts new file mode 100644 index 0000000000000..ad92548c84fc9 --- /dev/null +++ b/internal/interaction-harness/src/busy.test.ts @@ -0,0 +1,144 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file busy.test.ts + * @input Synthetic tracked frames (no browser) + * @output The busy-main-thread verdict's limits: dropped frames a loaded + * machine produces pass, a frozen animation fails every criterion, and + * out-of-order frames are judged in presentation order. + * @position Unit test of ./busy and ./frames' ordering; the browser proof + * with a real compositor animation and a rAF control is in + * harness.interaction.browser.spec.ts. + */ + +import {describe, expect, it} from 'vitest'; +import {busyWindowProof} from './busy'; +import {inPresentationOrder, type TrackedFrame} from './frames'; + +const FRAME = 1000 / 60; +const BLOCK = {from: 1000, to: 1700}; + +/** A marker moving 3 px per frame from t = 800 to 1900, minus `dropped` frame indices. */ +function moving(dropped: readonly number[] = []): TrackedFrame[] { + const frames: TrackedFrame[] = []; + for (let i = 0; 800 + i * FRAME <= 1900; i += 1) { + if (dropped.includes(i)) { + continue; + } + frames.push({ + t: 800 + i * FRAME, + box: {x: 0, y: 3 * i, width: 10, height: 10}, + } as TrackedFrame); + } + return frames; +} + +/** Index of the first frame at or after `t`. */ +const at = (t: number) => Math.ceil((t - 800) / FRAME); + +describe('busyWindowProof verdict', () => { + it('passes a compositor animation with every frame', () => { + const proof = busyWindowProof(moving(), BLOCK); + expect(proof.verdict.failures).toEqual([]); + expect(proof.longestStallFrames).toBe(1); + }); + + it('passes one and two consecutive dropped frames (3 intervals)', () => { + const k = at(1300); + expect(busyWindowProof(moving([k]), BLOCK).verdict.failures).toEqual([]); + const two = busyWindowProof(moving([k, k + 1]), BLOCK); + expect(two.longestStallFrames).toBe(3); + expect(two.verdict.failures).toEqual([]); + }); + + it('fails three consecutive dropped frames', () => { + const k = at(1300); + const proof = busyWindowProof(moving([k, k + 1, k + 2]), BLOCK); + expect(proof.longestStallFrames).toBe(4); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + }); + + it('fails when more than 10% of the expected frames are missing', () => { + const k = at(1050); + // 39 frames expected: up to 3 may be missing (10% = 3.9). + const scattered = [0, 12, 24].map(d => k + d); + expect(busyWindowProof(moving(scattered), BLOCK).verdict.failures).toEqual( + [], + ); + const many = [0, 6, 12, 18, 24].map(d => k + d); + expect( + busyWindowProof(moving(many), BLOCK).verdict.failures.join('\n'), + ).toMatch(/expected frames moved/); + }); + + it('lets a short window miss a frame, not three', () => { + // 150 ms judged: 8 whole intervals, one miss allowed (10% would be 0.8). + const short = {from: 1000, to: 1190}; + const k = at(1040); + expect(busyWindowProof(moving([k]), short).verdict.failures).toEqual([]); + // Three of nine missing fails, on frame count or cadence. + expect(busyWindowProof(moving([k, k + 3, k + 6]), short).verdict.pass).toBe( + false, + ); + }); + + it('fails a frozen animation on every criterion', () => { + // Frames before and after the block only: nothing changed during it. + const frozen = moving().filter(f => f.t < BLOCK.from || f.t > BLOCK.to); + const proof = busyWindowProof(frozen, BLOCK); + expect(proof.changed).toBe(0); + expect(proof.verdict.pass).toBe(false); + expect(proof.verdict.failures.join('\n')).toMatch(/median frame interval/); + expect(proof.verdict.failures.join('\n')).toMatch(/longest stall/); + expect(proof.verdict.failures.join('\n')).toMatch(/expected frames moved/); + }); + + it('fails a half-rate animation (median interval above display rate)', () => { + const odd = moving().filter((_, i) => i % 2 === 1); + expect(busyWindowProof(odd, BLOCK).verdict.failures.join('\n')).toMatch( + /median frame interval/, + ); + }); +}); + +describe('busyWindowProof settledWithin', () => { + it('ends the judgment at a rounded easing tail, which would read as a stall', () => { + // An ease-out to y = 24 as the screencast reports it: whole pixels, so the + // tail repeats values while it still moves. + const ys = [ + 120, 100, 82, 67, 55, 46, 39, 34, 30, 28, 27, 26, 26, 26, 25, 25, 24, 24, + 24, 24, + ]; + const frames = ys.map( + (y, i) => + ({ + t: 1000 + i * FRAME, + box: {x: 0, y, width: 10, height: 10}, + }) as TrackedFrame, + ); + const window = {from: 1000, to: 1000 + ys.length * FRAME}; + const loose = busyWindowProof(frames, window, { + settledWithin: 2, + marginMs: 0, + }); + expect(loose.verdict.failures).toEqual([]); + const strict = busyWindowProof(frames, window, { + settledWithin: 1, + marginMs: 0, + }); + expect(strict.longestStallFrames).toBeGreaterThanOrEqual(3); + }); +}); + +describe('inPresentationOrder', () => { + it('sorts frames delivered out of order, so motion stays monotonic', () => { + const frames = moving(); + const k = at(1040); + const shuffled = [...frames]; + [shuffled[k], shuffled[k + 1]] = [shuffled[k + 1], shuffled[k]]; + expect(busyWindowProof(shuffled, BLOCK).monotonic).toBe(false); + const proof = busyWindowProof(inPresentationOrder(shuffled), BLOCK); + expect(proof.monotonic).toBe(true); + expect(proof.verdict.failures).toEqual([]); + }); +}); diff --git a/internal/interaction-harness/src/busy.ts b/internal/interaction-harness/src/busy.ts new file mode 100644 index 0000000000000..2f98844603496 --- /dev/null +++ b/internal/interaction-harness/src/busy.ts @@ -0,0 +1,283 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file busy.ts + * @input Uses a Playwright `Page` + * @output `blockMainThread` — a synchronous busy loop on the page's main thread + * that reports exactly when it ran — and `busyWindowProof`, which reads frame + * evidence for that window. + * @position The busy-main-thread proof: an animation that runs on the + * compositor keeps advancing at display rate while JavaScript holds the main + * thread; one driven from the main thread (rAF, style writes, layout + * properties) freezes for the whole block. Pair it with ./frames + * (`startScreencast` + `trackMarker`), which observes off the main thread. + * + * SYNC: Keep the README's busy-main-thread section equal to this contract. + */ + +import type {Locator, Page} from '@playwright/test'; +import {frameStats, type FrameStats, type TrackedFrame} from './frames'; + +export interface BusyWindow { + /** Epoch ms when the loop started and ended, on the page's own clock. */ + readonly from: number; + readonly to: number; +} + +/** + * Holds the page's main thread for `ms` (a synchronous loop: no task, frame, + * rAF, event or microtask runs meanwhile). Resolves with the window it held. + * + * Block MID-FLIGHT only: start the gesture or animation first. A touch that + * starts while the main thread is busy does not scroll or begin a gesture in + * headless Chromium, even with only passive listeners, and an animation that + * has not started cannot start (see `waitForAnimation`). Finger tracking + * under a busy main thread is a simulator proof, not a Playwright one. + * + * With `during`, the page itself waits until an animation on that element (or + * inside it) has run `minProgressMs`, then blocks in the same frame. A round + * trip from the test would start the block later the busier the machine is, + * leaving less of a short animation to judge. + */ +export async function blockMainThread( + page: Page, + ms: number, + { + during, + minProgressMs = 16, + timeout = 5_000, + }: {during?: Locator; minProgressMs?: number; timeout?: number} = {}, +): Promise { + if (during == null) { + return page.evaluate(duration => { + const from = performance.timeOrigin + performance.now(); + const end = performance.now() + duration; + let spins = 0; + while (performance.now() < end) { + spins += 1; + } + return {from, to: performance.timeOrigin + performance.now(), spins}; + }, ms); + } + return during.evaluate( + (element, {duration, minProgress, wait}) => + new Promise((resolve, reject) => { + const deadline = performance.now() + wait; + const tick = () => { + const running = element + .getAnimations({subtree: true}) + .some( + a => + a.playState === 'running' && + Number(a.currentTime ?? 0) >= minProgress, + ); + if (running) { + // The same loop as above, in the frame that saw the animation. + const from = performance.timeOrigin + performance.now(); + const end = performance.now() + duration; + while (performance.now() < end) { + // Busy: nothing else runs. + } + resolve({from, to: performance.timeOrigin + performance.now()}); + } else if (performance.now() > deadline) { + reject(new Error('blockMainThread: no animation started')); + } else { + requestAnimationFrame(tick); + } + }; + requestAnimationFrame(tick); + }), + {duration: ms, minProgress: minProgressMs, wait: timeout}, + ); +} + +export interface BusyVerdict { + readonly pass: boolean; + /** Each criterion the window missed, with its measured value and limit. */ + readonly failures: readonly string[]; +} + +export interface BusyProof extends FrameStats { + readonly window: BusyWindow; + /** The part of the window judged: inside the margins, up to `settledWithin`. */ + readonly judged: {readonly from: number; readonly to: number}; + /** + * The screencast's frame interval, from the frames around the window as well + * as inside it (a short window has few intervals; a frozen one has none). + */ + readonly cadenceMs: number; + /** + * Whole frame intervals in the window at the measured cadence: the frames + * the display shows there at least (one more, depending on phase). + */ + readonly expectedFrames: number; + /** + * Longest time inside the window with no presented change, the window's own + * edges included. A frozen animation presents no new frame at all (nothing + * on screen changed, so the compositor sends none): its stall is the whole + * window. + */ + readonly longestStallMs: number; + /** + * `longestStallMs` in screencast intervals, rounded: 1 is no frame missed, + * 2 is one dropped frame. Rounding absorbs timestamp jitter. + */ + readonly longestStallFrames: number; + /** + * Frames in the window that presented a new position (the first compared + * with the last frame before the window). + */ + readonly moved: number; + /** `moved` over `expectedFrames`. */ + readonly movedFraction: number; + /** The judgment against the limits (see `busyWindowProof`). */ + readonly verdict: BusyVerdict; + /** One line with the raw numbers, for an annotation or a failure message. */ + readonly summary: string; +} + +/** + * Frame evidence for a busy window: how many frames the screencast presented + * inside it, how many of them moved the tracked axis, and the longest stall. + * The window is trimmed by `marginMs` at both ends so the frame the block + * started on and the one it ended on are not counted. + * + * The verdict tolerates what a loaded machine does to a screencast (a dropped + * frame or two) and nothing a main-thread animation does (it presents no + * frame for the whole block). It passes when: + * - the median interval between presented frames is at most + * `maxMedianFrameMs` (display rate); + * - no stall spans more than `maxStallFrames` screencast intervals; + * - at least `minMovedFraction` of the frames the window should present at + * that cadence moved the tracked axis (a short window may still miss one). + */ +export function busyWindowProof( + tracked: readonly TrackedFrame[], + window: BusyWindow, + { + axis = 'y', + marginMs = 20, + settledWithin, + maxMedianFrameMs = 20, + maxStallFrames = 3, + minMovedFraction = 0.9, + }: { + axis?: 'x' | 'y' | 'width' | 'height'; + marginMs?: number; + /** Display rate: the most the median frame interval may be, ms. */ + maxMedianFrameMs?: number; + /** The most screencast intervals one stall may span (3 = two dropped frames). */ + maxStallFrames?: number; + /** + * The least fraction of the expected frames that must move. One missed + * frame is always allowed, so a window of under ten frames is not judged + * stricter than a long one. + */ + minMovedFraction?: number; + /** + * Stop judging once the value is within this many px of where it ends. + * The screencast reports whole CSS pixels, so an easing tail that still + * moves shows plateaus (26, 26, 26 against a final 24) that would read + * as a stall: 2 is the usual value. + */ + settledWithin?: number; + } = {}, +): BusyProof { + let end = window.to - marginMs; + if (settledWithin != null) { + const valued = tracked.flatMap(f => + f.box == null ? [] : [{t: f.t, value: f.box[axis]}], + ); + const final = valued.length > 0 ? valued[valued.length - 1].value : null; + const arrived = valued.find( + f => + f.t >= window.from && + final != null && + Math.abs(f.value - final) <= settledWithin, + ); + if (arrived != null) { + end = Math.min(end, arrived.t); + } + } + const inner = {from: window.from + marginMs, to: end}; + const stats = frameStats( + tracked.map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})), + inner, + ); + const values = tracked + .filter(f => f.t >= inner.from && f.t <= inner.to) + .map(f => ({t: f.t, value: f.box == null ? null : f.box[axis]})); + const before = [...tracked].reverse().find(f => f.t < inner.from); + let previous: number | null = before?.box == null ? null : before.box[axis]; + const changes: number[] = []; + for (const v of values) { + if ( + v.value != null && + previous != null && + Math.abs(v.value - previous) > 0.25 + ) { + changes.push(v.t); + } + if (v.value != null) { + previous = v.value; + } + } + const edges = [inner.from, ...changes, inner.to]; + const longestStallMs = Math.max( + ...edges.slice(1).map((t, i) => t - edges[i]), + ); + const around = tracked + .filter(f => f.t >= inner.from - 250 && f.t <= inner.to + 250) + .map(f => f.t); + const intervals = around + .slice(1) + .map((t, i) => t - around[i]) + .sort((a, b) => a - b); + const cadence = + intervals.length >= 4 + ? intervals[Math.floor(intervals.length / 2)] + : stats.medianFrameMs || 1000 / 60; + const expectedFrames = Math.floor((inner.to - inner.from) / cadence); + const longestStallFrames = Math.round(longestStallMs / cadence); + const moved = changes.length; + const movedFraction = moved / Math.max(1, expectedFrames); + const failures: string[] = []; + if (stats.frames === 0 || stats.medianFrameMs > maxMedianFrameMs) { + failures.push( + `median frame interval ${stats.medianFrameMs.toFixed(1)} ms > ${maxMedianFrameMs} ms (${stats.frames} frames presented)`, + ); + } + if (longestStallFrames > maxStallFrames) { + failures.push( + `longest stall ${longestStallMs.toFixed(1)} ms = ${longestStallFrames} intervals > ${maxStallFrames}`, + ); + } + const allowedMisses = Math.max( + 1, + Math.floor((1 - minMovedFraction) * expectedFrames + 1e-9), + ); + if (expectedFrames - moved > allowedMisses) { + failures.push( + `${moved} of ${expectedFrames} expected frames moved (${(movedFraction * 100).toFixed(0)}% < ${(minMovedFraction * 100).toFixed(0)}%)`, + ); + } + const summary = + `${(inner.to - inner.from).toFixed(0)} ms judged at ${cadence.toFixed(1)} ms/frame: ${stats.frames} frames presented, ` + + `${moved}/${expectedFrames} moved (${(movedFraction * 100).toFixed(0)}%), ` + + `median ${stats.medianFrameMs.toFixed(1)} ms, ` + + `longest stall ${longestStallMs.toFixed(1)} ms (${longestStallFrames} intervals), ` + + `${stats.monotonic ? 'monotonic' : 'not monotonic'}`; + return { + ...stats, + window, + judged: inner, + cadenceMs: cadence, + longestStallMs, + expectedFrames, + longestStallFrames, + moved, + movedFraction, + verdict: {pass: failures.length === 0, failures}, + summary, + }; +} diff --git a/internal/interaction-harness/src/curves.ts b/internal/interaction-harness/src/curves.ts new file mode 100644 index 0000000000000..5e9a04511b453 --- /dev/null +++ b/internal/interaction-harness/src/curves.ts @@ -0,0 +1,467 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file curves.ts + * @input Uses a Playwright `Page`, a `Locator` for the animated element, and + * the Web Animations API in the page (CSS transitions and CSS animations are + * Web Animations too) + * @output `holdAnimations` (pause every running animation and describe it), + * `seekAnimations` (move them all to t ms), `sampleCurve` (the presented + * transform/opacity at each t, with the individual `translate`, `rotate` and + * `scale` properties composed in), `releaseAnimations`, `curveOf`, which + * picks one channel out of the samples for ./math's `compareCurve`, and the + * animation-state waits `waitForAnimation` and `waitForRest`. + * @position Curve sampling: the exact presented value of an animation at a + * fixed time, independent of frame timing, so a curve can be compared with + * the UIKit reference to a fraction of a pixel. A paused animation's effect + * is applied at its current time and `getComputedStyle` reads it. + * + * SYNC: Keep ./storyboard.ts using these for its animation filmstrips. + */ + +import type {Locator, Page} from '@playwright/test'; +import {parseTransform, type TimedValue, type Transform2D} from './math'; + +/** How far inside its end a seek past an animation's end lands, ms. */ +export const END_INSET_MS = 0.5; + +/** + * Installs `window.__harnessCurveEnd(animation)`: the local time (ms, delay + * included) where the animation's effect stops changing. An animation whose + * keyframes end in a run of identical values (an end value held so that a + * later interruption can read it) ends where that run starts; any other ends + * at its end time. + */ +async function installCurveEnd(page: Page): Promise { + await page.evaluate(() => { + const w = window as unknown as { + __harnessCurveEnd?: (animation: Animation) => number; + }; + if (w.__harnessCurveEnd) { + return; + } + const timingKeys = new Set([ + 'offset', + 'computedOffset', + 'easing', + 'composite', + ]); + const values = (k: ComputedKeyframe) => + JSON.stringify( + Object.keys(k) + .filter(p => !timingKeys.has(p)) + .sort() + .map(p => [p, String(k[p])]), + ); + w.__harnessCurveEnd = animation => { + const effect = animation.effect as KeyframeEffect | null; + const timing = effect?.getComputedTiming(); + const delay = Number(timing?.delay ?? 0); + const active = Number(timing?.activeDuration ?? 0); + if ( + effect == null || + !Number.isFinite(active) || + Number(timing?.iterations ?? 1) !== 1 + ) { + return delay + active; + } + const frames = effect.getKeyframes(); + let first = frames.length - 1; + while (first > 0 && values(frames[first - 1]) === values(frames[first])) { + first -= 1; + } + return delay + Number(frames[first]?.computedOffset ?? 1) * active; + }; + }); +} + +export interface HeldAnimation { + readonly index: number; + /** `CSSTransition`, `CSSAnimation` or `Animation` (script-created). */ + readonly kind: string; + /** Transitioned property or animation name, when CSS made it. */ + readonly name: string | null; + readonly target: string | null; + readonly durationMs: number; + readonly delayMs: number; + /** + * Where its curve ends, ms from its start (delay included): before a held + * end value, if its keyframes end in one; otherwise its end time. + */ + readonly curveEndMs: number; + readonly easing: string; + readonly keyframes: readonly Record[]; + /** Where it was when held, ms. */ + readonly heldAtMs: number | null; +} + +/** + * Pauses every animation in the document (optionally only those whose target + * is inside `scope`) and returns what was held. Later `seekAnimations` calls + * move exactly this set. + */ +export async function holdAnimations( + page: Page, + scope?: Locator, +): Promise { + const scopeHandle = scope ? await scope.elementHandle() : null; + await installCurveEnd(page); + return page.evaluate(root => { + const curveEnd = ( + window as unknown as {__harnessCurveEnd: (a: Animation) => number} + ).__harnessCurveEnd; + const describe = (node: Element | null) => + node == null + ? null + : node.tagName.toLowerCase() + + (node.id ? `#${node.id}` : '') + + (node.classList.length + ? `.${[...node.classList].slice(0, 2).join('.')}` + : ''); + const all = document.getAnimations().filter(animation => { + const target = + (animation.effect as KeyframeEffect | null)?.target ?? null; + return ( + root == null || + (target != null && (root === target || root.contains(target))) + ); + }); + const held = all.map(animation => { + animation.pause(); + return animation; + }); + (window as unknown as {__harnessHeld: Animation[]}).__harnessHeld = held; + return held.map((animation, index) => { + const effect = animation.effect as KeyframeEffect | null; + const timing = effect?.getComputedTiming(); + const cssName = + 'transitionProperty' in animation + ? (animation as unknown as {transitionProperty: string}) + .transitionProperty + : 'animationName' in animation + ? (animation as unknown as {animationName: string}).animationName + : null; + return { + index, + kind: animation.constructor.name, + name: cssName, + target: describe((effect?.target as Element | null) ?? null), + durationMs: Number(timing?.duration ?? 0), + delayMs: Number(timing?.delay ?? 0), + curveEndMs: curveEnd(animation), + easing: String(effect?.getTiming().easing ?? 'linear'), + keyframes: (effect?.getKeyframes() ?? []).map( + (k): Record => ({...k}), + ), + heldAtMs: + animation.currentTime == null ? null : Number(animation.currentTime), + }; + }); + }, scopeHandle); +} + +/** + * Moves every held animation to `timeMs` (from its own start, delay included). + * A time at or past an animation's end is clamped to END_INSET_MS before it: + * an animation without `fill` stops applying its effect exactly at the end, + * and the frame wanted there is the last one it presents. (Half a millisecond, + * not less: engines round `currentTime` writes to their timer resolution.) + */ +export async function seekAnimations( + page: Page, + timeMs: number, +): Promise { + await page.evaluate( + ({t, inset}) => { + for (const animation of ( + window as unknown as {__harnessHeld?: Animation[]} + ).__harnessHeld ?? []) { + const end = Number( + animation.effect?.getComputedTiming().endTime ?? Infinity, + ); + animation.currentTime = Math.min(t, end - inset); + } + }, + {t: timeMs, inset: END_INSET_MS}, + ); +} + +/** Resumes (default) or finishes every held animation. */ +export async function releaseAnimations( + page: Page, + {finish = false}: {finish?: boolean} = {}, +): Promise { + await page.evaluate(shouldFinish => { + const held = + (window as unknown as {__harnessHeld?: Animation[]}).__harnessHeld ?? []; + for (const animation of held) { + if (shouldFinish) { + animation.finish(); + } else { + animation.play(); + } + } + (window as unknown as {__harnessHeld?: Animation[]}).__harnessHeld = []; + }, finish); +} + +export interface CurveSample { + readonly t: number; + /** + * The presented 2D transform: `translate` · `rotate` · `scale` · + * `transform`, composed in the order CSS applies them. + */ + readonly transform: Transform2D; + readonly opacity: number; + /** Raw computed values, for the JSON trace. */ + readonly raw: { + readonly transform: string; + readonly translate: string; + readonly rotate: string; + readonly scale: string; + readonly opacity: string; + }; +} + +/** + * The presented transform and opacity of `locator` at each time (ms from the + * held animations' start). Hold the animations first. An animation of the + * individual `translate`, `rotate` or `scale` properties is included: they + * are composed with `transform` into one matrix. + */ +export async function sampleCurve( + locator: Locator, + times: readonly number[], +): Promise { + const raw = await locator.evaluate( + (element, {ts, inset}) => { + const held = + (window as unknown as {__harnessHeld?: Animation[]}).__harnessHeld ?? + []; + if (held.length === 0) { + throw new Error( + 'sampleCurve: no held animations (call holdAnimations first)', + ); + } + return ts.map(t => { + for (const animation of held) { + // Just inside the end: see seekAnimations. + const end = Number( + animation.effect?.getComputedTiming().endTime ?? Infinity, + ); + animation.currentTime = Math.min(t, end - inset); + } + const style = getComputedStyle(element); + // translate · rotate · scale · transform, as CSS applies them (all + // about the same transform origin, which `transform` excludes too). + const box = element as HTMLElement; + const length = (value: string, size: number) => + value.endsWith('%') + ? (parseFloat(value) / 100) * size + : parseFloat(value) || 0; + const matrix = new DOMMatrix(); + if (style.translate !== 'none') { + const [x = '0px', y = '0px'] = style.translate.split(' '); + matrix.translateSelf( + length(x, box.offsetWidth ?? 0), + length(y, box.offsetHeight ?? 0), + ); + } + if (style.rotate !== 'none') { + // "45deg", "z 45deg" or "0 0 1 45deg": the angle is the last token. + const angle = style.rotate.split(' ').at(-1) ?? '0deg'; + const turns: Record = { + deg: 1, + rad: 180 / Math.PI, + grad: 0.9, + turn: 360, + }; + const unit = /[a-z]+$/.exec(angle)?.[0] ?? 'deg'; + matrix.rotateSelf(parseFloat(angle) * (turns[unit] ?? 1)); + } + if (style.scale !== 'none') { + const [sx = 1, sy = sx] = style.scale.split(' ').map(Number); + matrix.scaleSelf(sx, sy); + } + if (style.transform !== 'none') { + matrix.multiplySelf(new DOMMatrix(style.transform)); + } + const {a, b, c, d, e, f} = matrix; + return { + t, + matrix: `matrix(${a}, ${b}, ${c}, ${d}, ${e}, ${f})`, + transform: style.transform, + translate: style.translate, + rotate: style.rotate, + scale: style.scale, + opacity: style.opacity, + }; + }); + }, + {ts: [...times], inset: END_INSET_MS}, + ); + return raw.map(r => ({ + t: r.t, + transform: parseTransform(r.matrix), + opacity: Number(r.opacity), + raw: { + transform: r.transform, + translate: r.translate, + rotate: r.rotate, + scale: r.scale, + opacity: r.opacity, + }, + })); +} + +export type Channel = + 'translateX' | 'translateY' | 'scaleX' | 'scaleY' | 'rotate' | 'opacity'; + +/** One channel of a sampled curve, ready for `compareCurve`. */ +export function curveOf( + samples: readonly CurveSample[], + channel: Channel, +): TimedValue[] { + return samples.map(s => ({ + t: s.t, + value: channel === 'opacity' ? s.opacity : s.transform[channel], + })); +} + +export interface ActiveAnimation { + readonly kind: string; + readonly name: string | null; + /** + * Epoch ms (page clock) when the animation's curve starts and ends. A held + * end value (see `HeldAnimation.curveEndMs`) is not part of the curve. + */ + readonly from: number; + readonly to: number; + readonly durationMs: number; + readonly easing: string; + readonly keyframes: readonly Record[]; +} + +/** + * Waits until an animation is running on `locator`'s element (or inside it) + * and has presented at least one frame, and returns its active interval on + * the page clock. Use it before blocking the main thread: an animation that + * has not started yet cannot start while the thread is blocked. + */ +export async function waitForAnimation( + locator: Locator, + { + timeout = 5_000, + minProgressMs = 1, + }: {timeout?: number; minProgressMs?: number} = {}, +): Promise { + const deadline = Date.now() + timeout; + await installCurveEnd(locator.page()); + for (;;) { + const found = await locator.evaluate((element, minProgress) => { + const curveEnd = ( + window as unknown as {__harnessCurveEnd: (a: Animation) => number} + ).__harnessCurveEnd; + return element + .getAnimations({subtree: true}) + .filter( + a => + a.playState === 'running' && + a.startTime != null && + Number(a.currentTime ?? 0) >= minProgress, + ) + .map(a => { + const effect = a.effect as KeyframeEffect | null; + const timing = effect?.getComputedTiming(); + const startedAt = performance.timeOrigin + Number(a.startTime); + const name = + 'transitionProperty' in a + ? (a as unknown as {transitionProperty: string}) + .transitionProperty + : 'animationName' in a + ? (a as unknown as {animationName: string}).animationName + : null; + return { + kind: a.constructor.name, + name, + from: startedAt + Number(timing?.delay ?? 0), + to: startedAt + curveEnd(a), + durationMs: Number(timing?.duration ?? 0), + easing: String(effect?.getTiming().easing ?? 'linear'), + keyframes: (effect?.getKeyframes() ?? []).map( + (k): Record => ({...k}), + ), + }; + }); + }, minProgressMs); + if (found.length > 0) { + return found; + } + if (Date.now() > deadline) { + throw new Error('waitForAnimation: no animation started on the element'); + } + await new Promise(resolve => setTimeout(resolve, 4)); + } +} + +/** + * Resolves once `locator`'s element is at rest: no animation on it (or inside + * it) is pending or still before the end of its curve, and its presented + * `transform`, `translate`, `rotate`, `scale` and `opacity` (and `::backdrop` + * opacity) have held still for `frames` consecutive frames. + * + * It judges curves and presented values, not `playState`: an animation that + * holds its end value (a run of identical final keyframes kept so a later + * interruption can read it) still reports `running` while nothing moves. + */ +export async function waitForRest( + locator: Locator, + {timeout = 5_000, frames = 3}: {timeout?: number; frames?: number} = {}, +): Promise { + await installCurveEnd(locator.page()); + await locator.evaluate( + (element, {ms, need}) => + new Promise((resolve, reject) => { + const deadline = performance.now() + ms; + const curveEnd = ( + window as unknown as {__harnessCurveEnd: (a: Animation) => number} + ).__harnessCurveEnd; + const read = () => { + const style = getComputedStyle(element); + const backdrop = getComputedStyle(element, '::backdrop').opacity; + return [ + style.transform, + style.translate, + style.rotate, + style.scale, + style.opacity, + backdrop, + ].join(' | '); + }; + let last = read(); + let still = 0; + const tick = () => { + const now = read(); + const moving = element + .getAnimations({subtree: true}) + .some( + a => + a.pending || + (a.playState === 'running' && + Number(a.currentTime ?? 0) < curveEnd(a)), + ); + still = !moving && now === last ? still + 1 : 0; + last = now; + if (still >= need) { + resolve(); + } else if (performance.now() > deadline) { + reject(new Error(`waitForRest: still moving (${now})`)); + } else { + requestAnimationFrame(tick); + } + }; + requestAnimationFrame(tick); + }), + {ms: timeout, need: frames}, + ); +} diff --git a/internal/interaction-harness/src/frames.ts b/internal/interaction-harness/src/frames.ts new file mode 100644 index 0000000000000..f7497e211029c --- /dev/null +++ b/internal/interaction-harness/src/frames.ts @@ -0,0 +1,493 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file frames.ts + * @input Uses a Playwright `Page`, a `Locator` for the element under test, and + * the engine's screencast (Chromium CDP `Page.startScreencast`; WebKit's + * `Screencast` domain) + * @output `startScreencast` (timestamped composited frames), `markElement` + * (a marker outline the frames can find), `trackMarker` (per-frame marker + * box), `sampleRects` (per-rAF rect sampling), `frameStats`. + * @position Frame evidence for motion proofs. The screencast is captured off + * the page's main thread, from what the compositor presents, so it keeps + * delivering frames while a script blocks the main thread: that is what makes + * the busy-main-thread proof possible. Rect sampling is the cheap path when + * the main thread is free. + * + * Marker tracking needs no image library: frames are decoded and scanned in a + * scratch page of the same browser (createImageBitmap + canvas), so the + * harness adds no dependency. + * + * SYNC: Keep the README's frame-evidence section equal to what this measures. + */ + +import type {CDPSession, Locator, Page} from '@playwright/test'; +import {webkitPageSession} from './touch'; + +export interface CastFrame { + /** Presentation time, epoch ms (the compositor's frame timestamp). */ + readonly t: number; + /** Base64 PNG/JPEG. */ + readonly data: string; + readonly format: 'png' | 'jpeg'; + /** CSS px of the viewport the frame shows. */ + readonly viewportWidth: number; + readonly viewportHeight: number; +} + +export interface Screencast { + /** Stops and returns every frame captured, in order. */ + stop(): Promise; +} + +export interface ScreencastOptions { + /** Lossless frames by default: marker colours survive PNG, not JPEG. */ + readonly format?: 'png' | 'jpeg'; + readonly quality?: number; +} + +/** + * Starts capturing composited frames. Chromium delivers one per presented + * frame (16.7 ms at 60 Hz), timestamped by the compositor; WebKit's own + * screencast delivers one per composited update (about every 14 ms in the + * Linux build), stamped on its monotonic clock and rebased here onto the wall + * clock. Both keep delivering while the page's main thread is blocked, and + * neither sends a frame when nothing on screen changed. + */ +export async function startScreencast( + page: Page, + options: ScreencastOptions = {}, +): Promise { + const name = page.context().browser()?.browserType().name(); + const viewport = page.viewportSize() ?? {width: 0, height: 0}; + if (name === 'chromium') { + return chromiumScreencast(page, viewport, options); + } + if (name === 'webkit') { + return webkitScreencast(page, viewport, options); + } + throw new Error( + `startScreencast: unsupported browser ${name ?? '(unknown)'}`, + ); +} + +async function chromiumScreencast( + page: Page, + viewport: {width: number; height: number}, + {format = 'png', quality = 90}: ScreencastOptions, +): Promise { + const cdp: CDPSession = await page.context().newCDPSession(page); + const frames: CastFrame[] = []; + const onFrame = (frame: { + data: string; + sessionId: number; + metadata: {timestamp?: number; deviceWidth: number; deviceHeight: number}; + }) => { + frames.push({ + t: (frame.metadata.timestamp ?? Date.now() / 1000) * 1000, + data: frame.data, + format, + viewportWidth: frame.metadata.deviceWidth || viewport.width, + viewportHeight: frame.metadata.deviceHeight || viewport.height, + }); + cdp + .send('Page.screencastFrameAck', {sessionId: frame.sessionId}) + .catch(() => {}); + }; + cdp.on('Page.screencastFrame', onFrame); + // One CSS px per frame px keeps decoding cheap; everyNthFrame 1 = every presented frame. + await cdp.send('Page.startScreencast', { + format, + quality, + maxWidth: viewport.width, + maxHeight: viewport.height, + everyNthFrame: 1, + }); + return { + async stop() { + await cdp.send('Page.stopScreencast'); + cdp.off('Page.screencastFrame', onFrame); + await cdp.detach().catch(() => {}); + return inPresentationOrder(frames); + }, + }; +} + +async function webkitScreencast( + page: Page, + viewport: {width: number; height: number}, + {quality = 90}: ScreencastOptions, +): Promise { + const session = webkitPageSession(page); + const frames: CastFrame[] = []; + let generation = 0; + const onFrame = (payload: unknown) => { + const frame = payload as {data: string; timestamp: number}; + frames.push({ + // WebKit stamps frames in seconds on the monotonic clock; rebased below. + t: frame.timestamp * 1000, + data: frame.data, + format: 'jpeg', + viewportWidth: viewport.width, + viewportHeight: viewport.height, + }); + session.send('Screencast.screencastFrameAck', {generation}).catch(() => {}); + }; + session.on('Screencast.screencastFrame', onFrame); + const started = (await session.send('Screencast.startScreencast', { + width: viewport.width, + height: viewport.height, + toolbarHeight: 0, + quality, + })) as {generation: number}; + generation = started.generation; + const wallAtStart = Date.now(); + return { + async stop() { + await session.send('Screencast.stopScreencast'); + session.off?.('Screencast.screencastFrame', onFrame); + if (frames.length === 0) { + return frames; + } + const ordered = inPresentationOrder(frames); + // Rebase the monotonic stamps onto the wall clock (first frame ≈ start). + const offset = wallAtStart - ordered[0].t; + return ordered.map(f => ({...f, t: f.t + offset})); + }, + }; +} + +/** + * Frames sorted by their presentation timestamp. A loaded machine can deliver + * screencast frames out of order (a frame stamped 35 ms arriving after one + * stamped 46 ms); judged in arrival order, a steadily moving element would + * appear to step backwards. + */ +export function inPresentationOrder( + frames: readonly T[], +): T[] { + return [...frames].sort((a, b) => a.t - b.t); +} + +/** The marker colour: magenta never occurs in Astryx themes. */ +export const MARKER_RGB: readonly [number, number, number] = [255, 0, 255]; + +export interface MarkerOptions { + readonly rgb?: readonly [number, number, number]; + /** + * `chip` (default): a small square child pinned inside the element's + * top-left corner, above every sibling (z-index max), so nothing the + * component paints can cover it. The element must be its containing block: + * a static element is made `position: relative` while marked. + * `outline`: an inset outline on the element itself (no DOM insertion), for + * elements that cannot take a child; children with their own layers can + * cover parts of it. + */ + readonly mode?: 'chip' | 'outline'; + /** Chip size or outline width, CSS px. */ + readonly size?: number; + /** Chip inset from the element's top-left corner, CSS px. */ + readonly inset?: number; +} + +/** + * Marks the element so screencast frames can find it: the mark moves, scales + * and fades with the element's own layer and never changes its layout. + * Returns an undo. + */ +export async function markElement( + locator: Locator, + options: MarkerOptions = {}, +): Promise<() => Promise> { + const { + rgb = MARKER_RGB, + mode = 'chip', + size = mode === 'chip' ? 16 : 6, + inset = 8, + } = options; + const id = `m${Math.random().toString(36).slice(2, 8)}`; + await locator.evaluate( + (element, {id: markerId, color, size: px, inset: offset, mode: kind}) => { + const el = element as HTMLElement; + el.setAttribute('data-harness-marker', markerId); + if (kind === 'outline') { + const style = document.createElement('style'); + style.setAttribute('data-harness-marker-style', markerId); + style.textContent = `[data-harness-marker="${markerId}"]{outline:${px}px solid rgb(${color.join(',')}) !important;outline-offset:-${px}px !important;}`; + document.head.append(style); + return; + } + if (getComputedStyle(el).position === 'static') { + el.setAttribute('data-harness-marker-position', el.style.position); + el.style.position = 'relative'; + } + const chip = document.createElement('div'); + chip.setAttribute('data-harness-marker-chip', markerId); + chip.setAttribute('aria-hidden', 'true'); + chip.style.cssText = `position:absolute;top:${offset}px;left:${offset}px;width:${px}px;height:${px}px;background:rgb(${color.join(',')});z-index:2147483647;pointer-events:none;`; + el.append(chip); + }, + {id, color: [...rgb], size, inset, mode}, + ); + return async () => { + await locator.page().evaluate(markerId => { + const el = document.querySelector( + `[data-harness-marker="${markerId}"]`, + ); + if (el?.hasAttribute('data-harness-marker-position')) { + el.style.position = + el.getAttribute('data-harness-marker-position') ?? ''; + el.removeAttribute('data-harness-marker-position'); + } + el?.removeAttribute('data-harness-marker'); + document + .querySelector(`[data-harness-marker-chip="${markerId}"]`) + ?.remove(); + document + .querySelector(`[data-harness-marker-style="${markerId}"]`) + ?.remove(); + }, id); + }; +} + +export interface Box { + readonly x: number; + readonly y: number; + readonly width: number; + readonly height: number; +} + +export interface TrackedFrame { + /** Epoch ms. */ + readonly t: number; + /** The marker's bounding box in CSS px, or null when it is not on screen. */ + readonly box: Box | null; + /** Marker pixels found (a fading element finds fewer at the same tolerance). */ + readonly pixels: number; +} + +/** + * Finds the marker in every frame. Decoding runs in a scratch page of the same + * browser, so neither the page under test nor Node needs an image decoder. + */ +export async function trackMarker( + page: Page, + frames: readonly CastFrame[], + { + rgb = MARKER_RGB, + tolerance = 40, + }: {rgb?: readonly [number, number, number]; tolerance?: number} = {}, +): Promise { + const scratch = await page.context().newPage(); + try { + const out: TrackedFrame[] = []; + // Batches keep each evaluate payload moderate. + for (let i = 0; i < frames.length; i += 12) { + const batch = frames.slice(i, i + 12); + const boxes = await scratch.evaluate( + async ({batch: items, color, tol}) => { + const results: { + box: {x: number; y: number; width: number; height: number} | null; + pixels: number; + }[] = []; + for (const item of items) { + const blob = await ( + await fetch(`data:image/${item.format};base64,${item.data}`) + ).blob(); + const bitmap = await createImageBitmap(blob); + const canvas = new OffscreenCanvas(bitmap.width, bitmap.height); + const context = canvas.getContext('2d'); + if (context == null) { + throw new Error('no 2d context'); + } + context.drawImage(bitmap, 0, 0); + const {data, width, height} = context.getImageData( + 0, + 0, + bitmap.width, + bitmap.height, + ); + let minX = Infinity; + let minY = Infinity; + let maxX = -Infinity; + let maxY = -Infinity; + let pixels = 0; + for (let y = 0; y < height; y += 1) { + for (let x = 0; x < width; x += 1) { + const o = (y * width + x) * 4; + if ( + Math.abs(data[o] - color[0]) <= tol && + Math.abs(data[o + 1] - color[1]) <= tol && + Math.abs(data[o + 2] - color[2]) <= tol + ) { + pixels += 1; + if (x < minX) { + minX = x; + } + if (x > maxX) { + maxX = x; + } + if (y < minY) { + minY = y; + } + if (y > maxY) { + maxY = y; + } + } + } + } + const scale = width / item.viewportWidth; + results.push({ + box: + pixels === 0 + ? null + : { + x: minX / scale, + y: minY / scale, + width: (maxX - minX + 1) / scale, + height: (maxY - minY + 1) / scale, + }, + pixels, + }); + } + return results; + }, + { + batch: batch.map(f => ({ + data: f.data, + format: f.format, + viewportWidth: f.viewportWidth, + })), + color: [...rgb], + tol: tolerance, + }, + ); + boxes.forEach((result, k) => + out.push({t: batch[k].t, box: result.box, pixels: result.pixels}), + ); + } + return out; + } finally { + await scratch.close(); + } +} + +export interface RectSample { + /** Epoch ms of the rAF callback. */ + readonly t: number; + readonly box: Box | null; + readonly transform: string; + readonly opacity: number; +} + +/** + * Samples the element's rect, transform and opacity on every animation frame + * for `durationMs`. Main-thread sampling: it stops while the main thread is + * blocked, which is the point of the screencast path above. + */ +export async function sampleRects( + locator: Locator, + durationMs: number, +): Promise { + return locator.evaluate( + (element, duration) => + new Promise< + { + t: number; + box: {x: number; y: number; width: number; height: number} | null; + transform: string; + opacity: number; + }[] + >(resolve => { + const samples: { + t: number; + box: {x: number; y: number; width: number; height: number} | null; + transform: string; + opacity: number; + }[] = []; + const end = performance.now() + duration; + const tick = (now: number) => { + const rect = element.getBoundingClientRect(); + const style = getComputedStyle(element); + samples.push({ + t: performance.timeOrigin + now, + box: {x: rect.x, y: rect.y, width: rect.width, height: rect.height}, + transform: style.transform, + opacity: Number(style.opacity), + }); + if (now < end) { + requestAnimationFrame(tick); + } else { + resolve(samples); + } + }; + requestAnimationFrame(tick); + }), + durationMs, + ); +} + +export interface FrameStats { + readonly frames: number; + /** Frames whose tracked value differs from the frame before. */ + readonly changed: number; + /** Longest stretch without a change inside the window (an open stall at its end included), ms. */ + readonly longestStallMs: number; + /** Median spacing between frames, ms. */ + readonly medianFrameMs: number; + readonly monotonic: boolean; + readonly first: number | null; + readonly last: number | null; +} + +/** Summarises per-frame values (e.g. the marker's y) inside a time window. */ +export function frameStats( + samples: readonly {readonly t: number; readonly value: number | null}[], + window: {readonly from: number; readonly to: number} = { + from: -Infinity, + to: Infinity, + }, + epsilon = 0.25, +): FrameStats { + const inside = samples.filter( + s => s.t >= window.from && s.t <= window.to && s.value != null, + ) as {t: number; value: number}[]; + let changed = 0; + let increasing = 0; + let decreasing = 0; + let stallStart = inside[0]?.t ?? 0; + let longestStallMs = 0; + for (let i = 1; i < inside.length; i += 1) { + const delta = inside[i].value - inside[i - 1].value; + if (Math.abs(delta) > epsilon) { + changed += 1; + if (delta > 0) { + increasing += 1; + } else { + decreasing += 1; + } + longestStallMs = Math.max(longestStallMs, inside[i].t - stallStart); + stallStart = inside[i].t; + } + } + // A stall still open when the window closes counts too: a frozen animation never changes at all. + if (inside.length > 0) { + longestStallMs = Math.max( + longestStallMs, + inside[inside.length - 1].t - stallStart, + ); + } + const gaps = inside + .slice(1) + .map((s, i) => s.t - inside[i].t) + .sort((a, b) => a - b); + return { + frames: inside.length, + changed, + longestStallMs, + medianFrameMs: gaps.length ? gaps[Math.floor(gaps.length / 2)] : 0, + monotonic: increasing === 0 || decreasing === 0, + first: inside[0]?.value ?? null, + last: inside[inside.length - 1]?.value ?? null, + }; +} diff --git a/internal/interaction-harness/src/harness.interaction.browser.spec.ts b/internal/interaction-harness/src/harness.interaction.browser.spec.ts new file mode 100644 index 0000000000000..030812f5a8947 --- /dev/null +++ b/internal/interaction-harness/src/harness.interaction.browser.spec.ts @@ -0,0 +1,611 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file harness.interaction.browser.spec.ts + * @input Fixture pages written inline (no Storybook), real Chromium/WebKit + * @output Proof that each harness instrument measures what it claims, with a + * negative control beside every positive one: scripted timestamps reach the + * page; a flick flings natively (Chromium); a second finger and a pinch + * arrive as distinct pointers; the screencast shows a compositor animation + * advancing through a blocked main thread while a rAF animation freezes; + * the trace counts zero Layout for a transform animation and one per frame + * for a `top` animation; sampled curves match the oracles, individual + * transform properties included; the animation-state waits report a + * running transition and its end, and judge rest by presented values even + * while an animation holds its end; storyboards write a sheet and a trace + * that follows the curve, and the timer gate holds a component timer until + * the filmstrip is done. + * @position The harness's own test. A component spec trusts these + * instruments; this file is why it may. + */ + +import {expect, test, type Page} from '@playwright/test'; +import { + animationStoryboard, + blockMainThread, + busyWindowProof, + compareCurve, + countBetween, + cubicBezier, + curveOf, + drag, + flick, + freezeTimers, + holdAnimations, + installInputLog, + installTimerGate, + markElement, + merge, + parseTransform, + PARTIAL_LIFT_SKIP, + pinch, + readInputLog, + sampleCurve, + sampleRects, + springStep, + startScreencast, + tap, + thawTimers, + timeline, + touchDriver, + touchStoryboard, + traceRendering, + trackMarker, + waitForAnimation, + waitForRest, + withFinger, +} from './index'; + +const FIXTURE = ` + +
+
+
${Array.from({length: 200}, (_, i) => `
row ${i}
`).join('')}
`; + +// A real navigation (not setContent, whose document.open() erases the window +// listeners the input log installs). +async function fixture(page: Page) { + await installInputLog(page); + await page.route('http://harness.test/', route => + route.fulfill({contentType: 'text/html', body: FIXTURE}), + ); + await page.goto('http://harness.test/'); +} + +test.describe('touch input', () => { + test('the page receives trusted touches and pointers at the scripted positions and times', async ({ + page, + browserName, + }) => { + await fixture(page); + const driver = await touchDriver(page); + const script = drag( + {x: 100, y: 100}, + {x: 100, y: 220}, + {durationMs: 200, hz: 60, restMs: 120}, + ); + await driver.play(script); + const log = await readInputLog(page); + const moves = log.filter(r => r.type === 'pointermove'); + const down = log.find(r => r.type === 'pointerdown'); + const up = log.find(r => r.type === 'pointerup'); + expect(down?.isTrusted).toBe(true); + expect(down?.pointerType).toBe('touch'); + expect(up, JSON.stringify(log.map(r => r.type))).toBeDefined(); + // (Not `cancelable`: with only passive listeners an engine may dispatch + // touchmove as non-blocking, which reads cancelable false.) + expect( + log.filter(r => r.type === 'touchmove').every(r => r.isTrusted), + ).toBe(true); + expect(moves.at(-1)?.y).toBe(220); + if (browserName === 'chromium') { + // The scripted 16.67 ms spacing arrives exactly, and the 120 ms rest + // before the lift is in the timestamps too. + const deltas = moves + .slice(1) + .map((m, i) => m.timeStamp - moves[i].timeStamp); + for (const delta of deltas) { + expect(delta).toBeCloseTo(1000 / 60, 0); + } + expect( + (up?.timeStamp ?? NaN) - (moves.at(-1)?.timeStamp ?? NaN), + ).toBeCloseTo(120, 0); + } else { + // WebKit stamps the real dispatch time: real-time pacing keeps it close. + const deltas = moves + .slice(1) + .map((m, i) => m.timeStamp - moves[i].timeStamp); + const mean = deltas.reduce((a, b) => a + b, 0) / deltas.length; + expect(mean).toBeGreaterThan(14); + expect(mean).toBeLessThan(20); + } + }); + + test('a flick flings a native scroller, farther for a faster finger', async ({ + page, + browserName, + }) => { + test.skip( + browserName !== 'chromium', + 'injected WebKit touches drive no native scrolling (README: engine limits)', + ); + await fixture(page); + const driver = await touchDriver(page); + const scroller = page.locator('#scroller'); + const distanceFor = async (velocity: number) => { + await scroller.evaluate(el => (el.scrollTop = 0)); + await driver.play( + flick( + {x: 200, y: 680}, + {velocity: {x: 0, y: -velocity}, durationMs: 60, hz: 120}, + ), + ); + await expect + .poll( + async () => { + const a = await scroller.evaluate(el => el.scrollTop); + await page.waitForTimeout(150); + return a === (await scroller.evaluate(el => el.scrollTop)); + }, + {timeout: 8_000}, + ) + .toBe(true); + return scroller.evaluate(el => el.scrollTop); + }; + const slow = await distanceFor(800); + const fast = await distanceFor(2400); + expect(slow).toBeGreaterThan(48); // it coasted past the finger's own travel + expect(fast).toBeGreaterThan(slow * 1.8); + }); + + test('lifting one of two fingers releases only that finger', async ({ + page, + }) => { + await fixture(page); + const driver = await touchDriver(page); + test.skip(!driver.partialLifts, PARTIAL_LIFT_SKIP); + const first = drag( + {x: 80, y: 80}, + {x: 80, y: 240}, + {durationMs: 400, hz: 60}, + ); + const second = withFinger(tap({x: 300, y: 120}, {holdMs: 100}), 1, 100); + await driver.play(merge(first, second)); + const log = await readInputLog(page); + const [stays, lifts] = log.filter(r => r.type === 'pointerdown'); + const upOf = (down?: {pointerId?: number}) => + log.find(r => r.type === 'pointerup' && r.pointerId === down?.pointerId); + const lifted = upOf(lifts); + const last = upOf(stays); + if (stays == null || lifted == null || last == null) { + throw new Error('expected two fingers down, each released'); + } + // Released when it lifted (~200 ms in), while the first finger was down. + expect(lifted.timeStamp).toBeLessThan(last.timeStamp - 150); + expect( + log.filter( + r => + r.type === 'pointermove' && + r.pointerId === stays.pointerId && + r.timeStamp > lifted.timeStamp, + ).length, + 'the other finger keeps moving', + ).toBeGreaterThan(5); + // As iOS sends it: the lift's touchend still lists the finger down. + expect(log.filter(r => r.type === 'touchend').map(r => r.touches)).toEqual([ + 1, 0, + ]); + }); + + test('a second finger and a pinch arrive as distinct touch pointers', async ({ + page, + }) => { + await fixture(page); + const driver = await touchDriver(page); + const first = drag( + {x: 80, y: 80}, + {x: 80, y: 200}, + {durationMs: 300, hz: 60}, + ); + const second = withFinger(tap({x: 300, y: 120}, {holdMs: 100}), 1, 120); + await driver.play(merge(first, second)); + const downs = (await readInputLog(page, {clear: true})).filter( + r => r.type === 'pointerdown', + ); + expect(downs).toHaveLength(2); + expect(new Set(downs.map(d => d.pointerId)).size).toBe(2); + + await driver.play( + pinch({ + center: {x: 200, y: 150}, + fromDistance: 60, + toDistance: 200, + durationMs: 200, + }), + ); + const log = await readInputLog(page); + expect(log.filter(r => r.type === 'pointerdown')).toHaveLength(2); + expect( + Math.max( + ...log.filter(r => r.type === 'touchmove').map(r => r.touches ?? 0), + ), + ).toBe(2); + }); +}); + +test.describe('frame evidence', () => { + test('a compositor animation advances every frame through a blocked main thread; a rAF one freezes', async ({ + page, + }, testInfo) => { + await fixture(page); + const box = page.locator('#box'); + await markElement(box, {mode: 'outline'}); + + const run = async (driver: 'compositor' | 'raf') => { + await box.evaluate((el, how) => { + el.getAnimations().forEach(a => a.cancel()); + el.style.transform = 'none'; + if (how === 'compositor') { + el.animate( + [{transform: 'translateX(0px)'}, {transform: 'translateX(280px)'}], + {duration: 1600, easing: 'linear'}, + ); + } else { + const start = performance.now(); + const step = (now: number) => { + const p = Math.min(1, (now - start) / 1600); + el.style.transform = `translateX(${280 * p}px)`; + if (p < 1) { + requestAnimationFrame(step); + } + }; + requestAnimationFrame(step); + } + }, driver); + const cast = await startScreencast(page); + await page.waitForTimeout(200); + const window = await blockMainThread(page, 700); + await page.waitForTimeout(200); + const tracked = await trackMarker(page, await cast.stop()); + return busyWindowProof(tracked, window, {axis: 'x'}); + }; + + const compositor = await run('compositor'); + testInfo.annotations.push({ + type: 'busy-main-thread compositor', + description: compositor.summary, + }); + expect(compositor.verdict.failures, compositor.summary).toEqual([]); + expect(compositor.monotonic).toBe(true); + + const raf = await run('raf'); + testInfo.annotations.push({ + type: 'busy-main-thread rAF control', + description: raf.summary, + }); + // Frozen for the whole block: no presented frame moved (the compositor + // sends none at all when nothing on screen changes), and the verdict that + // passes the compositor animation fails it on every criterion. + expect(raf.changed).toBe(0); + expect(raf.longestStallMs).toBeGreaterThan(600); + expect(raf.verdict.pass).toBe(false); + expect(raf.verdict.failures.join('\n')).toMatch(/longest stall/); + expect(raf.verdict.failures.join('\n')).toMatch(/expected frames moved/); + }); + + test('rect sampling reads one rect per animation frame while the main thread is free', async ({ + page, + browserName, + }) => { + await fixture(page); + const box = page.locator('#box'); + await box.evaluate(el => { + el.animate( + [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}], + {duration: 600, easing: 'linear'}, + ); + }); + const samples = await sampleRects(box, 400); + const xs = samples.map(s => s.box?.x ?? NaN); + const gaps = samples + .slice(1) + .map((s, i) => s.t - samples[i].t) + .sort((a, b) => a - b); + const cadence = gaps[Math.floor(gaps.length / 2)]; + expect(cadence).toBeLessThan(34); + expect(samples.length).toBeGreaterThanOrEqual(10); + if (browserName === 'chromium') { + // One sample per presented frame at 60 Hz. (Headless WebKit's animation + // frames are irregular, so only the cadence is checked there.) + expect(samples.length).toBeGreaterThanOrEqual(20); + } + expect(xs.every((x, i) => i === 0 || x >= xs[i - 1])).toBe(true); + expect(xs.at(-1) ?? 0).toBeGreaterThan(xs[0] + 80); + }); + + test('the trace counts no Layout for a transform animation and one per frame for `top`', async ({ + page, + browserName, + }) => { + test.skip( + browserName !== 'chromium', + 'rendering traces are Chromium-only (README: engine limits)', + ); + await fixture(page); + const box = page.locator('#box'); + // Count the animation's steady middle: starting and ending it may lay out once each. + const animate = async (property: 'transform' | 'top') => { + let started = 0; + const trace = await traceRendering(page, async () => { + started = await box.evaluate((el, prop) => { + const keyframes = + prop === 'transform' + ? [ + {transform: 'translateY(0px)'}, + {transform: 'translateY(200px)'}, + ] + : [{top: '360px'}, {top: '560px'}]; + el.animate(keyframes, {duration: 600}); + return performance.timeOrigin + performance.now(); + }, property); + await page.waitForTimeout(700); + }); + return countBetween(trace, started + 100, started + 500); + }; + const transform = await animate('transform'); + const top = await animate('top'); + expect(transform.Layout ?? 0).toBe(0); + expect(transform.Paint ?? 0).toBe(0); + // ~24 frames in 400 ms: one layout per frame. + expect(top.Layout ?? 0).toBeGreaterThan(18); + }); +}); + +test.describe('curve sampling', () => { + test('a CSS transition samples onto its cubic-bezier to 0.05 px', async ({ + page, + }) => { + await fixture(page); + const box = page.locator('#box'); + await box.evaluate(el => { + el.style.transition = 'transform 400ms cubic-bezier(0.24, 1, 0.4, 1)'; + void (el as HTMLElement).offsetWidth; + el.style.transform = 'translateY(200px)'; + }); + const held = await holdAnimations(page, box); + expect(held).toHaveLength(1); + expect(held[0].kind).toBe('CSSTransition'); + const samples = curveOf( + await sampleCurve(box, timeline(400)), + 'translateY', + ); + const ease = cubicBezier(0.24, 1, 0.4, 1); + const result = compareCurve(samples, t => 200 * ease(t / 400), 0.05); + expect(result.maxError, JSON.stringify(result.worst)).toBeLessThanOrEqual( + 0.05, + ); + }); + + test('a spring baked into linear() matches the SwiftUI spring oracle', async ({ + page, + }) => { + await fixture(page); + const box = page.locator('#box'); + const spring = {response: 0.4, dampingFraction: 0.75}; + const duration = 1200; + await box.evaluate( + (el, {stops, ms}) => { + el.animate( + [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}], + {duration: ms, easing: `linear(${stops})`}, + ); + }, + { + stops: timeline(duration, 4) + .map(t => springStep(spring, t).toFixed(6)) + .join(', '), + ms: duration, + }, + ); + await holdAnimations(page, box); + const samples = curveOf( + await sampleCurve(box, timeline(duration, 1000 / 60)), + 'translateX', + ); + // linear() draws chords between the 4 ms stops: at most h²/8 · |x''| = + // 0.004² / 8 · 200 · (2π / 0.4)² ≈ 0.1 px off the exact oracle. + const result = compareCurve( + samples, + t => 200 * springStep(spring, t), + 0.15, + ); + expect(result.maxError, JSON.stringify(result.worst)).toBeLessThanOrEqual( + 0.15, + ); + expect(Math.max(...samples.map(s => s.value))).toBeGreaterThan(200); // the overshoot is presented + }); +}); + +test.describe('animation state', () => { + test('waitForAnimation reports a running transition and waitForRest resolves after it', async ({ + page, + }) => { + await fixture(page); + const box = page.locator('#box'); + await box.evaluate(el => { + el.style.transition = 'transform 300ms linear'; + void (el as HTMLElement).offsetWidth; + el.style.transform = 'translateY(100px)'; + }); + const [running] = await waitForAnimation(box); + expect(running.kind).toBe('CSSTransition'); + expect(running.name).toBe('transform'); + expect(running.durationMs).toBe(300); + expect(running.to - running.from).toBeCloseTo(300, 0); + await waitForRest(box); + const settledAt = await page.evaluate( + () => performance.timeOrigin + performance.now(), + ); + expect(settledAt).toBeGreaterThanOrEqual(running.to); + const transform = await box.evaluate(el => getComputedStyle(el).transform); + expect(parseTransform(transform).translateY).toBeCloseTo(100, 1); + }); + + test('an end value held by the keyframes is not part of the curve, and individual transform properties are sampled', async ({ + page, + }) => { + await fixture(page); + const box = page.locator('#box'); + // 300 ms of motion on the individual `translate` and `scale` properties, + // then 5 s holding the end values: playState stays 'running' throughout. + await box.evaluate(el => { + el.animate( + [ + {translate: '0px 0px', scale: '1', offset: 0}, + {translate: '0px 120px', scale: '1.5', offset: 300 / 5300}, + {translate: '0px 120px', scale: '1.5', offset: 1}, + ], + {duration: 5300, fill: 'forwards'}, + ); + }); + const startedAt = await page.evaluate(() => performance.now()); + const [active] = await waitForAnimation(box); + expect(active.to - active.from).toBeCloseTo(300, 0); // the curve, not the hold + await waitForRest(box, {timeout: 3_000}); + const after = await box.evaluate(el => ({ + at: performance.now(), + running: el.getAnimations().some(a => a.playState === 'running'), + translate: getComputedStyle(el).translate, + })); + expect(after.running).toBe(true); // still holding its end value + expect(after.at - startedAt).toBeGreaterThanOrEqual(300); + expect(after.at - startedAt).toBeLessThan(1_500); + expect(after.translate).toBe('0px 120px'); + + const held = await holdAnimations(page, box); + expect(held[0].curveEndMs).toBeCloseTo(300, 0); + const [mid, end] = await sampleCurve(box, [150, 300]); + expect(mid.raw.translate).toBe('0px 60px'); + expect(mid.transform.translateY).toBeCloseTo(60, 1); + expect(mid.transform.scaleX).toBeCloseTo(1.25, 3); + expect(end.transform.translateY).toBeCloseTo(120, 1); + expect(end.transform.scaleY).toBeCloseTo(1.5, 3); + }); +}); + +test.describe('storyboards', () => { + test('an animation filmstrip records every frame and its trace follows the curve', async ({ + page, + }, testInfo) => { + await fixture(page); + const box = page.locator('#box'); + await box.evaluate(el => + el.animate( + [{transform: 'translateX(0)'}, {transform: 'translateX(240px)'}], + {duration: 200, easing: 'ease-out'}, + ), + ); + const {storyboard} = await animationStoryboard(page, { + name: 'harness-self-test-animation', + clip: {x: 0, y: 340, width: 402, height: 120}, + measure: async () => ({ + x: await box.evaluate(el => el.getBoundingClientRect().x), + }), + columns: 4, + }); + const files = await storyboard.write(testInfo); + const xs = storyboard.frames.map(f => f.data.x as number); + expect(xs).toHaveLength(13); // t = 0, 16.7 … 200 + expect(xs[0]).toBeCloseTo(20, 1); + expect(xs.at(-1)).toBeCloseTo(260, 1); + expect(xs.every((x, i) => i === 0 || x >= xs[i - 1])).toBe(true); + expect(files['sheet.png']).toMatch(/sheet\.png$/); + }); + + test('a touch filmstrip photographs every scripted event', async ({ + page, + }, testInfo) => { + await fixture(page); + await page.evaluate(() => { + const pad = document.getElementById('pad'); + if (pad == null) { + throw new Error('no pad'); + } + const dot = document.createElement('div'); + dot.id = 'dot'; + dot.style.cssText = + 'position:absolute;width:24px;height:24px;border-radius:12px;background:#c33;left:-30px;top:0'; + pad.append(dot); + pad.addEventListener('pointermove', e => { + dot.style.transform = `translate(${e.clientX - 12 + 30}px, ${e.clientY - 12}px)`; + }); + }); + const driver = await touchDriver(page); + const storyboard = await touchStoryboard( + page, + driver, + drag({x: 60, y: 60}, {x: 300, y: 240}, {durationMs: 100, hz: 60}), + { + name: 'harness-self-test-touch', + clip: {x: 0, y: 0, width: 402, height: 300}, + measure: async () => ({ + x: await page + .locator('#dot') + .evaluate(el => el.getBoundingClientRect().x + 12), + }), + columns: 4, + }, + ); + await storyboard.write(testInfo); + // before + touchStart + 6 moves + touchEnd + expect(storyboard.frames).toHaveLength(9); + expect(storyboard.frames.at(-2)?.data.x).toBeCloseTo(300, 0); + }); + + test('the timer gate holds a component timer until the filmstrip is done', async ({ + page, + }, testInfo) => { + await fixture(page); + await installTimerGate(page); + const box = page.locator('#box'); + // A component that ends its own animation from a timer (a transitionend + // backstop): without the gate it would finish behind the held storyboard. + await box.evaluate(el => { + const animation = el.animate( + [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}], + {duration: 300, easing: 'linear'}, + ); + setTimeout(() => { + animation.finish(); + (window as unknown as {timerRan?: boolean}).timerRan = true; + }, 40); + }); + await freezeTimers(page); + await page.waitForTimeout(120); // the timer is due now, and held + const {storyboard} = await animationStoryboard(page, { + name: 'harness-self-test-timer-gate', + clip: {x: 0, y: 340, width: 402, height: 120}, + measure: async () => ({ + x: await box.evaluate(el => el.getBoundingClientRect().x), + }), + columns: 6, + }); + const timerRan = () => + page.evaluate( + () => (window as unknown as {timerRan?: boolean}).timerRan === true, + ); + expect(await timerRan()).toBe(false); + await thawTimers(page); + expect(await timerRan()).toBe(true); + await storyboard.write(testInfo); + const xs = storyboard.frames.map(f => f.data.x as number); + expect(xs).toHaveLength(19); // t = 0 … 300 ms every 16.7 ms + expect(xs.every((x, i) => i === 0 || x > xs[i - 1])).toBe(true); + }); +}); diff --git a/internal/interaction-harness/src/index.ts b/internal/interaction-harness/src/index.ts new file mode 100644 index 0000000000000..7a45ff45dfe41 --- /dev/null +++ b/internal/interaction-harness/src/index.ts @@ -0,0 +1,24 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file index.ts + * @input Re-exports the harness modules + * @output The `@astryxdesign/interaction-harness` entry point for Playwright + * specs. Pure helpers are also reachable without Playwright through + * `@astryxdesign/interaction-harness/math` and `/scripts` (unit tests). + * @position Imported only from `*.interaction.browser.spec.ts` files (run by + * playwright.interaction.config.ts) and from this package's own tests. + * + * SYNC: When a module is added, export it here and describe it in README.md. + */ + +export * from './busy'; +export * from './curves'; +export * from './frames'; +export * from './math'; +export * from './output'; +export * from './scripts'; +export * from './story'; +export * from './storyboard'; +export * from './touch'; +export * from './trace'; diff --git a/internal/interaction-harness/src/math.test.ts b/internal/interaction-harness/src/math.test.ts new file mode 100644 index 0000000000000..645e42fa6b221 --- /dev/null +++ b/internal/interaction-harness/src/math.test.ts @@ -0,0 +1,154 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import {describe, expect, it} from 'vitest'; +import { + compareCurve, + cubicBezier, + parseTransform, + projectDecay, + rubberBand, + springStep, + timeline, + timingFunction, + velocityAt, +} from './math'; + +describe('cubicBezier', () => { + it('matches the CSS keyword curves at known points', () => { + const linear = cubicBezier(0, 0, 1, 1); + expect(linear(0.3)).toBeCloseTo(0.3, 6); + // ease (0.25, 0.1, 0.25, 1) at x = 0.5 is 0.8024 (CSS reference value). + expect(cubicBezier(0.25, 0.1, 0.25, 1)(0.5)).toBeCloseTo(0.8024, 3); + expect(cubicBezier(0.24, 1, 0.4, 1)(0)).toBe(0); + expect(cubicBezier(0.24, 1, 0.4, 1)(1)).toBe(1); + }); + it('parses timing functions', () => { + expect(timingFunction('cubic-bezier(0, 0, 1, 1)')(0.25)).toBeCloseTo( + 0.25, + 6, + ); + expect(timingFunction('ease-out')(0.5)).toBeCloseTo( + cubicBezier(0, 0, 0.58, 1)(0.5), + 9, + ); + expect(() => timingFunction('steps(4)')).toThrow(); + }); +}); + +describe('springStep (SwiftUI response / dampingFraction)', () => { + const critical = {response: 0.5, dampingFraction: 1}; + it('starts at 0 with the given velocity and ends at 1', () => { + expect(springStep(critical, 0)).toBe(0); + expect(springStep(critical, 3000)).toBeCloseTo(1, 6); + const v = 4; // progress units per second + const slope = springStep({...critical, initialVelocity: v}, 0.01) / 0.00001; + expect(slope).toBeCloseTo(v, 1); + }); + it('matches the closed forms in each regime', () => { + const w0 = (2 * Math.PI) / 0.5; + const t = 0.12; + expect(springStep(critical, t * 1000)).toBeCloseTo( + 1 - Math.exp(-w0 * t) * (1 + w0 * t), + 9, + ); + const z = 0.5; + const wd = w0 * Math.sqrt(1 - z * z); + const under = + 1 - + Math.exp(-z * w0 * t) * + (Math.cos(wd * t) + ((z * w0) / wd) * Math.sin(wd * t)); + expect( + springStep({response: 0.5, dampingFraction: z}, t * 1000), + ).toBeCloseTo(under, 9); + // Over-damped and critical agree as damping approaches 1 from above. + expect( + springStep({response: 0.5, dampingFraction: 1.000001}, 120), + ).toBeCloseTo(springStep(critical, 120), 4); + }); + it('overshoots only when under-damped', () => { + const peak = (dampingFraction: number) => + Math.max( + ...timeline(2000, 4).map(ms => + springStep({response: 0.4, dampingFraction}, ms), + ), + ); + expect(peak(0.6)).toBeGreaterThan(1.05); + expect(peak(1)).toBeLessThanOrEqual(1); + expect(peak(1.5)).toBeLessThanOrEqual(1); + }); +}); + +describe('UIKit scroll physics', () => { + it('projects a release with UIScrollView deceleration', () => { + // 1000 pt/s at 0.998/ms coasts 499 pt; at 0.99/ms (fast) 99 pt. + expect(projectDecay(1000)).toBeCloseTo(499, 6); + expect(projectDecay(1000, 0.99)).toBeCloseTo(99, 6); + expect(projectDecay(-500)).toBeCloseTo(-249.5, 6); + }); + it('rubber-bands with c = 0.55', () => { + expect(rubberBand(0, 800)).toBeCloseTo(0, 9); + expect(rubberBand(100, 800)).toBeCloseTo( + (1 - 1 / ((100 * 0.55) / 800 + 1)) * 800, + 9, + ); + expect(rubberBand(-100, 800)).toBeCloseTo(-rubberBand(100, 800), 9); + expect(rubberBand(1e9, 800)).toBeLessThan(800); + }); +}); + +describe('parseTransform', () => { + it('decomposes computed transforms', () => { + expect(parseTransform('none')).toEqual({ + translateX: 0, + translateY: 0, + scaleX: 1, + scaleY: 1, + rotate: 0, + }); + expect(parseTransform('matrix(1, 0, 0, 1, 12.5, -40)')).toMatchObject({ + translateX: 12.5, + translateY: -40, + scaleX: 1, + }); + const rotated = parseTransform( + `matrix(${Math.cos(0.5)}, ${Math.sin(0.5)}, ${-Math.sin(0.5)}, ${Math.cos(0.5)}, 0, 0)`, + ); + expect(rotated.rotate).toBeCloseTo((0.5 * 180) / Math.PI, 6); + expect( + parseTransform( + 'matrix3d(2, 0, 0, 0, 0, 2, 0, 0, 0, 0, 1, 0, 5, 6, 0, 1)', + ), + ).toMatchObject({translateX: 5, translateY: 6, scaleX: 2, scaleY: 2}); + expect(() => parseTransform('translateX(4px)')).toThrow(); + }); +}); + +describe('velocityAt', () => { + it('fits the last window and reports zero for a rest', () => { + const samples = timeline(200, 10).map(t => ({t, value: t * 0.5})); + expect(velocityAt(samples, 200)).toBeCloseTo(500, 6); + expect(velocityAt(samples, 400)).toBe(0); + }); +}); + +describe('compareCurve', () => { + it('reports the worst point', () => { + const result = compareCurve( + [ + {t: 0, value: 0}, + {t: 10, value: 1.2}, + ], + t => t / 10, + 0.1, + ); + expect(result.pass).toBe(false); + expect(result.worst.t).toBe(10); + expect(result.maxError).toBeCloseTo(0.2, 9); + }); +}); + +describe('timeline', () => { + it('includes both ends', () => { + expect(timeline(50, 1000 / 60)).toEqual([0, 16.667, 33.333, 50]); + }); +}); diff --git a/internal/interaction-harness/src/math.ts b/internal/interaction-harness/src/math.ts new file mode 100644 index 0000000000000..0006424980b3b --- /dev/null +++ b/internal/interaction-harness/src/math.ts @@ -0,0 +1,291 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file math.ts + * @input Nothing: pure functions + * @output Test oracles for motion: `cubicBezier`, `springStep` (SwiftUI + * response / dampingFraction semantics, with initial velocity), + * `projectDecay` (UIScrollView deceleration), `rubberBand` (UIKit's + * formula), `parseTransform`, `velocityAt`, and curve comparison helpers. + * @position Imported by the browser helpers and by the specs that compare a + * sampled curve against an expected one. Deliberately independent of the + * motion code under test in packages/core: an oracle that shares code with + * the implementation cannot catch the implementation's mistakes. + * + * Units: time in milliseconds, distance in CSS px (= points), velocity in + * px/s unless a name says otherwise. Spring progress is unitless (0 -> 1). + * + * SYNC: Keep ./math.test.ts covering every export. + */ + +/** A CSS `cubic-bezier(x1, y1, x2, y2)` timing function: progress in, eased progress out. */ +export function cubicBezier( + x1: number, + y1: number, + x2: number, + y2: number, +): (progress: number) => number { + const cx = 3 * x1; + const bx = 3 * (x2 - x1) - cx; + const ax = 1 - cx - bx; + const cy = 3 * y1; + const by = 3 * (y2 - y1) - cy; + const ay = 1 - cy - by; + const sampleX = (s: number) => ((ax * s + bx) * s + cx) * s; + const sampleY = (s: number) => ((ay * s + by) * s + cy) * s; + const slopeX = (s: number) => (3 * ax * s + 2 * bx) * s + cx; + return progress => { + if (progress <= 0) { + return 0; + } + if (progress >= 1) { + return 1; + } + // Newton first (fast, converges for well-behaved curves), bisection as the fallback. + let s = progress; + for (let i = 0; i < 8; i += 1) { + const error = sampleX(s) - progress; + if (Math.abs(error) < 1e-7) { + return sampleY(s); + } + const slope = slopeX(s); + if (Math.abs(slope) < 1e-6) { + break; + } + s -= error / slope; + } + let lo = 0; + let hi = 1; + s = progress; + for (let i = 0; i < 64; i += 1) { + const x = sampleX(s); + if (Math.abs(x - progress) < 1e-7) { + break; + } + if (x < progress) { + lo = s; + } else { + hi = s; + } + s = (lo + hi) / 2; + } + return sampleY(s); + }; +} + +/** Parses `cubic-bezier(a, b, c, d)` or one of the CSS keywords into a timing function. */ +export function timingFunction(css: string): (progress: number) => number { + const named: Record = { + ease: [0.25, 0.1, 0.25, 1], + 'ease-in': [0.42, 0, 1, 1], + 'ease-out': [0, 0, 0.58, 1], + 'ease-in-out': [0.42, 0, 0.58, 1], + }; + const value = css.trim(); + if (value === 'linear') { + return progress => Math.min(1, Math.max(0, progress)); + } + const match = /^cubic-bezier\(([^)]+)\)$/.exec(value); + const parts = match + ? match[1].split(',').map(Number) + : (named[value] as number[] | undefined); + if (parts == null || parts.length !== 4 || parts.some(Number.isNaN)) { + throw new Error(`unsupported timing function: ${css}`); + } + return cubicBezier(parts[0], parts[1], parts[2], parts[3]); +} + +export interface SpringParameters { + /** SwiftUI `response`: the period of the undamped spring, in seconds. */ + readonly response: number; + /** SwiftUI `dampingFraction` (UIKit `dampingRatio`): 1 = critical, < 1 bounces. */ + readonly dampingFraction: number; + /** + * Initial velocity in progress units per second, UIKit's `initialVelocity` + * convention: release velocity (px/s) divided by the distance to travel (px). + */ + readonly initialVelocity?: number; +} + +/** + * Progress (0 -> 1, may overshoot) of a spring released from rest-or-moving at + * progress 0 towards 1, after `timeMs`. Mass 1, stiffness (2π/response)², + * damping 4π·dampingFraction/response: SwiftUI's mapping. + */ +export function springStep( + {response, dampingFraction, initialVelocity = 0}: SpringParameters, + timeMs: number, +): number { + if (timeMs <= 0) { + return 0; + } + const t = timeMs / 1000; + const w0 = (2 * Math.PI) / response; + const z = dampingFraction; + // y = x - 1: displacement from the target, y(0) = -1, y'(0) = v0. + const v0 = initialVelocity; + let y: number; + if (Math.abs(z - 1) < 1e-9) { + y = Math.exp(-w0 * t) * (-1 + (v0 - w0) * t); + } else if (z < 1) { + const wd = w0 * Math.sqrt(1 - z * z); + const b = (v0 - z * w0) / wd; + y = Math.exp(-z * w0 * t) * (-Math.cos(wd * t) + b * Math.sin(wd * t)); + } else { + const root = Math.sqrt(z * z - 1); + const r1 = -w0 * (z - root); + const r2 = -w0 * (z + root); + const c1 = (v0 + r2) / (r1 - r2); + const c2 = -1 - c1; + y = c1 * Math.exp(r1 * t) + c2 * Math.exp(r2 * t); + } + return 1 + y; +} + +/** + * Where a surface released at `velocity` px/s comes to rest, relative to the + * release point, under UIScrollView deceleration (`rate` per millisecond: + * 0.998 normal, 0.99 fast). UIKit's own projection formula. + */ +export function projectDecay(velocity: number, rate = 0.998): number { + return ((velocity / 1000) * rate) / (1 - rate); +} + +/** UIKit's rubber band: `(1 - 1 / (x * c / d + 1)) * d`, c = 0.55. */ +export function rubberBand( + offset: number, + dimension: number, + constant = 0.55, +): number { + const sign = Math.sign(offset); + const x = Math.abs(offset); + return sign * (1 - 1 / ((x * constant) / dimension + 1)) * dimension; +} + +export interface Transform2D { + readonly translateX: number; + readonly translateY: number; + readonly scaleX: number; + readonly scaleY: number; + /** Rotation in degrees. */ + readonly rotate: number; +} + +const IDENTITY: Transform2D = { + translateX: 0, + translateY: 0, + scaleX: 1, + scaleY: 1, + rotate: 0, +}; + +/** Decomposes a computed `transform` (`none`, `matrix(...)`, `matrix3d(...)`) into 2D parts. */ +export function parseTransform(css: string): Transform2D { + const value = css.trim(); + if (value === '' || value === 'none') { + return IDENTITY; + } + const match = /^matrix(3d)?\(([^)]+)\)$/.exec(value); + if (match == null) { + throw new Error(`not a computed transform: ${css}`); + } + const n = match[2].split(',').map(Number); + const [a, b, c, d, e, f] = match[1] + ? [n[0], n[1], n[4], n[5], n[12], n[13]] + : n; + const scaleX = Math.hypot(a, b); + const rotate = (Math.atan2(b, a) * 180) / Math.PI; + const scaleY = (a * d - b * c) / (scaleX || 1); + return {translateX: e, translateY: f, scaleX, scaleY, rotate}; +} + +export interface TimedValue { + /** Milliseconds. */ + readonly t: number; + readonly value: number; +} + +/** + * Velocity (units/s) at the end of `samples`, by least squares over the last + * `windowMs`, the way a release velocity is judged: one slow sample before the + * lift must not cancel a throw, and a finger that rested has no velocity. + */ +export function velocityAt( + samples: readonly TimedValue[], + endMs: number, + windowMs = 100, +): number { + const recent = samples.filter(s => s.t <= endMs && s.t >= endMs - windowMs); + if (recent.length < 2) { + return 0; + } + const meanT = recent.reduce((sum, s) => sum + s.t, 0) / recent.length; + const meanV = recent.reduce((sum, s) => sum + s.value, 0) / recent.length; + let num = 0; + let den = 0; + for (const s of recent) { + num += (s.t - meanT) * (s.value - meanV); + den += (s.t - meanT) ** 2; + } + return den === 0 ? 0 : (num / den) * 1000; +} + +export interface CurveComparison { + readonly pass: boolean; + readonly tolerance: number; + readonly maxError: number; + readonly rmsError: number; + readonly worst: { + readonly t: number; + readonly actual: number; + readonly expected: number; + }; + readonly points: readonly { + readonly t: number; + readonly actual: number; + readonly expected: number; + }[]; +} + +/** Compares sampled values with an expected curve, point by point. */ +export function compareCurve( + samples: readonly TimedValue[], + expected: (timeMs: number) => number, + tolerance: number, +): CurveComparison { + if (samples.length === 0) { + throw new Error('compareCurve: no samples'); + } + const points = samples.map(s => ({ + t: s.t, + actual: s.value, + expected: expected(s.t), + })); + let worst = points[0]; + let sumSq = 0; + for (const p of points) { + const error = Math.abs(p.actual - p.expected); + sumSq += error * error; + if (error > Math.abs(worst.actual - worst.expected)) { + worst = p; + } + } + const maxError = Math.abs(worst.actual - worst.expected); + return { + pass: maxError <= tolerance, + tolerance, + maxError, + rmsError: Math.sqrt(sumSq / points.length), + worst, + points, + }; +} + +/** `from`, `from + step`, … up to and including `to` (within float error). */ +export function timeline(to: number, step = 1000 / 60, from = 0): number[] { + const out: number[] = []; + for (let i = 0; from + i * step <= to + 1e-6; i += 1) { + out.push(+(from + i * step).toFixed(3)); + } + return out; +} diff --git a/internal/interaction-harness/src/output.ts b/internal/interaction-harness/src/output.ts new file mode 100644 index 0000000000000..59926b82c9aef --- /dev/null +++ b/internal/interaction-harness/src/output.ts @@ -0,0 +1,66 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file output.ts + * @input Uses node:fs, node:path and the Playwright `TestInfo` + * @output `evidenceDir` / `writeEvidence` — where storyboards and curve traces + * go, and the one way they are written and attached to the test report. + * @position Every artifact the harness produces lands under one gitignored + * directory, `test-results/interaction-evidence/` by default + * (`ASTRYX_INTERACTION_EVIDENCE` overrides it). Playwright wipes its own + * output directory per run; this one is separate, so a storyboard run never + * deletes another's evidence. Generated evidence is never committed: a pull + * request links or attaches it. + */ + +import fs from 'node:fs'; +import path from 'node:path'; +import type {TestInfo} from '@playwright/test'; + +export function evidenceDir(): string { + return path.resolve( + process.env.ASTRYX_INTERACTION_EVIDENCE ?? + 'test-results/interaction-evidence', + ); +} + +/** A filesystem-safe name: letters, digits, dot, dash and underscore. */ +export function safeName(name: string): string { + return ( + name + .replace(/[^A-Za-z0-9._-]+/g, '-') + .replace(/^-+|-+$/g, '') + .slice(0, 120) || 'evidence' + ); +} + +/** + * Writes `files` (name → contents) under `//` and attaches + * them to the test's report when a `TestInfo` is given. Returns the paths. + */ +export async function writeEvidence( + group: string, + files: Readonly>, + testInfo?: TestInfo, +): Promise> { + // `group` may nest ("storyboards/"): each segment is made safe on its own. + const dir = path.join(evidenceDir(), ...group.split('/').map(safeName)); + fs.mkdirSync(dir, {recursive: true}); + const written: Record = {}; + for (const [name, contents] of Object.entries(files)) { + const file = path.join(dir, safeName(name)); + fs.writeFileSync(file, contents); + written[name] = file; + if (testInfo != null) { + await testInfo.attach(`${group}/${safeName(name)}`, { + path: file, + contentType: name.endsWith('.png') + ? 'image/png' + : name.endsWith('.json') + ? 'application/json' + : 'text/plain', + }); + } + } + return written; +} diff --git a/internal/interaction-harness/src/scripts.test.ts b/internal/interaction-harness/src/scripts.test.ts new file mode 100644 index 0000000000000..b85d4b526de55 --- /dev/null +++ b/internal/interaction-harness/src/scripts.test.ts @@ -0,0 +1,142 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import {describe, expect, it} from 'vitest'; +import { + drag, + flick, + holdThenDrag, + merge, + path, + pinch, + releaseVelocity, + rotate, + tap, + toTouchEvents, + withFinger, +} from './scripts'; + +describe('touch scripts', () => { + it('tap presses and lifts in place', () => { + expect( + toTouchEvents(tap({x: 5, y: 6}, {holdMs: 80})).map(e => [e.t, e.type]), + ).toEqual([ + [0, 'touchStart'], + [80, 'touchEnd'], + ]); + }); + + it('drag moves at 60 Hz, holds, rests and lifts', () => { + const script = drag( + {x: 0, y: 0}, + {x: 0, y: 100}, + {durationMs: 100, hz: 60, holdMs: 500, restMs: 150}, + ); + const events = toTouchEvents(script); + expect(events[0]).toMatchObject({t: 0, type: 'touchStart'}); + const moves = events.filter(e => e.type === 'touchMove'); + expect(moves).toHaveLength(6); + expect(moves[0].t).toBeCloseTo(516.667, 3); + expect(moves.at(-1)).toMatchObject({ + t: 600, + fingers: [{id: 0, x: 0, y: 100}], + }); + expect(events.at(-1)).toMatchObject({ + t: 750, + type: 'touchEnd', + fingers: [], + }); + // A rest of 100 ms or more releases nothing in motion. + expect(releaseVelocity(script)).toEqual({x: 0, y: 0}); + }); + + it('flick releases at exactly the scripted velocity', () => { + const script = flick( + {x: 10, y: 10}, + {velocity: {x: 0, y: 1500}, durationMs: 80, hz: 120}, + ); + expect(releaseVelocity(script).y).toBeCloseTo(1500, 6); + const last = script.at(-2)!; + expect(last.fingers[0].y).toBeCloseTo(10 + 1500 * 0.08, 6); + // The lift lands a quarter frame after the last move, still moving. + expect(script.at(-1)!.t - last.t).toBeCloseTo(1000 / 120 / 4, 6); + }); + + it('flick follows a velocity profile', () => { + const script = flick( + {x: 0, y: 0}, + {velocity: {x: 1000, y: 0}, durationMs: 100, hz: 100, profile: p => p}, + ); + // Linear ramp: distance = v * T / 2. + expect(script.at(-2)!.fingers[0].x).toBeCloseTo(50, 6); + }); + + it('pinch and rotate keep two fingers about the centre', () => { + const spread = pinch({ + center: {x: 100, y: 100}, + fromDistance: 40, + toDistance: 200, + durationMs: 100, + }); + const end = spread.at(-2)!.fingers; + expect(Math.hypot(end[1].x - end[0].x, end[1].y - end[0].y)).toBeCloseTo( + 200, + 6, + ); + const turn = rotate({ + center: {x: 0, y: 0}, + distance: 100, + fromAngle: 0, + toAngle: 90, + durationMs: 100, + }); + expect(turn.at(-2)!.fingers[1].x).toBeCloseTo(0, 6); + expect(turn.at(-2)!.fingers[1].y).toBeCloseTo(50, 6); + expect(toTouchEvents(spread)[0]).toMatchObject({ + type: 'touchStart', + changed: [{id: 0}, {id: 1}], + }); + }); + + it('merge lands a second finger mid-drag and lifts it first', () => { + const first = drag({x: 0, y: 0}, {x: 0, y: 90}, {durationMs: 300, hz: 10}); + const second = withFinger(tap({x: 50, y: 50}, {holdMs: 100}), 1, 120); + const events = toTouchEvents(merge(first, second)); + const starts = events.filter(e => e.type === 'touchStart'); + expect( + starts.map(e => [e.t, e.changed.map(f => f.id), e.fingers.length]), + ).toEqual([ + [0, [0], 1], + [120, [1], 2], + ]); + const ends = events.filter(e => e.type === 'touchEnd'); + expect(ends[0]).toMatchObject({t: 220, changed: [{id: 1}]}); + expect(ends[0].fingers.map(f => f.id)).toEqual([0]); + expect(ends.at(-1)!.fingers).toEqual([]); + expect(() => merge(first, first)).toThrow(/two scripts/); + }); + + it('path follows any curve; holdThenDrag holds before the first move', () => { + const arc = path( + p => ({x: 100 * Math.cos(p * Math.PI), y: 100 * Math.sin(p * Math.PI)}), + {durationMs: 100, hz: 40}, + ); + const last = arc.at(-2)!.fingers[0]; + expect(last.x).toBeCloseTo(-100, 6); + expect(last.y).toBeCloseTo(0, 6); + expect(arc.at(-1)!.fingers).toEqual([]); + const held = holdThenDrag( + {x: 0, y: 0}, + {x: 30, y: 0}, + {durationMs: 50, hz: 20, holdMs: 500}, + ); + const moves = toTouchEvents(held).filter(e => e.type === 'touchMove'); + expect(moves[0].t).toBeCloseTo(550, 6); + expect(moves.at(-1)!.fingers[0].x).toBeCloseTo(30, 6); + }); + + it('refuses a script that leaves a finger down', () => { + expect(() => + toTouchEvents([{t: 0, fingers: [{id: 0, x: 0, y: 0}]}]), + ).toThrow(/lifted/); + }); +}); diff --git a/internal/interaction-harness/src/scripts.ts b/internal/interaction-harness/src/scripts.ts new file mode 100644 index 0000000000000..111d9e01234aa --- /dev/null +++ b/internal/interaction-harness/src/scripts.ts @@ -0,0 +1,395 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file scripts.ts + * @input Nothing: pure functions + * @output Touch scripts — timed lists of the fingers that are down — and the + * builders that make them: `tap`, `drag`, `flick`, `holdThenDrag`, `pinch`, + * `rotate`, `path`, `withFinger`, `merge`, plus `toTouchEvents`, which turns + * a script into the per-event sequence a touch driver dispatches. + * @position The input half of every interaction proof. A script is data, so a + * test states exactly what the finger does — positions and timestamps — and + * a unit test can check a builder without a browser. + * + * A script is a list of frames ordered by time. Each frame lists every finger + * that is down at that moment; a finger missing from a frame has lifted. The + * last frame is always empty (every finger up). Times are milliseconds from + * the gesture's start; coordinates are CSS px in the viewport. + * + * SYNC: Keep ./scripts.test.ts covering every builder. + */ + +export interface Point { + readonly x: number; + readonly y: number; +} + +export interface Finger extends Point { + /** Stable per finger for the whole gesture. */ + readonly id: number; +} + +export interface TouchFrame { + /** Milliseconds from the start of the gesture. */ + readonly t: number; + /** Every finger that is down at `t`. Empty = all lifted. */ + readonly fingers: readonly Finger[]; +} + +export type TouchScript = readonly TouchFrame[]; + +/** One dispatchable event: what changed at `t`, and the fingers down after it. */ +export interface TouchEventStep { + readonly t: number; + readonly type: 'touchStart' | 'touchMove' | 'touchEnd' | 'touchCancel'; + /** Fingers down after this event (CDP semantics: the full active set). */ + readonly fingers: readonly Finger[]; + /** Fingers this event pressed, moved or released. */ + readonly changed: readonly Finger[]; +} + +const DEFAULT_HZ = 60; + +function frameTimes(durationMs: number, hz: number): number[] { + const step = 1000 / hz; + const count = Math.max(1, Math.round(durationMs / step)); + return Array.from( + {length: count}, + (_, i) => +(((i + 1) * durationMs) / count).toFixed(3), + ); +} + +function lerp(a: Point, b: Point, p: number): Point { + return {x: a.x + (b.x - a.x) * p, y: a.y + (b.y - a.y) * p}; +} + +/** A press and lift in place. */ +export function tap( + at: Point, + {holdMs = 60, id = 0}: {holdMs?: number; id?: number} = {}, +): TouchScript { + return [ + {t: 0, fingers: [{id, ...at}]}, + {t: holdMs, fingers: []}, + ]; +} + +export interface DragOptions { + /** Time from the first move to the last, ms. */ + readonly durationMs: number; + /** Move samples per second (iOS reports touches at 60 Hz, 120 Hz on ProMotion). */ + readonly hz?: number; + /** Rest before the first move (press-and-hold), ms. */ + readonly holdMs?: number; + /** Rest after the last move before the lift, ms: >= 100 makes the release velocity 0. */ + readonly restMs?: number; + /** Progress easing over the move, default linear (constant velocity). */ + readonly easing?: (progress: number) => number; + readonly id?: number; +} + +/** Press at `from`, move to `to`, lift. Constant velocity unless `easing` says otherwise. */ +export function drag( + from: Point, + to: Point, + options: DragOptions, +): TouchScript { + const { + durationMs, + hz = DEFAULT_HZ, + holdMs = 0, + restMs = 0, + easing = (p: number) => p, + id = 0, + } = options; + const frames: TouchFrame[] = [{t: 0, fingers: [{id, ...from}]}]; + for (const t of frameTimes(durationMs, hz)) { + frames.push({ + t: holdMs + t, + fingers: [{id, ...lerp(from, to, easing(t / durationMs))}], + }); + } + // A real finger reports no move while it rests: the lift simply comes later. + const last = frames[frames.length - 1].t; + frames.push({t: last + Math.max(restMs, 1000 / hz / 4), fingers: []}); + return frames; +} + +/** `drag` after a press-and-hold (a long press that turns into a drag). */ +export function holdThenDrag( + from: Point, + to: Point, + options: DragOptions & {readonly holdMs: number}, +): TouchScript { + return drag(from, to, options); +} + +export interface FlickOptions { + /** Velocity at the lift, px/s. */ + readonly velocity: Point; + /** How long the finger moves before the lift, ms. */ + readonly durationMs?: number; + readonly hz?: number; + /** + * Velocity over the move as a fraction of the final velocity (0..1 → 0..∞), + * default constant. `p => p` is a finger accelerating linearly into the lift. + */ + readonly profile?: (progress: number) => number; + readonly id?: number; +} + +/** + * A throw: the finger moves with the given velocity profile and lifts while + * still moving (the lift lands a quarter frame after the last move, as on a + * device), so the release velocity is exactly `velocity`. + */ +export function flick(from: Point, options: FlickOptions): TouchScript { + const { + velocity, + durationMs = 80, + hz = 120, + profile = () => 1, + id = 0, + } = options; + const step = 1000 / hz; + const frames: TouchFrame[] = [{t: 0, fingers: [{id, ...from}]}]; + let x = from.x; + let y = from.y; + let previous = 0; + for (const t of frameTimes(durationMs, hz)) { + // Integrate the profile over (previous, t] with the midpoint rule. + const scale = profile((previous + t) / 2 / durationMs); + x += (velocity.x * scale * (t - previous)) / 1000; + y += (velocity.y * scale * (t - previous)) / 1000; + previous = t; + frames.push({t, fingers: [{id, x, y}]}); + } + frames.push({t: previous + step / 4, fingers: []}); + return frames; +} + +/** Any single-finger path: `at(progress)` for progress 0..1 over `durationMs`. */ +export function path( + at: (progress: number) => Point, + { + durationMs, + hz = DEFAULT_HZ, + holdMs = 0, + restMs = 0, + id = 0, + }: Omit, +): TouchScript { + const frames: TouchFrame[] = [{t: 0, fingers: [{id, ...at(0)}]}]; + for (const t of frameTimes(durationMs, hz)) { + frames.push({t: holdMs + t, fingers: [{id, ...at(t / durationMs)}]}); + } + frames.push({ + t: frames[frames.length - 1].t + Math.max(restMs, 1000 / hz / 4), + fingers: [], + }); + return frames; +} + +export interface PinchOptions { + readonly center: Point; + /** Finger separation at the start and at the end, px. */ + readonly fromDistance: number; + readonly toDistance: number; + /** Angle of the line between the fingers, degrees (0 = horizontal). */ + readonly fromAngle?: number; + readonly toAngle?: number; + readonly durationMs: number; + readonly hz?: number; + readonly restMs?: number; +} + +/** Two fingers (ids 0 and 1) pressed together, spread or pinched and turned, lifted together. */ +export function pinch(options: PinchOptions): TouchScript { + const { + center, + fromDistance, + toDistance, + fromAngle = 0, + toAngle = fromAngle, + durationMs, + hz = DEFAULT_HZ, + restMs = 0, + } = options; + const at = (p: number): Finger[] => { + const distance = fromDistance + (toDistance - fromDistance) * p; + const angle = ((fromAngle + (toAngle - fromAngle) * p) * Math.PI) / 180; + const dx = (Math.cos(angle) * distance) / 2; + const dy = (Math.sin(angle) * distance) / 2; + return [ + {id: 0, x: center.x - dx, y: center.y - dy}, + {id: 1, x: center.x + dx, y: center.y + dy}, + ]; + }; + const frames: TouchFrame[] = [{t: 0, fingers: at(0)}]; + for (const t of frameTimes(durationMs, hz)) { + frames.push({t, fingers: at(t / durationMs)}); + } + frames.push({t: durationMs + Math.max(restMs, 1000 / hz / 4), fingers: []}); + return frames; +} + +/** Two fingers turning around `center` at a fixed separation. */ +export function rotate( + options: Omit & { + readonly distance: number; + }, +): TouchScript { + return pinch({ + ...options, + fromDistance: options.distance, + toDistance: options.distance, + }); +} + +/** Re-labels a single-finger script's finger, and shifts it in time. */ +export function withFinger( + script: TouchScript, + id: number, + delayMs = 0, +): TouchScript { + return script.map(frame => ({ + t: frame.t + delayMs, + fingers: frame.fingers.map(f => ({...f, id})), + })); +} + +/** + * Plays several scripts at once (each with its own finger ids): at every time + * any of them changes, each finger is where its own script last put it. + * Used for a second finger that lands mid-gesture. + */ +export function merge(...scripts: TouchScript[]): TouchScript { + const times = [...new Set(scripts.flatMap(s => s.map(f => f.t)))].sort( + (a, b) => a - b, + ); + const stateAt = (script: TouchScript, t: number): readonly Finger[] => { + let current: readonly Finger[] = []; + for (const frame of script) { + if (frame.t > t) { + break; + } + current = frame.fingers; + } + return current; + }; + const owner = new Map(); + scripts.forEach((script, index) => { + for (const frame of script) { + for (const f of frame.fingers) { + if ((owner.get(f.id) ?? index) !== index) { + throw new Error(`merge: finger ${f.id} is used by two scripts`); + } + owner.set(f.id, index); + } + } + }); + return times.map(t => ({t, fingers: scripts.flatMap(s => stateAt(s, t))})); +} + +/** + * Turns frames into events: a finger that appears is a `touchStart`, one that + * disappears a `touchEnd`, one that moved a `touchMove`. A frame that both + * moves some fingers and lifts others yields the move first, then the lift. + * Unchanged frames yield nothing (a resting finger reports nothing). + */ +export function toTouchEvents(script: TouchScript): TouchEventStep[] { + const events: TouchEventStep[] = []; + let down = new Map(); + for (const frame of script) { + const next = new Map(frame.fingers.map(f => [f.id, f])); + const pressed = frame.fingers.filter(f => !down.has(f.id)); + const released = [...down.values()].filter(f => !next.has(f.id)); + const moved = frame.fingers.filter(f => { + const before = down.get(f.id); + return before != null && (before.x !== f.x || before.y !== f.y); + }); + if (moved.length > 0) { + const after = [...down.values()].map(f => next.get(f.id) ?? f); + events.push({ + t: frame.t, + type: 'touchMove', + fingers: after, + changed: moved, + }); + down = new Map(after.map(f => [f.id, f])); + } + if (released.length > 0) { + const remaining = [...down.values()].filter(f => next.has(f.id)); + events.push({ + t: frame.t, + type: 'touchEnd', + fingers: remaining, + changed: released.map(f => down.get(f.id) ?? f), + }); + down = new Map(remaining.map(f => [f.id, f])); + } + if (pressed.length > 0) { + const all = [...down.values(), ...pressed]; + events.push({ + t: frame.t, + type: 'touchStart', + fingers: all, + changed: pressed, + }); + down = new Map(all.map(f => [f.id, f])); + } + } + if (down.size > 0) { + throw new Error( + 'toTouchEvents: a script must end with every finger lifted', + ); + } + return events; +} + +/** + * The velocity (px/s) of finger `id` at its lift: the slope over the last + * `windowMs` of moves, or zero when the finger rested `windowMs` or longer + * before lifting (a finger that rested released nothing in motion). + */ +export function releaseVelocity( + script: TouchScript, + id = 0, + windowMs = 100, +): Point { + const samples = script.flatMap(frame => + frame.fingers + .filter(f => f.id === id) + .map(f => ({t: frame.t, x: f.x, y: f.y})), + ); + if (samples.length < 2) { + return {x: 0, y: 0}; + } + const lift = script.find( + frame => frame.t > samples[0].t && !frame.fingers.some(f => f.id === id), + ); + // The last sample that actually moved: a resting finger reports nothing. + let lastMove = samples.length - 1; + while ( + lastMove > 0 && + samples[lastMove].x === samples[lastMove - 1].x && + samples[lastMove].y === samples[lastMove - 1].y + ) { + lastMove -= 1; + } + const last = samples[lastMove]; + if (lift != null && lift.t - last.t >= windowMs) { + return {x: 0, y: 0}; + } + const recent = samples + .slice(0, lastMove + 1) + .filter(s => s.t >= last.t - windowMs); + const first = recent[0]; + const dt = last.t - first.t; + return dt <= 0 + ? {x: 0, y: 0} + : { + x: ((last.x - first.x) / dt) * 1000, + y: ((last.y - first.y) / dt) * 1000, + }; +} diff --git a/internal/interaction-harness/src/story.ts b/internal/interaction-harness/src/story.ts new file mode 100644 index 0000000000000..0058fe5b579e2 --- /dev/null +++ b/internal/interaction-harness/src/story.ts @@ -0,0 +1,163 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file story.ts + * @input Uses a Playwright `Page` and test object, and the a11y-spec package's + * static server over a built Storybook + * @output `phone` (the touch-phone browser context every interaction proof + * runs in), `withStorybook` (serve the built Storybook for a spec file), and + * `openStory` (load one story and wait until it has rendered). + * @position Stories are the reproduction path, as for every other browser + * check in this repository: a scenario used for verification is checked in, + * not built in a one-off page. + * + * The phone is an iPhone 17 Pro in Safari: 402 × 874 pt screen, 402 × 714 pt + * visible page with the toolbars shown, 3 device pixels per point, touch, + * mobile viewport rules. Measured on the iOS 26.2 simulator + * (`innerWidth`/`innerHeight`, `screen`, `devicePixelRatio`). + * + * SYNC: Keep `phone` equal to the projects in playwright.interaction.config.ts. + */ + +import type {Page, TestType} from '@playwright/test'; +import { + DEFAULT_STORYBOOK_DIR, + serveStorybook, + type StaticServer, +} from '@astryxdesign/a11y-spec/storybook'; + +export const phone = { + viewport: {width: 402, height: 714}, + screen: {width: 402, height: 874}, + deviceScaleFactor: 3, + isMobile: true, + hasTouch: true, +} as const; + +/** + * Serves the built Storybook for the spec file's tests (`ASTRYX_STORYBOOK_DIR` + * overrides the default `apps/storybook/dist`). Returns its origin getter. + */ +// eslint-disable-next-line @typescript-eslint/no-explicit-any -- any fixture set +export function withStorybook(test: TestType): () => string { + let server: StaticServer | undefined; + test.beforeAll(async () => { + server = await serveStorybook( + process.env.ASTRYX_STORYBOOK_DIR ?? DEFAULT_STORYBOOK_DIR, + ); + }); + test.afterAll(async () => { + await server?.close(); + }); + return () => { + if (server == null) { + throw new Error( + 'withStorybook: the Storybook server is not running (call inside a test)', + ); + } + return server.origin; + }; +} + +/** Loads a story in the page and waits for its first render, its fonts and two frames. */ +export async function openStory( + page: Page, + origin: string, + storyId: string, + args?: Readonly>, +): Promise { + const params = new URLSearchParams({id: storyId, viewMode: 'story'}); + if (args != null) { + params.set( + 'args', + Object.entries(args) + .map(([k, v]) => `${k}:${v}`) + .join(';'), + ); + } + await page.goto(`${origin}/iframe.html?${params.toString()}`); + await page + .locator('#storybook-root > *') + .first() + .waitFor({state: 'attached'}); + await page.evaluate(async () => { + await document.fonts.ready; + await new Promise(resolve => + requestAnimationFrame(() => requestAnimationFrame(() => resolve())), + ); + }); +} + +/** Resolves after `count` animation frames have been presented in the page. */ +export async function frames(page: Page, count = 2): Promise { + await page.evaluate( + n => + new Promise(resolve => { + let left = n; + const tick = () => + --left <= 0 ? resolve() : requestAnimationFrame(tick); + requestAnimationFrame(tick); + }), + count, + ); +} + +const TIMER_GATE = `(() => { + if (window.__harnessTimerGate) return; + const gate = {frozen: false, queue: []}; + window.__harnessTimerGate = gate; + const realTimeout = window.setTimeout.bind(window); + const realInterval = window.setInterval.bind(window); + const realFrame = window.requestAnimationFrame.bind(window); + const hold = run => (gate.frozen ? gate.queue.push(run) : run()); + window.setTimeout = (fn, ms, ...args) => realTimeout(() => hold(() => (typeof fn === 'function' ? fn(...args) : 0)), ms); + window.setInterval = (fn, ms, ...args) => realInterval(() => { if (!gate.frozen && typeof fn === 'function') fn(...args); }, ms); + window.requestAnimationFrame = fn => realFrame(time => hold(() => fn(gate.frozen ? performance.now() : time))); +})()`; + +/** + * Installs a gate on the page's setTimeout / setInterval / + * requestAnimationFrame. Install it BEFORE the animation under test starts: + * only callbacks scheduled through the gate can be held. Event timestamps, + * Date, performance.now and the Web Animations timeline are untouched. + */ +export async function installTimerGate(page: Page): Promise { + await page.evaluate(TIMER_GATE); +} + +/** + * Holds every gated timer and frame callback until `thawTimers`, so a + * component cannot finish an animation behind a held storyboard's back (a + * transitionend backstop timer, a cleanup render). Callbacks that come due + * meanwhile run, in order, on thaw; intervals skip their frozen ticks. + */ +export async function freezeTimers(page: Page): Promise { + await page.evaluate(() => { + const gate = (window as unknown as {__harnessTimerGate?: {frozen: boolean}}) + .__harnessTimerGate; + if (gate == null) { + throw new Error( + 'freezeTimers: call installTimerGate before the animation starts', + ); + } + gate.frozen = true; + }); +} + +/** Releases the gate: held callbacks run now, in the order they came due. */ +export async function thawTimers(page: Page): Promise { + await page.evaluate(() => { + const gate = ( + window as unknown as { + __harnessTimerGate?: {frozen: boolean; queue: (() => void)[]}; + } + ).__harnessTimerGate; + if (gate == null) { + return; + } + gate.frozen = false; + for (const run of gate.queue.splice(0)) { + run(); + } + }); +} diff --git a/internal/interaction-harness/src/storyboard.ts b/internal/interaction-harness/src/storyboard.ts new file mode 100644 index 0000000000000..898994d6ba3b3 --- /dev/null +++ b/internal/interaction-harness/src/storyboard.ts @@ -0,0 +1,366 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file storyboard.ts + * @input Uses a Playwright `Page`, a clip (`Locator` or box), ./curves to seek + * animations, ./touch to play scripted touches, and ./output to write + * @output `Storyboard` (collect frames, write a PNG contact sheet + JSON trace), + * `animationStoryboard` (seek every held animation to t = 0, 16.7, 33.3 … ms + * and photograph each), `touchStoryboard` (photograph the page after every + * scripted touch event). + * @position Deterministic filmstrips: frames come from seeking paused + * animations or from stepping scripted input, never from wall-clock capture, + * so the same code produces the same sheet. The JSON trace carries the + * measured value of every frame and is what the spec asserts; the sheet is + * the human evidence attached to the report (and linked from a pull + * request, never committed). + * + * The contact sheet is composed in a scratch page of the same browser (a + * canvas grid with a label per cell), so the harness needs no image library. + * + * SYNC: Keep the README's storyboard section and `pnpm storyboards` in step. + */ + +import type {Locator, Page, TestInfo} from '@playwright/test'; +import {holdAnimations, seekAnimations, type HeldAnimation} from './curves'; +import {timeline} from './math'; +import {writeEvidence} from './output'; +import type {TouchScript} from './scripts'; +import type {TouchDriver} from './touch'; + +export interface ClipBox { + readonly x: number; + readonly y: number; + readonly width: number; + readonly height: number; +} + +export interface StoryboardFrame { + readonly label: string; + /** Milliseconds on the storyboard's own timeline. */ + readonly t: number; + /** Measurements taken at this frame; the spec asserts these. */ + readonly data: Readonly>; + readonly png: Buffer; +} + +export interface StoryboardOptions { + readonly name: string; + /** What to photograph: an element, a box, or the whole viewport (default). */ + readonly clip?: Locator | ClipBox; + /** Grid columns in the sheet (default: up to 8). */ + readonly columns?: number; + /** Cell scale relative to CSS px (default 0.5). */ + readonly cellScale?: number; +} + +export class Storyboard { + readonly frames: StoryboardFrame[] = []; + private clipBox: ClipBox | undefined; + + constructor( + private readonly page: Page, + readonly options: StoryboardOptions, + ) {} + + /** Photographs the clip now and records `data` with it. */ + async capture( + label: string, + t: number, + data: Readonly> = {}, + ): Promise { + const clip = await this.resolveClip(); + const png = await this.page.screenshot({ + clip, + scale: 'css', + animations: 'allow', + caret: 'hide', + }); + this.frames.push({label, t, data, png}); + } + + private async resolveClip(): Promise { + if (this.clipBox != null) { + return this.clipBox; + } + const {clip} = this.options; + if (clip == null) { + return undefined; + } + if ('boundingBox' in clip) { + // A locator is measured once, at the first frame, so every cell shows the same region. + const box = await clip.boundingBox(); + if (box == null) { + throw new Error( + `storyboard ${this.options.name}: clip element is not visible`, + ); + } + const viewport = this.page.viewportSize(); + const x = Math.max(0, Math.floor(box.x)); + const y = Math.max(0, Math.floor(box.y)); + const right = Math.min( + viewport?.width ?? Infinity, + Math.ceil(box.x + box.width), + ); + const bottom = Math.min( + viewport?.height ?? Infinity, + Math.ceil(box.y + box.height), + ); + this.clipBox = {x, y, width: right - x, height: bottom - y}; + } else { + this.clipBox = clip; + } + return this.clipBox; + } + + /** The JSON trace: every frame's label, time and data. */ + trace( + extra: Readonly> = {}, + ): Record { + return { + name: this.options.name, + browser: this.page.context().browser()?.browserType().name() ?? null, + viewport: this.page.viewportSize(), + clip: this.clipBox ?? null, + ...extra, + frames: this.frames.map(({label, t, data}) => ({label, t, data})), + }; + } + + /** Composes the sheet, writes `sheet.png` + `trace.json`, attaches both. */ + async write( + testInfo?: TestInfo, + extra: Readonly> = {}, + ): Promise> { + if (this.frames.length === 0) { + throw new Error(`storyboard ${this.options.name}: no frames`); + } + const sheet = await composeSheet(this.page, this.frames, this.options); + const browser = + this.page.context().browser()?.browserType().name() ?? 'browser'; + return writeEvidence( + `storyboards/${this.options.name}.${browser}`, + { + 'sheet.png': sheet, + 'trace.json': JSON.stringify(this.trace(extra), null, 2), + }, + testInfo, + ); + } +} + +async function composeSheet( + page: Page, + frames: readonly StoryboardFrame[], + options: StoryboardOptions, +): Promise { + const scratch = await page.context().newPage(); + try { + const dataUrl = await scratch.evaluate( + async ({cells, columns, scale, title}) => { + const images = await Promise.all( + cells.map(async cell => + createImageBitmap( + await (await fetch(`data:image/png;base64,${cell.png}`)).blob(), + ), + ), + ); + const cellW = Math.ceil(images[0].width * scale); + const cellH = Math.ceil(images[0].height * scale); + const labelH = 30; + const pad = 8; + const cols = Math.min(columns, cells.length); + const rows = Math.ceil(cells.length / cols); + const canvas = document.createElement('canvas'); + canvas.width = pad + cols * (cellW + pad); + canvas.height = 28 + pad + rows * (cellH + labelH + pad); + const context = canvas.getContext('2d'); + if (context == null) { + throw new Error('no 2d context'); + } + context.fillStyle = '#ffffff'; + context.fillRect(0, 0, canvas.width, canvas.height); + context.fillStyle = '#111111'; + context.font = '600 14px sans-serif'; + context.fillText(title, pad, 19); + cells.forEach((cell, i) => { + const x = pad + (i % cols) * (cellW + pad); + const y = 28 + pad + Math.floor(i / cols) * (cellH + labelH + pad); + context.drawImage(images[i], x, y, cellW, cellH); + context.strokeStyle = '#999999'; + context.lineWidth = 1; + context.strokeRect(x - 0.5, y - 0.5, cellW + 1, cellH + 1); + context.fillStyle = '#111111'; + context.font = '600 11px sans-serif'; + context.fillText(cell.label, x, y + cellH + 13, cellW); + context.fillStyle = '#555555'; + context.font = '10px monospace'; + context.fillText(cell.caption, x, y + cellH + 26, cellW); + }); + return canvas.toDataURL('image/png'); + }, + { + cells: frames.map(f => ({ + png: f.png.toString('base64'), + label: f.label, + caption: Object.entries(f.data) + .filter( + ([, v]) => + typeof v === 'number' || + typeof v === 'string' || + typeof v === 'boolean', + ) + .map( + ([k, v]) => + `${k}=${typeof v === 'number' ? +v.toFixed(1) : String(v)}`, + ) + .join(' '), + })), + columns: options.columns ?? 8, + scale: options.cellScale ?? 0.5, + title: options.name, + }, + ); + return Buffer.from(dataUrl.slice(dataUrl.indexOf(',') + 1), 'base64'); + } finally { + await scratch.close(); + } +} + +export interface AnimationStoryboardOptions extends StoryboardOptions { + /** + * Frame times, ms (default every 16.7 ms to the end of the longest held + * animation's curve; a held end value is not filmed). + */ + readonly times?: readonly number[]; + /** Restrict to animations inside this element. */ + readonly scope?: Locator; + /** Measurements per frame (runs after the seek). */ + readonly measure?: () => Promise>>; +} + +/** + * Holds every running animation, then seeks them together to each time and + * photographs the clip. Call it right after starting the animation under test. + */ +export async function animationStoryboard( + page: Page, + options: AnimationStoryboardOptions, +): Promise<{storyboard: Storyboard; held: HeldAnimation[]}> { + const held = await holdAnimations(page, options.scope); + if (held.length === 0) { + throw new Error(`storyboard ${options.name}: no running animation to hold`); + } + const longest = Math.max(...held.map(a => a.curveEndMs)); + const times = options.times ?? timeline(longest); + const storyboard = new Storyboard(page, options); + for (const t of times) { + await seekAnimations(page, t); + const data = (await options.measure?.()) ?? {}; + await storyboard.capture(`t=${+t.toFixed(1)} ms`, t, data); + } + return {storyboard, held}; +} + +export interface TouchStoryboardOptions extends StoryboardOptions { + readonly measure?: () => Promise>>; +} + +/** + * Plays a touch script without wall-clock pacing and photographs the page + * after every event. In Chromium the page still sees the scripted timestamps, + * so velocity-dependent behavior is unchanged; in WebKit the page sees the + * real (slower) dispatch times. + */ +export async function touchStoryboard( + page: Page, + driver: TouchDriver, + script: TouchScript, + options: TouchStoryboardOptions, +): Promise { + const storyboard = new Storyboard(page, options); + await storyboard.capture('before', 0, (await options.measure?.()) ?? {}); + await driver.play(script, { + realtime: false, + onEvent: async step => { + // Let the page run its handlers and present the frame before the photo. + await page.evaluate( + () => + new Promise(resolve => + requestAnimationFrame(() => requestAnimationFrame(() => resolve())), + ), + ); + const data = (await options.measure?.()) ?? {}; + const where = step.changed + .map(f => `${Math.round(f.x)},${Math.round(f.y)}`) + .join(' '); + await storyboard.capture( + `${step.type} t=${+step.t.toFixed(1)} @${where}`, + step.t, + data, + ); + }, + }); + return storyboard; +} + +/** + * A filmstrip of screencast frames (real presented frames, e.g. the frames of + * a busy-main-thread window), labelled with their time relative to `zero`. + * Evidence for what the compositor presented, not a deterministic storyboard. + */ +export async function castStoryboard( + page: Page, + frames: readonly { + readonly t: number; + readonly data: string; + readonly format: 'png' | 'jpeg'; + }[], + options: StoryboardOptions & { + readonly zero: number; + readonly from?: number; + readonly to?: number; + readonly data?: (t: number) => Readonly>; + }, +): Promise { + const storyboard = new Storyboard(page, options); + for (const frame of frames) { + if (options.from != null && frame.t < options.from) { + continue; + } + if (options.to != null && frame.t > options.to) { + continue; + } + const png = + frame.format === 'png' + ? Buffer.from(frame.data, 'base64') + : await toPng(page, frame.data); + const t = frame.t - options.zero; + storyboard.frames.push({ + label: `t=${t.toFixed(1)} ms`, + t, + data: options.data?.(frame.t) ?? {}, + png, + }); + } + return storyboard; +} + +async function toPng(page: Page, jpegBase64: string): Promise { + const scratch = await page.context().newPage(); + try { + const url = await scratch.evaluate(async data => { + const bitmap = await createImageBitmap( + await (await fetch(`data:image/jpeg;base64,${data}`)).blob(), + ); + const canvas = document.createElement('canvas'); + canvas.width = bitmap.width; + canvas.height = bitmap.height; + canvas.getContext('2d')?.drawImage(bitmap, 0, 0); + return canvas.toDataURL('image/png'); + }, jpegBase64); + return Buffer.from(url.slice(url.indexOf(',') + 1), 'base64'); + } finally { + await scratch.close(); + } +} diff --git a/internal/interaction-harness/src/touch.ts b/internal/interaction-harness/src/touch.ts new file mode 100644 index 0000000000000..1c16c4b1f82e0 --- /dev/null +++ b/internal/interaction-harness/src/touch.ts @@ -0,0 +1,312 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file touch.ts + * @input Uses a Playwright `Page` (Chromium or WebKit) and a touch script from + * ./scripts + * @output `touchDriver(page)` — plays touch scripts as REAL browser touch input + * — and `installInputLog` / `readInputLog`, the page-side record of what the + * page actually received. + * @position The input half of every interaction proof. Untrusted DOM events + * are not input: they never start native scrolling, never claim a gesture, + * and a component can tell them apart (`isTrusted`). Both drivers here go + * through the engine's own input pipeline instead. + * + * Chromium: CDP `Input.dispatchTouchEvent` with an explicit `timestamp` per + * event. The page's `event.timeStamp` IS that timestamp, whatever the + * dispatch latency, so a velocity profile reaches the page exactly. The + * events go through the compositor's gesture detection: native scrolling, + * flings (velocity from those timestamps), `touch-action`, pointer events + * and `pointercancel` all behave as for a finger. + * + * WebKit (Playwright's Linux build): the engine's own inspector command + * `Input.dispatchTouchEvent` on the page proxy. Touches and the pointer + * events derived from them are trusted and cancelable, multi-touch works. + * Limits, measured with WebKit 26.5: no timestamp parameter (the page sees + * the real dispatch time, so velocity comes from real-time pacing, ±1 ms); + * integer coordinates only; no native scrolling, fling or pinch-zoom from + * these touches; a `touchstart` lists only the fingers that landed in it + * (pointer events are exact); no click synthesised from a raw touch sequence (use + * `tap`, which dispatches WebKit's tap gesture and does produce the + * compatibility mouse events and click). The command is reached through + * Playwright's in-process server object, so it needs a locally launched + * browser (the default), not a remote `connect()`. + * + * SYNC: Keep the engine limits above equal to the README's table and to the + * harness self-test (../src/harness.interaction.browser.spec.ts). + */ + +import type {CDPSession, Page} from '@playwright/test'; +import { + toTouchEvents, + type Finger, + type Point, + type TouchEventStep, + type TouchScript, +} from './scripts'; + +export type Engine = 'chromium' | 'webkit'; + +export interface PlayOptions { + /** + * Pace dispatch on the wall clock so each event leaves at its scripted time + * (default true). Required in WebKit for a velocity to mean anything; in + * Chromium the page sees the scripted timestamps either way, but native + * behaviors that run on the compositor clock (a fling) want real pacing. + */ + readonly realtime?: boolean; + /** Called after each event is dispatched, before the next one (storyboards). */ + readonly onEvent?: ( + step: TouchEventStep, + index: number, + ) => Promise | void; +} + +export interface PlayedEvent extends TouchEventStep { + /** Wall-clock milliseconds (epoch) at which the event was dispatched. */ + readonly dispatchedAt: number; + /** The timestamp handed to the engine, epoch ms (Chromium only). */ + readonly timestamp: number | null; +} + +export interface TouchDriver { + readonly engine: Engine; + /** True when the page's `event.timeStamp` follows the script exactly. */ + readonly scriptedTimestamps: boolean; + /** True when these touches can scroll, fling and pinch-zoom natively. */ + readonly drivesNativeGestures: boolean; + /** + * True when lifting one finger while others stay down reaches the page as + * iOS sends it: a `touchend` whose `touches` still lists the fingers down. + * Skip such proofs with `PARTIAL_LIFT_SKIP` where it is false. + */ + readonly partialLifts: boolean; + /** Plays a script; resolves when its last event has been dispatched. */ + play(script: TouchScript, options?: PlayOptions): Promise; + /** A tap through the engine's tap path (WebKit synthesises click only from this). */ + tap(at: Point): Promise; +} + +/** + * The skip reason for a proof that lifts one finger while another stays down, + * on a driver whose `partialLifts` is false: Playwright's WebKit (Linux build) + * sends that lift's `touchend` with no remaining `touches`, an emulation quirk + * iOS does not have. Those cases need a simulator capture. + */ +export const PARTIAL_LIFT_SKIP = + 'injected WebKit touches report no remaining touches when one finger of several lifts (README: engine limits); capture it on the iOS simulator'; + +const sleep = (ms: number) => new Promise(resolve => setTimeout(resolve, ms)); + +/** Waits until `performance.now()` reaches `deadline` with ~0.1 ms precision. */ +async function until(deadline: number): Promise { + const remaining = deadline - performance.now(); + if (remaining > 3) { + await sleep(remaining - 2); + } + while (performance.now() < deadline) { + // spin: setTimeout alone is ±1-4 ms late + } +} + +async function playSteps( + script: TouchScript, + options: PlayOptions, + dispatch: (step: TouchEventStep, timestamp: number) => Promise, + stampFor: (t: number) => number, +): Promise { + const {realtime = true, onEvent} = options; + const steps = toTouchEvents(script); + const played: PlayedEvent[] = []; + const origin = performance.now(); + for (const [index, step] of steps.entries()) { + if (realtime) { + await until(origin + step.t); + } + const timestamp = stampFor(step.t); + const dispatchedAt = Date.now(); + await dispatch(step, timestamp); + played.push({...step, dispatchedAt, timestamp}); + await onEvent?.(step, index); + } + return played; +} + +async function chromiumDriver(page: Page): Promise { + const cdp: CDPSession = await page.context().newCDPSession(page); + const touchPoints = (fingers: readonly Finger[]) => + fingers.map(f => ({ + x: f.x, + y: f.y, + id: f.id, + radiusX: 1, + radiusY: 1, + force: 1, + })); + return { + engine: 'chromium', + scriptedTimestamps: true, + drivesNativeGestures: true, + partialLifts: true, + async play(script, options = {}) { + // Stamps follow the wall clock the page's own clock is on: with real-time + // pacing an event's stamp and its dispatch agree to ~0.1 ms. + const base = Date.now(); + return playSteps( + script, + options, + async (step, timestamp) => { + // A start or move lists every finger down (CDP presses the new ones). + // An end lists the fingers that lift: that releases exactly those + // while the rest stay down and keep moving; the last lift lists none. + // (A move that omits a finger does not release it.) + await cdp.send('Input.dispatchTouchEvent', { + type: step.type, + touchPoints: + step.type !== 'touchEnd' + ? touchPoints(step.fingers) + : step.fingers.length > 0 + ? touchPoints(step.changed) + : [], + timestamp: timestamp / 1000, + }); + }, + t => base + t, + ); + }, + async tap(at) { + await page.touchscreen.tap(at.x, at.y); + }, + }; +} + +interface WebKitPageProxySession { + send(method: string, params?: Record): Promise; + on(event: string, listener: (payload: unknown) => void): unknown; + off?(event: string, listener: (payload: unknown) => void): unknown; +} + +/** + * Playwright's server-side WebKit page session, reached through the in-process + * connection. Exported for helpers that use WebKit's own inspector protocol. + */ +export function webkitPageSession(page: Page): WebKitPageProxySession { + const connection = ( + page as unknown as {_connection?: {toImpl?: (o: unknown) => unknown}} + )._connection; + const impl = connection?.toImpl?.(page) as + {delegate?: {_pageProxySession?: WebKitPageProxySession}} | undefined; + const session = impl?.delegate?._pageProxySession; + if (session == null) { + throw new Error( + "WebKit touch input needs Playwright's in-process server (a locally launched browser); this page has none.", + ); + } + return session; +} + +async function webkitDriver(page: Page): Promise { + const session = webkitPageSession(page); + // WebKit's command takes integer CSS px and applies the event's phase to + // every point it lists: a start lists only the fingers that land (listing a + // finger already down presses it again), an end only the fingers that lift + // (an empty list releases nothing), a move every finger down. + const points = (fingers: readonly Finger[]) => + fingers.map(f => ({x: Math.round(f.x), y: Math.round(f.y), id: f.id})); + return { + engine: 'webkit', + scriptedTimestamps: false, + drivesNativeGestures: false, + partialLifts: false, + async play(script, options = {}) { + return playSteps( + script, + options, + async step => { + await session.send('Input.dispatchTouchEvent', { + type: step.type, + touchPoints: points( + step.type === 'touchMove' ? step.fingers : step.changed, + ), + }); + }, + () => Date.now(), + ).then(played => played.map(p => ({...p, timestamp: null}))); + }, + async tap(at) { + await page.touchscreen.tap(Math.round(at.x), Math.round(at.y)); + }, + }; +} + +/** A touch driver for the page's engine. Firefox is not supported (no touch input path). */ +export async function touchDriver(page: Page): Promise { + const name = page.context().browser()?.browserType().name(); + if (name === 'chromium') { + return chromiumDriver(page); + } + if (name === 'webkit') { + return webkitDriver(page); + } + throw new Error( + `touchDriver: no real touch input path for ${name ?? 'this browser'}`, + ); +} + +export interface InputRecord { + readonly type: string; + /** The event's own `timeStamp`, ms since the page's time origin. */ + readonly timeStamp: number; + readonly isTrusted: boolean; + readonly cancelable: boolean; + readonly defaultPrevented: boolean; + readonly pointerId?: number; + readonly pointerType?: string; + readonly x?: number; + readonly y?: number; + readonly touches?: number; + readonly target: string; +} + +const INPUT_LOG_SCRIPT = `(() => { + if (window.__harnessInput) return; + const log = []; + window.__harnessInput = log; + const describe = node => node instanceof Element + ? node.tagName.toLowerCase() + (node.id ? '#' + node.id : '') + (node.getAttribute('aria-label') ? '[' + node.getAttribute('aria-label') + ']' : '') + : String(node && node.nodeName); + const types = ['pointerdown', 'pointermove', 'pointerup', 'pointercancel', 'touchstart', 'touchmove', + 'touchend', 'touchcancel', 'click', 'contextmenu', 'gotpointercapture', 'lostpointercapture']; + for (const type of types) { + // Bubble phase on window, so defaultPrevented reflects the page's own handlers. + window.addEventListener(type, event => { + const record = {type, timeStamp: event.timeStamp, isTrusted: event.isTrusted, cancelable: event.cancelable, + defaultPrevented: event.defaultPrevented, target: describe(event.target)}; + if ('pointerId' in event) Object.assign(record, {pointerId: event.pointerId, pointerType: event.pointerType, x: event.clientX, y: event.clientY}); + if (event.touches) Object.assign(record, {touches: event.touches.length, x: event.changedTouches[0] && event.changedTouches[0].clientX, y: event.changedTouches[0] && event.changedTouches[0].clientY}); + log.push(record); + }, {passive: true}); + } +})()`; + +/** Records every pointer/touch/click the page receives (install before navigation). */ +export async function installInputLog(page: Page): Promise { + await page.addInitScript(INPUT_LOG_SCRIPT); +} + +/** What the page received since navigation (or since the last `clear`). */ +export async function readInputLog( + page: Page, + {clear = false}: {clear?: boolean} = {}, +): Promise { + return page.evaluate(shouldClear => { + const log = + (window as unknown as {__harnessInput?: InputRecord[]}).__harnessInput ?? + []; + const copy = log.slice(); + if (shouldClear) { + log.length = 0; + } + return copy; + }, clear); +} diff --git a/internal/interaction-harness/src/trace.ts b/internal/interaction-harness/src/trace.ts new file mode 100644 index 0000000000000..773445067dd2f --- /dev/null +++ b/internal/interaction-harness/src/trace.ts @@ -0,0 +1,203 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * @file trace.ts + * @input Uses a Playwright `Page` (Chromium: CDP `Tracing`) and the page's + * PerformanceObserver for long tasks + * @output `traceRendering` — counts of the main thread's rendering work + * (Layout, Paint, UpdateLayoutTree, PrePaint, …) between two marks — and + * `installLongTaskLog` / `readLongTasks`. + * @position The "no Layout/Paint during a transform/opacity animation" proof. + * A compositor animation costs the main thread nothing per frame; one that + * animates a layout property, or writes styles from JavaScript each frame, + * shows a Layout (and usually a Paint) per frame. Chromium only: WebKit + * exposes no comparable trace to Playwright. + * + * Counting is scoped to the renderer main thread that ran the page's own + * `performance.mark` calls, and to the time between the start and end marks, + * so other pages, the browser process and setup work are excluded. + * + * SYNC: Keep RENDERING_EVENTS equal to the README's list. + */ + +import type {CDPSession, Page} from '@playwright/test'; + +/** Main-thread trace events that are rendering work, by their trace names. */ +export const RENDERING_EVENTS = [ + 'UpdateLayoutTree', // style recalculation + 'Layout', + 'PrePaint', + 'Paint', + 'Layerize', + 'UpdateLayer', + 'ScrollLayer', + 'HitTest', +] as const; + +export interface RenderingTrace { + /** Count per trace event name inside the window, on the page's main thread. */ + readonly counts: Readonly>; + /** Shorthand for the four that matter most. */ + readonly layout: number; + readonly paint: number; + readonly styleRecalc: number; + readonly prePaint: number; + /** Window length, ms. */ + readonly durationMs: number; + /** Main-thread `RunTask`s longer than 50 ms inside the window (long tasks), epoch ms. */ + readonly longTasks: readonly { + readonly start: number; + readonly durationMs: number; + }[]; + /** Every counted rendering event with its epoch-ms time, for sub-window counts. */ + readonly events: readonly {readonly name: string; readonly t: number}[]; + /** The window, epoch ms on the page's clock. */ + readonly window: {readonly from: number; readonly to: number}; +} + +/** Counts rendering events by name inside `[from, to]` (epoch ms). */ +export function countBetween( + trace: RenderingTrace, + from: number, + to: number, +): Record { + const counts: Record = {}; + for (const e of trace.events) { + if (e.t >= from && e.t <= to) { + counts[e.name] = (counts[e.name] ?? 0) + 1; + } + } + return counts; +} + +interface TraceEvent { + name: string; + cat: string; + ph: string; + ts: number; + dur?: number; + pid: number; + tid: number; + args?: Record; +} + +/** + * Traces the page while `body` runs and counts the rendering work its main + * thread did between a start mark (placed just before `body`) and an end mark + * (just after). + */ +export async function traceRendering( + page: Page, + body: () => Promise, +): Promise { + const cdp: CDPSession = await page.context().newCDPSession(page); + const events: TraceEvent[] = []; + cdp.on('Tracing.dataCollected', ({value}) => { + events.push(...(value as unknown as TraceEvent[])); + }); + const complete = new Promise(resolve => + cdp.once('Tracing.tracingComplete', () => resolve()), + ); + await cdp.send('Tracing.start', { + transferMode: 'ReportEvents', + traceConfig: { + includedCategories: [ + 'devtools.timeline', + 'disabled-by-default-devtools.timeline', + 'blink.user_timing', + 'toplevel', + ], + excludedCategories: ['*'], + }, + }); + const tag = `harness-${Math.random().toString(36).slice(2, 8)}`; + const mark = (name: string) => + page.evaluate( + n => performance.timeOrigin + performance.mark(n).startTime, + name, + ); + const fromEpoch = await mark(`${tag}:start`); + await body(); + const toEpoch = await mark(`${tag}:end`); + await cdp.send('Tracing.end'); + await complete; + await cdp.detach().catch(() => {}); + + const start = events.find(e => e.name === `${tag}:start`); + const end = events.find(e => e.name === `${tag}:end`); + if (start == null || end == null) { + throw new Error('traceRendering: the window marks were not traced'); + } + const onMain = (e: TraceEvent) => + e.pid === start.pid && + e.tid === start.tid && + e.ts >= start.ts && + e.ts <= end.ts; + // Trace time (µs, monotonic) -> epoch ms, anchored on the start mark. + const toEpochMs = (ts: number) => fromEpoch + (ts - start.ts) / 1000; + const counts: Record = {}; + const counted: {name: string; t: number}[] = []; + for (const e of events) { + if (!onMain(e) || e.ph === 'E') { + continue; + } + if ((RENDERING_EVENTS as readonly string[]).includes(e.name)) { + counts[e.name] = (counts[e.name] ?? 0) + 1; + counted.push({name: e.name, t: toEpochMs(e.ts)}); + } + } + const longTasks = events + .filter(e => onMain(e) && e.name === 'RunTask' && (e.dur ?? 0) > 50_000) + .map(e => ({start: toEpochMs(e.ts), durationMs: (e.dur ?? 0) / 1000})); + return { + counts, + layout: counts.Layout ?? 0, + paint: counts.Paint ?? 0, + styleRecalc: counts.UpdateLayoutTree ?? 0, + prePaint: counts.PrePaint ?? 0, + durationMs: (end.ts - start.ts) / 1000, + longTasks, + events: counted, + window: {from: fromEpoch, to: toEpoch}, + }; +} + +const LONG_TASK_SCRIPT = `(() => { + if (window.__harnessLongTasks || typeof PerformanceObserver === 'undefined') return; + const tasks = []; + window.__harnessLongTasks = tasks; + try { + new PerformanceObserver(list => { + for (const entry of list.getEntries()) tasks.push({start: performance.timeOrigin + entry.startTime, durationMs: entry.duration, name: entry.name}); + }).observe({type: 'longtask', buffered: true}); + } catch {} +})()`; + +/** Records every long task (> 50 ms) the page runs (install before navigation; Chromium). */ +export async function installLongTaskLog(page: Page): Promise { + await page.addInitScript(LONG_TASK_SCRIPT); +} + +export interface LongTask { + /** Epoch ms. */ + readonly start: number; + readonly durationMs: number; + readonly name: string; +} + +/** Long tasks recorded since navigation, optionally only those overlapping a window. */ +export async function readLongTasks( + page: Page, + range?: {from: number; to: number}, +): Promise { + const tasks = await page.evaluate( + () => + (window as unknown as {__harnessLongTasks?: LongTask[]}) + .__harnessLongTasks ?? [], + ); + return range == null + ? tasks + : tasks.filter( + t => t.start < range.to && t.start + t.durationMs > range.from, + ); +} diff --git a/internal/interaction-harness/tsconfig.json b/internal/interaction-harness/tsconfig.json new file mode 100644 index 0000000000000..596e2cf729a9f --- /dev/null +++ b/internal/interaction-harness/tsconfig.json @@ -0,0 +1,4 @@ +{ + "extends": "../../tsconfig.json", + "include": ["src"] +} diff --git a/package.json b/package.json index 9052c34ad02d0..99888d872a0e1 100644 --- a/package.json +++ b/package.json @@ -25,8 +25,10 @@ "test:coverage": "vitest --coverage", "test:a11y-contract": "playwright test --config playwright.config.ts --project chromium", "test:docsite-browser": "playwright test --config apps/docsite/playwright.config.ts", + "test:interaction": "playwright test --config playwright.interaction.config.ts", + "storyboards": "playwright test --config playwright.interaction.config.ts --grep @storyboard", "check:package-boundaries": "node scripts/check-package-boundaries.js", - "check:repo": "pnpm check:visual-owner && pnpm check:sync && pnpm check:neutral-palette-refs && pnpm check:stone-palette-refs && pnpm check:gothic-palette-refs && pnpm check:matcha-palette-refs && pnpm check:butter-palette-refs && pnpm check:y2k-palette-refs && pnpm check:data-tokens && pnpm check:token-docs && pnpm check:knowledge && pnpm check:package-boundaries && pnpm check:changesets && pnpm check:demo-media && pnpm check:executable-bits && pnpm check:cli-structure && pnpm check:use-client && pnpm check:portable-scripts && pnpm check:i18n-runtime && pnpm check:i18n-catalog && pnpm check:cldr-weekdays && pnpm check:fixtures && pnpm check:setup-contract", + "check:repo": "pnpm check:visual-owner && pnpm check:sync && pnpm check:neutral-palette-refs && pnpm check:stone-palette-refs && pnpm check:gothic-palette-refs && pnpm check:matcha-palette-refs && pnpm check:butter-palette-refs && pnpm check:y2k-palette-refs && pnpm check:data-tokens && pnpm check:token-docs && pnpm check:motion-roles && pnpm check:knowledge && pnpm check:package-boundaries && pnpm check:changesets && pnpm check:demo-media && pnpm check:executable-bits && pnpm check:cli-structure && pnpm check:use-client && pnpm check:portable-scripts && pnpm check:i18n-runtime && pnpm check:i18n-catalog && pnpm check:cldr-weekdays && pnpm check:fixtures && pnpm check:setup-contract", "check:visual-owner": "node scripts/check-visual-workflow-owner.mjs", "check:fixtures": "node internal/vibe-tests/scripts/fixture-suite.mjs verify", "generate:cldr-weekdays": "node scripts/generate-cldr-weekdays.mjs", @@ -46,6 +48,8 @@ "check:data-tokens": "node scripts/generate-data-tokens.mjs --check", "generate:token-docs": "node scripts/generate-token-docs.mjs", "check:token-docs": "node scripts/generate-token-docs.mjs --check", + "generate:motion-roles": "node scripts/generate-motion-roles.mjs", + "check:motion-roles": "node scripts/generate-motion-roles.mjs --check", "check:use-client": "node scripts/check-use-client.mjs", "lint": "pnpm check:repo && eslint . --cache", "lint:strict": "pnpm check:repo && ASTRYX_STRICT_LINT=1 eslint . --cache", diff --git a/packages/build/README.md b/packages/build/README.md index 0bc69e79c696d..033e85be2e50d 100644 --- a/packages/build/README.md +++ b/packages/build/README.md @@ -239,6 +239,8 @@ Wraps `@stylexjs/unplugin` and intercepts the dev CSS endpoint (`/virtual:stylex Unsupported declarations fail the build by default with StyleX's replacement guidance. Set `propertyValidationMode` explicitly in `stylexOverrides` (modern API) or `stylexOptions` (legacy API) only when a migration needs the upstream `warn` or `silent` behavior. +A production build also checks the StyleX stylesheet for generated custom properties (`--`) that it reads but nothing declares, which every browser would drop: a `.stylex.ts` export read in a style that is not a `defineVars`/`defineConsts` value, or one whose module never reached the build. By default it warns, naming the variable and the rule that reads it; `undefinedVariables: 'error'` (either API) fails the build instead. + --- ## Advanced Options diff --git a/packages/build/__fixtures__/stylex-consts/App.tsx b/packages/build/__fixtures__/stylex-consts/App.tsx new file mode 100644 index 0000000000000..c42a0171ddcef --- /dev/null +++ b/packages/build/__fixtures__/stylex-consts/App.tsx @@ -0,0 +1,31 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * Reads `defineConsts` values from other files, as a media-query key and as a + * plain value, from both sides of the layer split. Only the constants are + * imported: once StyleX compiles `stylex.create`, nothing in this file uses + * the imports at runtime any more. + */ + +import * as stylex from '@stylexjs/stylex'; +import {appConsts} from './appConsts.stylex'; +import {libraryConsts} from './library/libraryConsts.stylex'; + +const styles = stylex.create({ + row: { + minHeight: appConsts.rowHeight, + transitionProperty: {default: 'color', [appConsts.narrow]: 'none'}, + }, + panel: { + paddingTop: libraryConsts.inset, + maxWidth: {default: null, [libraryConsts.wide]: '640px'}, + }, +}); + +export function App() { + return ( +
+
row
+
+ ); +} diff --git a/packages/build/__fixtures__/stylex-consts/appConsts.stylex.ts b/packages/build/__fixtures__/stylex-consts/appConsts.stylex.ts new file mode 100644 index 0000000000000..5e41328b69880 --- /dev/null +++ b/packages/build/__fixtures__/stylex-consts/appConsts.stylex.ts @@ -0,0 +1,9 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import * as stylex from '@stylexjs/stylex'; + +/** Constants the app defines itself, beside the file that reads them. */ +export const appConsts = stylex.defineConsts({ + narrow: '@media (max-width: 599px)', + rowHeight: '52px', +}); diff --git a/packages/build/__fixtures__/stylex-consts/index.html b/packages/build/__fixtures__/stylex-consts/index.html new file mode 100644 index 0000000000000..ba86fc975ff35 --- /dev/null +++ b/packages/build/__fixtures__/stylex-consts/index.html @@ -0,0 +1,12 @@ + + + + + + stylex-consts fixture — defineConsts imported across files + + +
+ + + diff --git a/packages/build/__fixtures__/stylex-consts/library/libraryConsts.stylex.ts b/packages/build/__fixtures__/stylex-consts/library/libraryConsts.stylex.ts new file mode 100644 index 0000000000000..92dda6955e61f --- /dev/null +++ b/packages/build/__fixtures__/stylex-consts/library/libraryConsts.stylex.ts @@ -0,0 +1,12 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import * as stylex from '@stylexjs/stylex'; + +/** + * Constants on the library side of the layer split (the build test points + * `libraryPattern` at this folder), read by an app file on the product side. + */ +export const libraryConsts = stylex.defineConsts({ + wide: '@media (min-width: 900px)', + inset: '13px', +}); diff --git a/packages/build/__fixtures__/stylex-consts/main.tsx b/packages/build/__fixtures__/stylex-consts/main.tsx new file mode 100644 index 0000000000000..5811fe017bb3e --- /dev/null +++ b/packages/build/__fixtures__/stylex-consts/main.tsx @@ -0,0 +1,6 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import {createRoot} from 'react-dom/client'; +import {App} from './App'; + +createRoot(document.getElementById('root')!).render(); diff --git a/packages/build/__fixtures__/stylex-undefined-var/App.tsx b/packages/build/__fixtures__/stylex-undefined-var/App.tsx new file mode 100644 index 0000000000000..fbd157f1439ed --- /dev/null +++ b/packages/build/__fixtures__/stylex-undefined-var/App.tsx @@ -0,0 +1,17 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * Reads a plain `.stylex.ts` export as if it were a token. The build must fail + * and name the variable rather than ship a declaration every browser drops. + */ + +import * as stylex from '@stylexjs/stylex'; +import {plainTokens} from './plainTokens.stylex'; + +const styles = stylex.create({ + root: {rowGap: plainTokens.gap}, +}); + +export function App() { + return
gap
; +} diff --git a/packages/build/__fixtures__/stylex-undefined-var/index.html b/packages/build/__fixtures__/stylex-undefined-var/index.html new file mode 100644 index 0000000000000..a7732d79694cf --- /dev/null +++ b/packages/build/__fixtures__/stylex-undefined-var/index.html @@ -0,0 +1,12 @@ + + + + + + stylex-undefined-var fixture — a plain export read as a token + + +
+ + + diff --git a/packages/build/__fixtures__/stylex-undefined-var/main.tsx b/packages/build/__fixtures__/stylex-undefined-var/main.tsx new file mode 100644 index 0000000000000..5811fe017bb3e --- /dev/null +++ b/packages/build/__fixtures__/stylex-undefined-var/main.tsx @@ -0,0 +1,6 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +import {createRoot} from 'react-dom/client'; +import {App} from './App'; + +createRoot(document.getElementById('root')!).render(); diff --git a/packages/build/__fixtures__/stylex-undefined-var/plainTokens.stylex.ts b/packages/build/__fixtures__/stylex-undefined-var/plainTokens.stylex.ts new file mode 100644 index 0000000000000..fdb800e145dbd --- /dev/null +++ b/packages/build/__fixtures__/stylex-undefined-var/plainTokens.stylex.ts @@ -0,0 +1,7 @@ +// Copyright (c) Meta Platforms, Inc. and affiliates. + +/** + * A plain export, not `defineVars`/`defineConsts`: StyleX compiles a read of + * it in another file to a custom property that nothing declares. + */ +export const plainTokens = {gap: '12px'}; diff --git a/packages/build/src/vite.build.test.ts b/packages/build/src/vite.build.test.ts index ee860275a2752..2d2f4bae230ed 100644 --- a/packages/build/src/vite.build.test.ts +++ b/packages/build/src/vite.build.test.ts @@ -31,7 +31,7 @@ */ import {describe, it, expect, beforeAll, afterAll} from 'vitest'; -import {build} from 'vite'; +import {build, createLogger} from 'vite'; import react from '@vitejs/plugin-react'; import { mkdirSync, @@ -70,6 +70,20 @@ type Built = { async function buildFixture( name: string, htmlNames: string[] = ['index.html'], + { + libraryPattern = 'packages/core/', + zeroConfig = false, + undefinedVariables, + warnings = [], + }: { + /** Which source paths land in the library layer. */ + libraryPattern?: string; + /** Use `astryxStylex({...})` instead of the legacy `{stylexOptions}` form. */ + zeroConfig?: boolean; + undefinedVariables?: 'warn' | 'error'; + /** Receives every warning the build logs. */ + warnings?: string[]; + } = {}, ): Promise { const root = path.join(FIXTURES, name); const outDir = mkdtempSync(path.join(tmpdir(), `astryx-${name}-`)); @@ -77,6 +91,11 @@ async function buildFixture( await build({ root, logLevel: 'error', + customLogger: { + ...createLogger('error'), + warn: (message: string) => void warnings.push(message), + warnOnce: (message: string) => void warnings.push(message), + }, build: { outDir, emptyOutDir: true, @@ -94,17 +113,23 @@ async function buildFixture( resolve: {alias: {'@astryxdesign/core': CORE_SRC}}, plugins: [ react(), - ...astryxStylex({ - stylexOptions: { - dev: false, - unstable_moduleResolution: {type: 'commonJS', rootDir: REPO_ROOT}, - aliases: { - '@astryxdesign/core/*': [path.join(CORE_SRC, '*')], - '@astryxdesign/core': [CORE_SRC], - }, - }, - libraryPattern: 'packages/core/', - }), + ...(zeroConfig + ? astryxStylex({dev: false, rootDir: REPO_ROOT, libraryPattern}) + : astryxStylex({ + stylexOptions: { + dev: false, + unstable_moduleResolution: { + type: 'commonJS', + rootDir: REPO_ROOT, + }, + aliases: { + '@astryxdesign/core/*': [path.join(CORE_SRC, '*')], + '@astryxdesign/core': [CORE_SRC], + }, + }, + libraryPattern, + undefinedVariables, + })), ], }); } catch (error) { @@ -338,6 +363,83 @@ describe('a production build separates Astryx and product styles by layer', () = }); }); +// `defineConsts` values are compiled into the reading file as placeholders, +// `var(--)`, and the defining file contributes the values; the +// final CSS pass substitutes one for the other. Both halves must reach that +// pass. Storybook builds through the legacy `{stylexOptions}` form, whose +// StyleX instance ran without `treeshakeCompensation`: once `stylex.create` +// was compiled, nothing used the import at runtime, the TypeScript transform +// dropped it, and the defining module was never loaded. A media-query key was +// left as `var(--x…){…}` and failed the build ("Invalid empty selector"); a +// value shipped as a custom property nothing defines. And a constant defined +// on one side of the layer split and read on the other was substituted only +// within its own side. +describe('a production build resolves defineConsts imported across files', () => { + /** A constant reference StyleX left unsubstituted. */ + const PLACEHOLDER = /var\(--(?:astryx|x)[a-z0-9]{4,}\)/; + + for (const zeroConfig of [false, true]) { + describe( + zeroConfig ? 'zero-config form' : 'legacy stylexOptions form (Storybook)', + () => { + let built: Built; + beforeAll(async () => { + built = await buildFixture('stylex-consts', ['index.html'], { + libraryPattern: '__fixtures__/stylex-consts/library/', + zeroConfig, + }); + }, 180_000); + afterAll( + () => built && rmSync(built.outDir, {recursive: true, force: true}), + ); + + it('substitutes a constant read as a value', () => { + expect(built.css).toMatch(/min-height:\s*52px/); + }); + + it('substitutes a constant read as a media-query key', () => { + expect(built.css).toMatch( + /@media \((?:max-width: ?599px|width ?<= ?599px)\)\s*\{[^}]*transition-property:\s*none/, + ); + }); + + it('substitutes a library-side constant read from the product side', () => { + expect(built.css).toMatch(/padding-top:\s*13px/); + expect(built.css).toMatch( + /@media \((?:min-width: ?900px|width ?>= ?900px)\)\s*\{[^}]*max-width:\s*640px/, + ); + }); + + it('leaves no constant placeholder behind', () => { + expect(built.css).not.toMatch(PLACEHOLDER); + }); + }, + ); + } +}); + +describe('a production build checks the custom properties its stylesheet reads', () => { + const MESSAGE = + /reads 1 generated custom property that nothing declares[\s\S]*--x[a-z0-9]+, read by: [^\n]*row-gap/; + + it('warns by default, naming the variable and the rule that reads it', async () => { + const warnings: string[] = []; + const built = await buildFixture('stylex-undefined-var', ['index.html'], { + warnings, + }); + rmSync(built.outDir, {recursive: true, force: true}); + expect(warnings.join('\n')).toMatch(MESSAGE); + }, 180_000); + + it("fails the build with undefinedVariables: 'error'", async () => { + await expect( + buildFixture('stylex-undefined-var', ['index.html'], { + undefinedVariables: 'error', + }), + ).rejects.toThrow(MESSAGE); + }, 180_000); +}); + describe('a production build validates StyleX declarations', () => { it('fails with longhand guidance instead of dropping a border shorthand', async () => { await expect(buildFixture('invalid-border')).rejects.toThrow( diff --git a/packages/build/src/vite.test.ts b/packages/build/src/vite.test.ts index 8f618832c8a34..ebbeae7d3db17 100644 --- a/packages/build/src/vite.test.ts +++ b/packages/build/src/vite.test.ts @@ -13,7 +13,11 @@ import {tmpdir} from 'node:os'; import path from 'node:path'; import {fileURLToPath, pathToFileURL} from 'node:url'; import stylex from '@stylexjs/unplugin'; -import {astryxStylex, LIGHTNINGCSS_TARGETS} from './vite'; +import { + astryxStylex, + LIGHTNINGCSS_TARGETS, + undefinedGeneratedVariables, +} from './vite'; const __dirname = path.dirname(fileURLToPath(import.meta.url)); @@ -272,3 +276,38 @@ describe('astryxStylex optimizeDeps package discovery', () => { } }); }); + +describe('undefinedGeneratedVariables', () => { + it('reports a generated variable that is read but never declared', () => { + expect( + undefinedGeneratedVariables( + '.a{min-height:var(--astryx1abc2)}.b{color:var(--x9zz9z, red)}', + ['astryx', 'x'], + ), + ).toEqual(['--astryx1abc2', '--x9zz9z']); + }); + + it('accepts a variable declared anywhere in the stylesheet', () => { + expect( + undefinedGeneratedVariables( + ':root{--astryx1abc2:4px}.a{padding:var(--astryx1abc2)}', + ['astryx', 'x'], + ), + ).toEqual([]); + }); + + it('ignores dynamic-style and component variables, which carry a dash', () => { + expect( + undefinedGeneratedVariables( + '.a{width:var(--x-width)}.b{padding:var(--astryx-card-padding,8px)}', + ['astryx', 'x'], + ), + ).toEqual([]); + }); + + it('checks only the prefixes it is given', () => { + expect( + undefinedGeneratedVariables('.a{color:var(--x1abc2)}', ['astryx']), + ).toEqual([]); + }); +}); diff --git a/packages/build/src/vite.ts b/packages/build/src/vite.ts index 0581b58675edb..56242e066030b 100644 --- a/packages/build/src/vite.ts +++ b/packages/build/src/vite.ts @@ -40,6 +40,16 @@ export interface AstryxVitePluginLegacyOptions { library?: string; product?: string; }; + + /** + * What the build does when its StyleX stylesheet reads a generated custom + * property (`--`) that nothing declares, a declaration every + * browser drops: `'warn'` reports the variable and the rule that reads it; + * `'error'` fails the build. + * + * @default 'warn' + */ + undefinedVariables?: 'warn' | 'error'; } export interface AstryxVitePluginOptions { @@ -97,6 +107,16 @@ export interface AstryxVitePluginOptions { * Extra StyleX options to merge. */ stylexOverrides?: Record; + + /** + * What the build does when its StyleX stylesheet reads a generated custom + * property (`--`) that nothing declares, a declaration every + * browser drops: `'warn'` reports the variable and the rule that reads it; + * `'error'` fails the build. + * + * @default 'warn' + */ + undefinedVariables?: 'warn' | 'error'; } /** @@ -132,6 +152,7 @@ export function astryxStylex( lightningcssTargets = LIGHTNINGCSS_TARGETS, stylexPrefix = 'astryx', stylexOverrides = {}, + undefinedVariables = 'warn', } = opts; const libraryLayer = layers.library ?? 'astryx-base'; @@ -302,6 +323,8 @@ export function astryxStylex( libraryLayer, productLayer, lightningcssOptions: stylexOptions.lightningcssOptions, + classNamePrefixes: [stylexPrefix, 'x'], + undefinedVariables, }), ]; } @@ -336,18 +359,30 @@ function renderSplitLayers( const libraryRules: unknown[] = []; const productRules: unknown[] = []; + // `defineConsts` values. A file that reads a constant compiles it to a + // placeholder, `var(--)`, and `processStylexRules` substitutes + // the value from the defining file's rules, but only among the rules it is + // given. The defining file and the reading one can sit on opposite sides of + // the split, so every layer gets every constant. Constant rules emit no CSS + // of their own, so repeating them adds nothing to the output. + const constantRules: unknown[] = []; for (const [filePath, rules] of rulesById.entries()) { - if (filePath.includes(options.libraryPattern)) { - libraryRules.push(...rules); - } else { - productRules.push(...rules); + const own = filePath.includes(options.libraryPattern) + ? libraryRules + : productRules; + for (const rule of rules) { + if (isConstantRule(rule)) { + constantRules.push(rule); + } else { + own.push(rule); + } } } const render = (rules: unknown[], layer: string) => rules.length - ? `@layer ${layer} {\n${stylexBabelPlugin.processStylexRules(rules as never, {useLayers: true})}\n}` + ? `@layer ${layer} {\n${stylexBabelPlugin.processStylexRules([...constantRules, ...rules] as never, {useLayers: true})}\n}` : ''; return [ @@ -358,6 +393,52 @@ function renderSplitLayers( .join('\n\n'); } +/** + * The StyleX-generated custom properties `css` reads that it never declares. + * + * StyleX names the variables it generates `--` (a theme + * variable, or a constant waiting for its value). One that is read but never + * declared is a value that compiled to a variable nothing sets, so browsers + * drop the declaration or take its fallback: a constant whose defining module + * never reached the build, or a plain export read from a `.stylex.ts` file as + * if it were a token. The dynamic-style variables StyleX sets inline + * (`---`) and component variables with a dash after the + * prefix are not generated names and are never reported. + * + * SYNC: scripts/build-css.mjs `undefinedStyleXVariables` is the same check + * for the prebuilt package stylesheets. + */ +export function undefinedGeneratedVariables( + css: string, + prefixes: readonly string[], +): string[] { + const names = prefixes + .map(prefix => prefix.replace(/[^a-zA-Z0-9_-]/g, '')) + .filter(Boolean) + .join('|'); + const declared = new Set( + [...css.matchAll(new RegExp(`(--(?:${names})[a-z0-9]+)\\s*:`, 'g'))].map( + match => match[1], + ), + ); + const read = new Set( + [ + ...css.matchAll( + new RegExp(`var\\(\\s*(--(?:${names})[a-z0-9]+)\\s*[,)]`, 'g'), + ), + ].map(match => match[1]), + ); + return [...read].filter(name => !declared.has(name)).sort(); +} + +/** A `defineConsts` value as StyleX collects it: a rule with `constKey`/`constVal`. */ +function isConstantRule(rule: unknown): boolean { + const body = Array.isArray(rule) + ? (rule[1] as {constKey?: unknown; constVal?: unknown} | undefined) + : undefined; + return body?.constKey != null && body?.constVal != null; +} + /** * The build half of the split. StyleX appends one merged block of CSS to the * build's stylesheet, in its own top-level `@layer priority1…priorityN` — which @@ -387,6 +468,10 @@ function buildLayerSplitPlugin( libraryLayer: string; productLayer: string; lightningcssOptions?: unknown; + /** The StyleX class-name prefixes in play: library, then product. */ + classNamePrefixes: readonly string[]; + /** Warn about, or fail on, generated variables nothing declares. */ + undefinedVariables: 'warn' | 'error'; }, ): Plugin { const stylex = basePlugin as { @@ -414,6 +499,31 @@ function buildLayerSplitPlugin( options.lightningcssOptions, ); + const undeclared = undefinedGeneratedVariables( + split, + options.classNamePrefixes, + ); + if (undeclared.length > 0) { + const report = ( + options.undefinedVariables === 'error' ? this.error : this.warn + ).bind(this); + report( + 'astryx-build-layer-split: the StyleX stylesheet reads ' + + `${undeclared.length} generated custom ` + + `propert${undeclared.length === 1 ? 'y' : 'ies'} ` + + 'that nothing declares, so those declarations would be dropped ' + + 'in every browser:\n' + + undeclared + .map( + name => + ` ${name}, read by: ${rulesReading(split, name).join(' ') || '(rule not found)'}`, + ) + .join('\n') + + '\nA `.stylex.ts` export read in a style must be a ' + + '`defineVars`/`defineConsts` value whose module reaches the build.', + ); + } + const outDir = outputOptions.dir ? outputOptions.dir : outputOptions.file @@ -472,6 +582,14 @@ function buildLayerSplitPlugin( }; } +/** The rules of `css` that read `name`, for the failure message. */ +function rulesReading(css: string, name: string): string[] { + return [...css.matchAll(/[^{}]+\{[^{}]*\}/g)] + .map(match => match[0].trim().replace(/\s+/g, ' ')) + .filter(rule => rule.includes(`var(${name}`)) + .slice(0, 3); +} + /** Every `.css` file under a directory, recursively. */ function listCssFiles(dir: string): string[] { const out: string[] = []; @@ -576,6 +694,7 @@ function astryxStylexLegacy(options: AstryxVitePluginLegacyOptions): Plugin[] { libraryPattern = LIBRARY_PATTERN, stylexPrefix = 'astryx', layers = {}, + undefinedVariables = 'warn', } = options; const libraryLayer = layers.library ?? 'astryx-base'; @@ -586,6 +705,14 @@ function astryxStylexLegacy(options: AstryxVitePluginLegacyOptions): Plugin[] { ).lightningcssOptions; const validatedStylexOptions = { propertyValidationMode: 'throw', + // The unplugin applies its Vite default (true) only to its own StyleX + // instance, which runs after the babel wrapper below has already compiled + // every StyleX call. Without it here, a file that reads only `defineConsts` + // from a `.stylex.ts` module loses that import once `stylex.create` is + // compiled (the TypeScript transform drops it as unused), the module is + // never loaded, and its constants never reach the CSS: a media-query key + // fails the build and a value ships as an undefined custom property. + treeshakeCompensation: true, ...(stylexOptions as any), lightningcssOptions: { targets: LIGHTNINGCSS_TARGETS, @@ -677,6 +804,12 @@ function astryxStylexLegacy(options: AstryxVitePluginLegacyOptions): Plugin[] { lightningcssOptions: ( validatedStylexOptions as {lightningcssOptions?: unknown} ).lightningcssOptions, + classNamePrefixes: [ + stylexPrefix, + (validatedStylexOptions as {classNamePrefix?: string}) + .classNamePrefix ?? 'x', + ], + undefinedVariables, }), ]; } diff --git a/packages/cli/assets/docs/author-a-theme.doc.mjs b/packages/cli/assets/docs/author-a-theme.doc.mjs index 65190f615007e..e2eaf33c5a4f6 100644 --- a/packages/cli/assets/docs/author-a-theme.doc.mjs +++ b/packages/cli/assets/docs/author-a-theme.doc.mjs @@ -6,160 +6,452 @@ export const docs = { name: 'author-a-theme', title: 'Author a theme', category: 'guide', - keywords: ['defineTheme', 'create theme', 'custom theme', 'extends', 'component overrides', 'custom variants', 'adaptations', 'breakpoints', 'theme build', 'theme family', 'tokenVar', 'resolveThemeTokens', 'useTheme', 'palette', 'generate palette', 'seed colors', 'color ramp', 'icon registry'], - description: 'Create, customize, and build a theme: defineTheme, palette generation, token overrides, component style overrides, custom variants, responsive adaptations, and production builds.', + keywords: [ + 'defineTheme', + 'create theme', + 'custom theme', + 'extends', + 'component overrides', + 'custom variants', + 'adaptations', + 'breakpoints', + 'theme build', + 'theme family', + 'tokenVar', + 'resolveThemeTokens', + 'useTheme', + 'palette', + 'generate palette', + 'seed colors', + 'color ramp', + 'icon registry', + ], + description: + 'Create, customize, and build a theme: defineTheme, palette generation, token overrides, component style overrides, custom variants, responsive adaptations, and production builds.', sections: [ { id: 'creating-a-custom-theme', title: 'Custom themes', content: [ - {type: 'prose', text: "Use an installed built theme as the base for ordinary customization. Import it into your source and pass it as `extends` to `defineTheme({extends: importedTheme, ...})`. Build the result, then add the local slug. Only eject when you need to own and maintain a full source fork."}, - {type: 'code', lang: 'bash', label: 'Add to use, eject to fork', code: '{@command theme list}\n{@command theme add stone}\n{@command theme eject stone}\n{@command theme build src/themes/stone/stoneTheme.ts}\n{@command theme add stone}'}, - {type: 'prose', text: "For an annotated map of the whole surface (every defineTheme field, the token families, and the component override syntax, each with the CLI command that prints its reference), run `{@command theme template}`. It writes `theme.template.ts` into your project to read and copy from (`{@command init --features theme}` writes it as part of project setup)."}, - {type: 'prose', text: 'To apply a theme you created, see {@link generic:use-a-theme}. To generate colors from seed values, see the Palette section below. Before you eject a package theme instead, see when that fits in {@link generic:use-a-theme}.'}, + { + type: 'prose', + text: 'Use an installed built theme as the base for ordinary customization. Import it into your source and pass it as `extends` to `defineTheme({extends: importedTheme, ...})`. Build the result, then add the local slug. Only eject when you need to own and maintain a full source fork.', + }, + { + type: 'code', + lang: 'bash', + label: 'Add to use, eject to fork', + code: '{@command theme list}\n{@command theme add stone}\n{@command theme eject stone}\n{@command theme build src/themes/stone/stoneTheme.ts}\n{@command theme add stone}', + }, + { + type: 'prose', + text: 'For an annotated map of the whole surface (every defineTheme field, the token families, and the component override syntax, each with the CLI command that prints its reference), run `{@command theme template}`. It writes `theme.template.ts` into your project to read and copy from (`{@command init --features theme}` writes it as part of project setup).', + }, + { + type: 'prose', + text: 'To apply a theme you created, see {@link generic:use-a-theme}. To generate colors from seed values, see the Palette section below. Before you eject a package theme instead, see when that fits in {@link generic:use-a-theme}.', + }, ], }, { id: 'palette', title: 'Generate a palette', content: [ - {type: 'prose', text: 'Name a few seed colors and generate every shade a theme needs, light and dark; {@link generic:generate-a-palette} explains what a palette holds.'}, - {type: 'code', lang: 'json', label: 'Minimal palette config (palette.config.json)', code: '{"families": [{"id": "ocean", "seed": "#0074e2"}]}'}, - {type: 'code', lang: 'bash', label: 'Generate and preview', code: '# Print the palette without writing files\n{@command theme palette generate palette.config.json}\n\n# Write palette + receipt, and open a visual preview\n{@command theme palette generate palette.config.json} \\\n --out src/themes/ocean/tokens/ocean.palette.ts \\\n --preview preview.html'}, - {type: 'prose', text: "The command writes a `.ts` file exporting 21 shades (stops 0–100) for both light and dark, plus a `.receipt.json` to regenerate later. Point your theme's tokens at the palette, then run defineTheme (see below)."}, - {type: 'prose', text: 'For the full palette config fields and CLI flags, run `{@command docs cli/commands/theme-palette-generate}`. For the integration-authoring walkthrough of connecting a palette to theme tokens, run `{@command docs authoring/building-blocks/themes/generate-a-palette}`.'}, + { + type: 'prose', + text: 'Name a few seed colors and generate every shade a theme needs, light and dark; {@link generic:generate-a-palette} explains what a palette holds.', + }, + { + type: 'code', + lang: 'json', + label: 'Minimal palette config (palette.config.json)', + code: '{"families": [{"id": "ocean", "seed": "#0074e2"}]}', + }, + { + type: 'code', + lang: 'bash', + label: 'Generate and preview', + code: '# Print the palette without writing files\n{@command theme palette generate palette.config.json}\n\n# Write palette + receipt, and open a visual preview\n{@command theme palette generate palette.config.json} \\\n --out src/themes/ocean/tokens/ocean.palette.ts \\\n --preview preview.html', + }, + { + type: 'prose', + text: "The command writes a `.ts` file exporting 21 shades (stops 0–100) for both light and dark, plus a `.receipt.json` to regenerate later. Point your theme's tokens at the palette, then run defineTheme (see below).", + }, + { + type: 'prose', + text: 'For the full palette config fields and CLI flags, run `{@command docs cli/commands/theme-palette-generate}`. For the integration-authoring walkthrough of connecting a palette to theme tokens, run `{@command docs authoring/building-blocks/themes/generate-a-palette}`.', + }, ], }, { title: 'defineTheme', content: [ - {type: 'prose', text: 'defineTheme creates a theme from explicit token overrides and optional scale configs. Existing themes may use `color` to expand an accent seed into neutral, surface, text, border, and related values; it is not an accent-only edit. Keep that configuration when preserving existing output. For new complete palettes, generate and review a committed palette module, then import its `palette` export and reference stable stops from semantic tokens. This keeps mapped roles coherent when the accepted palette is regenerated; generation itself does not rewrite the theme. Other scale configs generate typography, radius, and motion values from parameters. Explicit token overrides always take precedence over scale-generated values, token by token. When an existing `color` config is in use, overriding `--color-accent` in `tokens` re-points `--color-accent-muted`, `--color-text-accent`, and `--color-icon-accent`, but `--color-on-accent` stays baked from the `color.accent` seed. Prefer a `[light, dark]` tuple for `color.accent` over overriding `--color-accent` alone. localTokens accepts any valid CSS custom-property name; prefixes do not establish ownership.'}, - {type: 'code', lang: 'tsx', label: 'defineTheme with generated palette references and non-color scale configs', code: "import {defineTheme} from '@astryxdesign/core/theme';\nimport {palette} from './palette.generated';\n\nconst myTheme = defineTheme({\n name: 'my-theme',\n typography: {\n scale: { base: 14, ratio: 1.2 },\n body: { family: 'Inter', fallbacks: '-apple-system, sans-serif' },\n },\n radius: { base: 4, multiplier: 1 },\n motion: { fast: 175, medium: 410, ratio: 0.75 },\n tokens: {\n '--color-background-surface': [\n palette.neutral.light[100],\n palette.neutral.dark[15],\n ],\n },\n});"}, - {type: 'table', headers: ['Config', 'Generates', 'Parameters'], rows: [ - ['color (existing themes)', '--color-accent, --color-background-*, --color-text-*, --color-border, etc.', 'Supported broad scale: accent? (hex or [light, dark] tuple), neutralStyle? (warm|cool|neutral), contrast? (standard|high). Use theme palette generate for new complete palettes.'], - ['typography.scale', '--text-heading-*-size/weight/leading, --text-body-size/weight/leading', 'base (px), ratio'], - ['typography.body/heading/code', '--font-family-body, --font-family-heading, --font-family-code', 'family, fallbacks?, url?, weight?'], - ['radius', '--radius-inner, --radius-element, --radius-container, --radius-page, --radius-chat', 'base (px), multiplier (0–2)'], - ['motion', '--duration-fast-min/fast/fast-max, --duration-medium-min/medium/medium-max', 'fast (ms), medium (ms), ratio, easing?'], - ]}, - {type: 'prose', text: 'For dark mode tuples in token values, see the Dark mode section of {@link generic:use-a-theme}.'}, + { + type: 'prose', + text: 'defineTheme creates a theme from explicit token overrides and optional scale configs. Existing themes may use `color` to expand an accent seed into neutral, surface, text, border, and related values; it is not an accent-only edit. Keep that configuration when preserving existing output. For new complete palettes, generate and review a committed palette module, then import its `palette` export and reference stable stops from semantic tokens. This keeps mapped roles coherent when the accepted palette is regenerated; generation itself does not rewrite the theme. Other scale configs generate typography, radius, and motion values from parameters. Explicit token overrides always take precedence over scale-generated values, token by token. When an existing `color` config is in use, overriding `--color-accent` in `tokens` re-points `--color-accent-muted`, `--color-text-accent`, and `--color-icon-accent`, but `--color-on-accent` stays baked from the `color.accent` seed. Prefer a `[light, dark]` tuple for `color.accent` over overriding `--color-accent` alone. localTokens accepts any valid CSS custom-property name; prefixes do not establish ownership.', + }, + { + type: 'code', + lang: 'tsx', + label: + 'defineTheme with generated palette references and non-color scale configs', + code: "import {defineTheme} from '@astryxdesign/core/theme';\nimport {palette} from './palette.generated';\n\nconst myTheme = defineTheme({\n name: 'my-theme',\n typography: {\n scale: { base: 14, ratio: 1.2 },\n body: { family: 'Inter', fallbacks: '-apple-system, sans-serif' },\n },\n radius: { base: 4, multiplier: 1 },\n motion: { fast: 175, medium: 410, ratio: 0.75 },\n tokens: {\n '--color-background-surface': [\n palette.neutral.light[100],\n palette.neutral.dark[15],\n ],\n },\n});", + }, + { + type: 'table', + headers: ['Config', 'Generates', 'Parameters'], + rows: [ + [ + 'color (existing themes)', + '--color-accent, --color-background-*, --color-text-*, --color-border, etc.', + 'Supported broad scale: accent? (hex or [light, dark] tuple), neutralStyle? (warm|cool|neutral), contrast? (standard|high). Use theme palette generate for new complete palettes.', + ], + [ + 'typography.scale', + '--text-heading-*-size/weight/leading, --text-body-size/weight/leading', + 'base (px), ratio', + ], + [ + 'typography.body/heading/code', + '--font-family-body, --font-family-heading, --font-family-code', + 'family, fallbacks?, url?, weight?', + ], + [ + 'radius', + '--radius-inner, --radius-element, --radius-container, --radius-page, --radius-chat', + 'base (px), multiplier (0–2)', + ], + [ + 'motion', + '--duration-fast-min/fast/fast-max, --duration-medium-min/medium/medium-max; --motion- for each touch motion role set', + 'fast (ms), medium (ms), ratio, easing? (together or not at all); springs?, durations?, curves? by role ({@link generic:motion})', + ], + ], + }, + { + type: 'prose', + text: 'For dark mode tuples in token values, see the Dark mode section of {@link generic:use-a-theme}.', + }, ], }, { title: 'Extending a Theme', content: [ - {type: 'prose', text: "`extends` lets you derive a new theme from an existing one, inheriting its tokens, component overrides, icons, and fonts. Only specify what you want to change; everything else carries over from the base theme."}, - {type: 'code', lang: 'tsx', label: 'Extending the neutral theme', code: "import {defineTheme} from '@astryxdesign/core/theme';\nimport {neutralTheme} from '@astryxdesign/theme-neutral';\nimport {myIcons} from './icons';\n\nconst brandTheme = defineTheme({\n name: 'brand',\n extends: neutralTheme,\n icons: myIcons,\n tokens: {\n '--color-accent': ['#7B61FF', '#9B85FF'],\n },\n});"}, - {type: 'table', headers: ['Field', 'Merge behavior'], rows: [ - ['tokens', 'Base tokens are copied first, then child tokens override on top.'], - ['components', 'Deep-merged: child component rules override matching keys from the base.'], - ['icons', 'Shallow-merged: child icons override matching names from the base.'], - ['indicators', 'Shallow-merged: child indicators override matching names from the base.'], - ['onDark, onLight', "Deep-merged per surface: the base's resolved surface first, then the child's overrides."], - ['typography, motion, radius, color', 'Child config replaces base entirely (these are scale inputs, not additive).'], - ['adaptations', 'Width-breakpoint overrides merge by fixed name. Inherited ordered rules keep their relative order; child rules append and re-resolve against the child root axes.'], - ]}, - {type: 'prose', text: "Inheritance is resolved when the theme is defined, so an extended theme is flat: `{@command theme build}` emits one self-contained stylesheet holding everything the child inherited, and the base theme's CSS does not need to be loaded next to it. A base that is not a theme (most often an import that missed) throws when `defineTheme` runs, rather than producing a theme that silently inherits nothing."}, - {type: 'prose', text: 'For the integration-specific guidance on mapping a palette to tokens when extending, see `{@command docs authoring/building-blocks/themes/define-the-theme}`.'}, + { + type: 'prose', + text: '`extends` lets you derive a new theme from an existing one, inheriting its tokens, component overrides, icons, and fonts. Only specify what you want to change; everything else carries over from the base theme.', + }, + { + type: 'code', + lang: 'tsx', + label: 'Extending the neutral theme', + code: "import {defineTheme} from '@astryxdesign/core/theme';\nimport {neutralTheme} from '@astryxdesign/theme-neutral';\nimport {myIcons} from './icons';\n\nconst brandTheme = defineTheme({\n name: 'brand',\n extends: neutralTheme,\n icons: myIcons,\n tokens: {\n '--color-accent': ['#7B61FF', '#9B85FF'],\n },\n});", + }, + { + type: 'table', + headers: ['Field', 'Merge behavior'], + rows: [ + [ + 'tokens', + 'Base tokens are copied first, then child tokens override on top.', + ], + [ + 'components', + 'Deep-merged: child component rules override matching keys from the base.', + ], + [ + 'icons', + 'Shallow-merged: child icons override matching names from the base.', + ], + [ + 'indicators', + 'Shallow-merged: child indicators override matching names from the base.', + ], + [ + 'onDark, onLight', + "Deep-merged per surface: the base's resolved surface first, then the child's overrides.", + ], + [ + 'typography, motion, radius, color', + 'Child config replaces base entirely (these are scale inputs, not additive).', + ], + [ + 'adaptations', + 'Width-breakpoint overrides merge by fixed name. Inherited ordered rules keep their relative order; child rules append and re-resolve against the child root axes.', + ], + ], + }, + { + type: 'prose', + text: "Inheritance is resolved when the theme is defined, so an extended theme is flat: `{@command theme build}` emits one self-contained stylesheet holding everything the child inherited, and the base theme's CSS does not need to be loaded next to it. A base that is not a theme (most often an import that missed) throws when `defineTheme` runs, rather than producing a theme that silently inherits nothing.", + }, + { + type: 'prose', + text: 'For the integration-specific guidance on mapping a palette to tokens when extending, see `{@command docs authoring/building-blocks/themes/define-the-theme}`.', + }, ], }, { title: 'Component Style Overrides', content: [ - {type: 'prose', text: 'The `components` field in defineTheme uses semantic component keys and style keys, not raw CSS selectors. Use `base` for all instances, `variant:value` or `stateName` for specific props/states, and let the theme pipeline choose the underlying selector. For raw external CSS escape hatches, prefer the data-attribute selector surface documented in {@link generic:styling-advanced}.'}, - {type: 'code', lang: 'tsx', label: 'Component overrides with standard CSS', code: "components: {\n card: {\n base: { borderRadius: '20px', padding: '24px' },\n },\n button: {\n base: {\n borderRadius: '9999px',\n textTransform: 'uppercase',\n '--button-focus-offset': '3px',\n },\n 'variant:ghost': { borderWidth: '2px', borderStyle: 'solid' },\n },\n}"}, - {type: 'prose', text: "Run `{@command theme targets}` for every themeable key in the system (`{@command theme targets }` to scope it, `--json` to lint a theme against it), and `{@command component }` for one component's theming targets, public CSS variables, and which standard CSS properties are supported."}, - {type: 'list', style: 'do', items: ['Write standard CSS properties (borderRadius, padding); the pipeline expands them into internal vars.', 'Set public CSS vars directly when no standard property equivalent exists.']}, - {type: 'list', style: 'dont', items: ['Set private CSS vars (prefixed --_) directly. Use standard CSS properties instead. `{@command theme build}` will error.', 'Set a public CSS var the component does not define. It compiles to CSS that never applies, and the build does not warn. Take the names from `{@command component }`.']}, + { + type: 'prose', + text: 'The `components` field in defineTheme uses semantic component keys and style keys, not raw CSS selectors. Use `base` for all instances, `variant:value` or `stateName` for specific props/states, and let the theme pipeline choose the underlying selector. For raw external CSS escape hatches, prefer the data-attribute selector surface documented in {@link generic:styling-advanced}.', + }, + { + type: 'code', + lang: 'tsx', + label: 'Component overrides with standard CSS', + code: "components: {\n card: {\n base: { borderRadius: '20px', padding: '24px' },\n },\n button: {\n base: {\n borderRadius: '9999px',\n textTransform: 'uppercase',\n '--button-focus-offset': '3px',\n },\n 'variant:ghost': { borderWidth: '2px', borderStyle: 'solid' },\n },\n}", + }, + { + type: 'prose', + text: "Run `{@command theme targets}` for every themeable key in the system (`{@command theme targets }` to scope it, `--json` to lint a theme against it), and `{@command component }` for one component's theming targets, public CSS variables, and which standard CSS properties are supported.", + }, + { + type: 'list', + style: 'do', + items: [ + 'Write standard CSS properties (borderRadius, padding); the pipeline expands them into internal vars.', + 'Set public CSS vars directly when no standard property equivalent exists.', + ], + }, + { + type: 'list', + style: 'dont', + items: [ + 'Set private CSS vars (prefixed --_) directly. Use standard CSS properties instead. `{@command theme build}` will error.', + 'Set a public CSS var the component does not define. It compiles to CSS that never applies, and the build does not warn. Take the names from `{@command component }`.', + ], + }, ], }, { title: 'Custom Variants', content: [ - {type: 'prose', text: "Themes can add new prop values to any component. Any `prop:value` key where the value isn't a built-in gets treated as a new variant. Use `{@command theme build}` to generate TypeScript augmentations for type safety."}, - {type: 'code', lang: 'tsx', label: 'Adding custom variants', code: "components: {\n button: {\n 'variant:secondary': { backgroundColor: 'rgba(0,0,0,0.06)' },\n 'variant:primary-muted': {\n backgroundColor: 'light-dark(#F2F4F6, #28292C)',\n color: 'var(--color-text-primary)',\n },\n },\n banner: {\n 'status:neutral': {\n backgroundColor: 'var(--color-background-muted)',\n color: 'var(--color-text-secondary)',\n },\n },\n}"}, - {type: 'code', lang: 'tsx', label: 'Using custom variants', code: "// TypeScript knows about 'primary-muted' after astryx theme build\n