feat(interaction): cast filmstrips, rect sampling, a Storybook highlight default and the WebKit shader cache

internal/interaction-harness/README.md

@@ -35,6 +35,12 @@ under a rootful docker daemon the container runs as your user so its
 node internal/interaction-harness/scripts/webkit-container.mjs [playwright test args…]
 ```
 
+WebKit composites through Mesa's software GL there, which compiles each shader
+the first time a page draws it (450–700 ms for the first frame of a blurred or
+newly animated layer). The script keeps Mesa's shader cache on the host
+(`~/.cache/astryx-webkit-shaders/<image>`, or `ASTRYX_WEBKIT_SHADER_CACHE`), so
+only the first run on a machine pays for them.
+
 Evidence (storyboards, curve traces) is written to
 `test-results/interaction-evidence/` (override with
 `ASTRYX_INTERACTION_EVIDENCE`) and attached to the test report. It is never
@@ -46,12 +52,12 @@ committed: a pull request links or attaches it.
 | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | `scripts.ts`    | Touch scripts as data: `tap`, `drag` (hold, rest), `flick` (exact final-move speed, velocity profiles), `holdThenDrag`, `pinch`, `rotate`, `path`, `merge` (a second finger mid-gesture), `releaseVelocity`, `toTouchEvents`.                                                |
 | `touch.ts`      | `touchDriver(page).play(script)`: the script as real engine input. `installInputLog` / `readInputLog`: what the page received.                                                                                                                                               |
-| `frames.ts`     | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `frameStats`.                                                                                                                             |
+| `frames.ts`     | `startScreencast` (timestamped composited frames), `markElement` (a marker chip or outline), `trackMarker` (marker box per frame), `sampleRects` (per-rAF rects), `frameStats`.                                                                                              |
 | `busy.ts`       | `blockMainThread(page, ms)` and `busyWindowProof(tracked, window)`: frames presented, frames that moved, longest stall, a verdict; `presented` adds the compositor's own frames.                                                                                             |
 | `trace.ts`      | `traceRendering(page, body)`: main-thread counts of UpdateLayoutTree / Layout / PrePaint / Paint / Layerize / UpdateLayer / ScrollLayer / HitTest, `countBetween`, long tasks; `installLongTaskLog` / `readLongTasks`; `startCompositorTrace`: when the compositor presents. |
 | `curves.ts`     | `holdAnimations`, `seekAnimations`, `sampleCurve` (presented transform / opacity at any t), `waitForAnimation`, `watchAnimations`, `waitForRest`, `releaseAnimations`.                                                                                                       |
 | `math.ts`       | Oracles independent of the code under test: `cubicBezier`, `timingFunction`, `springStep` (SwiftUI response / dampingFraction, initial velocity), `projectDecay`, `rubberBand`, `parseTransform`, `compareCurve`, `timeline`.                                                |
-| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event): PNG contact sheet + JSON trace.                                                                                                                                      |
+| `storyboard.ts` | `animationStoryboard` (seek to t = 0, 16.7, 33.3 … ms), `touchStoryboard` (a frame per scripted event), `castStoryboard` (real screencast frames): PNG contact sheet + JSON trace.                                                                                           |
 | `output.ts`     | `evidenceDir`, `writeEvidence`: where evidence goes and how it is attached to the report.                                                                                                                                                                                    |
 | `story.ts`      | `phone`, `withStorybook`, `openStory`, `frames`, and the timer gate (`installTimerGate`, `freezeTimers`, `thawTimers`).                                                                                                                                                      |
 
@@ -67,6 +73,14 @@ dispatch latency, and the events go through the compositor's gesture detection
 the limits below. Read positions a frame after a move: Chromium delivers
 `pointermove` aligned to animation frames.
 
+**Stories, not Storybook.** `openStory` turns Storybook's highlight module off
+on every page it opens, and checks that its root element is absent. Left on,
+the module observes `#storybook-root` and, on every insertion, reads the
+computed style of every element on the page inside the inserting task (117
+reads and two forced style recalculations for one exit copy in a Button story),
+so a proof would measure Storybook as well as the component.
+`openStory(page, origin, id, args, {storybookHighlight: true})` keeps it on.
+
 **Curve sampling.** Pause every running animation (CSS transitions and
 animations are Web Animations too), seek to a time, read `getComputedStyle`:
 the presented value at that exact time, independent of frame timing. Finished
@@ -140,7 +154,8 @@ every frame, so one late frame is not "1 of 1 moved"). Pass
 `maxMedianFrameMs` (20 at 60 Hz) only in a proof that claims display rate.
 `verdict.failures` names each miss with its numbers, and `summary` carries the
 raw numbers for the report. Frames are judged in presentation order: a loaded
-machine can deliver them out of order.
+machine can deliver them out of order. `castStoryboard` turns the frames of the
+block into a filmstrip.
 
 A just-started animation can reach the screen after the block began (50–100 ms
 later in WebKit on a loaded machine). When the cast saw the element at rest
@@ -191,14 +206,14 @@ backstop, a cleanup render) would do so behind a held storyboard's back:
 `installTimerGate` before the animation starts and `freezeTimers` while
 photographing. The page can still cancel a held callback, a held frame callback
 gets the time it actually runs, and the harness's own waits (its frame waits,
-`blockMainThread` with `during`) are not held.
+`blockMainThread` with `during`, `sampleRects`) are not held.
 An animation that repeats forever does not set a storyboard's default length;
 with nothing else to film, pass `times`. A touch filmstrip keeps the scripted
 event `timeStamp`s (Chromium), but a handler that reads `performance.now()` or
 `Date.now()` sees one event per photograph, about 100 ms apart, so a
 clock-timed release flings slower when filmed than when played: a storyboard
 cannot show such a gesture; record it in real time instead, with
-`startScreencast`. Chromium can return an empty copy of
+`startScreencast` and `castStoryboard`. Chromium can return an empty copy of
 the page's surface right after it changes ("Unable to capture screenshot"); a
 storyboard takes that photograph again two frames later, up to three attempts
 in all, and lists each retake under `recaptures` in the trace.

internal/interaction-harness/scripts/webkit-container.mjs

@@ -16,10 +16,19 @@
  * docker daemon the container runs as the invoking user (rootless podman and
  * rootless docker already map it), so the `test-results/` it writes into the
  * mounted checkout stay the host user's and the next host run can clear them.
+ *
+ * WebKit composites through Mesa's software GL in the image, which compiles
+ * each shader variant the first time a page draws it: 450–700 ms for the
+ * first frame of a blurred or newly animated layer, long enough for a short
+ * animation to end before it is ever presented. Mesa caches what it compiles,
+ * and the cache is kept on the host (`~/.cache/astryx-webkit-shaders/<image>`,
+ * or `ASTRYX_WEBKIT_SHADER_CACHE`), so a run starts with every shader an
+ * earlier run compiled instead of paying for them in its first tests.
  */
 import {spawnSync} from 'node:child_process';
 import fs from 'node:fs';
 import {createRequire} from 'node:module';
+import os from 'node:os';
 import path from 'node:path';
 import {fileURLToPath} from 'node:url';
 
@@ -42,6 +51,21 @@ if (engine == null) {
   process.exit(1);
 }
 const args = process.argv.slice(2);
+// Where the container sees the host's shader cache (see the header).
+const containerShaderCache = '/astryx-shader-cache';
+// Resolved to an absolute path: a bare relative value would be mounted as a
+// named volume rather than a host directory. An empty value means the default.
+const shaderCache = process.env.ASTRYX_WEBKIT_SHADER_CACHE
+  ? path.resolve(process.env.ASTRYX_WEBKIT_SHADER_CACHE)
+  : path.join(
+      os.homedir(),
+      '.cache',
+      'astryx-webkit-shaders',
+      image.replace(/[^\w.-]+/g, '_'),
+    );
+fs.mkdirSync(shaderCache, {recursive: true});
+// The harness self-test reads this marker inside the container.
+fs.writeFileSync(path.join(shaderCache, 'ASTRYX-SHADER-CACHE'), `${image}\n`);
 const env = [
   'ASTRYX_STORYBOOK_DIR',
   'ASTRYX_INTERACTION_EVIDENCE',
@@ -79,6 +103,10 @@ const command = [
   '--ipc=host',
   '-v',
   `${root}:${root}`,
+  '-v',
+  `${shaderCache}:${containerShaderCache}`,
+  '-e',
+  `MESA_SHADER_CACHE_DIR=${containerShaderCache}`,
   '-w',
   root,
   ...env,

internal/interaction-harness/src/frames.ts

@@ -7,11 +7,12 @@
  *   `Screencast` domain)
  * @output `startScreencast` (timestamped composited frames), `markElement`
  *   (a marker outline the frames can find), `trackMarker` (per-frame marker
- *   box), `frameStats`.
+ *   box), `sampleRects` (per-rAF rect sampling), `frameStats`.
  * @position Frame evidence for motion proofs. The screencast is captured off
  *   the page's main thread, from what the compositor presents, so it keeps
  *   delivering frames while a script blocks the main thread: that is what makes
- *   the busy-main-thread proof possible.
+ *   the busy-main-thread proof possible. Rect sampling is the cheap path when
+ *   the main thread is free.
  *
  * Marker tracking needs no image library: frames are decoded and scanned in a
  * scratch page of the same browser (createImageBitmap + canvas), so the
@@ -399,6 +400,71 @@ export async function trackMarker(
   }
 }
 
+export interface RectSample {
+  /** Epoch ms of the rAF callback. */
+  readonly t: number;
+  readonly box: Box | null;
+  readonly transform: string;
+  readonly opacity: number;
+}
+
+/**
+ * Samples the element's rect, transform and opacity on every animation frame
+ * for `durationMs`. Main-thread sampling: it stops while the main thread is
+ * blocked, which is the point of the screencast path above.
+ */
+export async function sampleRects(
+  locator: Locator,
+  durationMs: number,
+): Promise<RectSample[]> {
+  return locator.evaluate(
+    (element, duration) =>
+      new Promise<
+        {
+          t: number;
+          box: {x: number; y: number; width: number; height: number} | null;
+          transform: string;
+          opacity: number;
+        }[]
+      >(resolve => {
+        // The harness's own waits run on the real timers: a frozen timer gate
+        // (story.ts) holds only the page's.
+        const gate = (
+          window as unknown as {
+            __harnessTimerGate?: {
+              frame?: typeof requestAnimationFrame;
+            };
+          }
+        ).__harnessTimerGate;
+        const frame = gate?.frame ?? window.requestAnimationFrame.bind(window);
+        const samples: {
+          t: number;
+          box: {x: number; y: number; width: number; height: number} | null;
+          transform: string;
+          opacity: number;
+        }[] = [];
+        const end = performance.now() + duration;
+        const tick = (now: number) => {
+          const rect = element.getBoundingClientRect();
+          const style = getComputedStyle(element);
+          samples.push({
+            t: performance.timeOrigin + now,
+            box: {x: rect.x, y: rect.y, width: rect.width, height: rect.height},
+            transform: style.transform,
+            opacity: Number(style.opacity),
+          });
+          if (now < end) {
+            frame(tick);
+          } else {
+            resolve(samples);
+          }
+        };
+        frame(tick);
+      }),
+    durationMs,
+  );
+}
+
 export interface FrameStats {
   readonly frames: number;
   /** Frames whose tracked value differs from the frame before. */

internal/interaction-harness/src/harness.interaction.browser.spec.ts

@@ -26,14 +26,19 @@
  *   two frames later; a touch filmstrip keeps scripted event times while a
  *   handler clock sees its pacing; the timer gate holds a component timer
  *   until the filmstrip is done, drops callbacks the page cancels, stamps held
- *   frames with the time they run, and leaves the harness its own frames.
+ *   frames with the time they run, and leaves the harness its own frames; a
+ *   cast filmstrip keeps exactly the presented frames of its window, labelled
+ *   from its zero, as PNG.
  * @position The harness's own test. A component spec trusts these
  *   instruments; this file is why it may.
  */
 
+import fs from 'node:fs';
+import path from 'node:path';
 import {expect, test, type Page} from '@playwright/test';
 import {
   animationStoryboard,
+  castStoryboard,
   blockMainThread,
   busyWindowProof,
   compareCurve,
@@ -57,6 +62,7 @@ import {
   readLongTasks,
   releaseAnimations,
   sampleCurve,
+  sampleRects,
   seekAnimations,
   springStep,
   startCompositorTrace,
@@ -509,6 +515,83 @@ test.describe('frame evidence', () => {
     expect(proof.monotonic).toBe(true);
   });
 
+  test('in the WebKit container, compiled shaders outlive the run', async ({
+    browser,
+  }, testInfo) => {
+    // The official Playwright image keeps its browsers here.
+    test.skip(
+      testInfo.project.name !== 'webkit' ||
+        process.env.PLAYWRIGHT_BROWSERS_PATH !== '/ms-playwright',
+      'runs inside webkit-container.mjs',
+    );
+    // A cache inside the container dies with it: every run then compiled each
+    // shader again, 450–700 ms in the first frame of a blurred layer.
+    const dir = process.env.MESA_SHADER_CACHE_DIR;
+    expect(dir, 'MESA_SHADER_CACHE_DIR names the host-kept cache').toBeTruthy();
+    // Written on the host by the script before the container starts.
+    expect(
+      fs.readFileSync(path.join(dir ?? '', 'ASTRYX-SHADER-CACHE'), 'utf8'),
+    ).toContain('playwright');
+    // Mesa writes what it compiles there once a page draws a blurred layer.
+    // Earlier runs have filled the cache already, so this run's browser gets
+    // an empty directory beside them, in the same host-kept mount, and the
+    // test watches its own writes arrive.
+    const fresh = fs.mkdtempSync(path.join(dir ?? '', 'self-test-'));
+    try {
+      const own = await browser.browserType().launch({
+        env: {...process.env, MESA_SHADER_CACHE_DIR: fresh},
+      });
+      try {
+        const page = await own.newPage();
+        await page.setContent(
+          '<div id="glass" style="width:120px;height:80px;backdrop-filter:blur(16px) saturate(180%)"></div>',
+        );
+        await page.evaluate(
+          () =>
+            document
+              .getElementById('glass')
+              ?.animate([{scale: '1'}, {scale: '0.9'}], {duration: 200})
+              .finished,
+        );
+      } finally {
+        await own.close();
+      }
+      expect(fs.readdirSync(fresh, {recursive: true})).not.toEqual([]);
+    } finally {
+      fs.rmSync(fresh, {recursive: true, force: true});
+    }
+  });
+
+  test('rect sampling reads one rect per animation frame while the main thread is free', async ({
+    page,
+    browserName,
+  }) => {
+    await fixture(page);
+    const box = page.locator('#box');
+    await box.evaluate(el => {
+      el.animate(
+        [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+        {duration: 600, easing: 'linear'},
+      );
+    });
+    const samples = await sampleRects(box, 400);
+    const xs = samples.map(s => s.box?.x ?? NaN);
+    const gaps = samples
+      .slice(1)
+      .map((s, i) => s.t - samples[i].t)
+      .sort((a, b) => a - b);
+    const cadence = gaps[Math.floor(gaps.length / 2)];
+    expect(cadence).toBeLessThan(34);
+    expect(samples.length).toBeGreaterThanOrEqual(10);
+    if (browserName === 'chromium') {
+      // One sample per presented frame at 60 Hz. (Headless WebKit's animation
+      // frames are irregular, so only the cadence is checked there.)
+      expect(samples.length).toBeGreaterThanOrEqual(20);
+    }
+    expect(xs.every((x, i) => i === 0 || x >= xs[i - 1])).toBe(true);
+    expect(xs.at(-1) ?? 0).toBeGreaterThan(xs[0] + 80);
+  });
+
   test('the trace counts no Layout for a transform animation and one per frame for `top`', async ({
     page,
     browserName,
@@ -1937,7 +2020,7 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
     }
   });
 
-  test("the harness's own waits run on the real timers: a frozen timer gate does not hold blockMainThread's block", async ({
+  test("the harness's own waits run on the real timers: a frozen timer gate holds neither blockMainThread's block nor sampleRects", async ({
     page,
   }) => {
     await fixture(page);
@@ -1961,6 +2044,7 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
     expect(
       await within(blockMainThread(page, 100, {during: box, timeout: 2_000})),
     ).toBe('settled');
+    expect(await within(sampleRects(box, 200))).toBe('settled');
     await thawTimers(page);
   });
 
@@ -2149,4 +2233,58 @@ test.describe('storyboards', {tag: '@storyboard'}, () => {
     expect(state).toMatchObject({timer: false, frame: false});
     expect(state.heldFor).toBeGreaterThanOrEqual(140);
   });
+
+  test('a cast filmstrip keeps exactly the presented frames of its window, labelled from its zero', async ({
+    page,
+  }, testInfo) => {
+    await fixture(page);
+    const box = page.locator('#box');
+    // JPEG frames (WebKit's are always JPEG) take the conversion path.
+    const cast = await startScreencast(page, {format: 'jpeg'});
+    await box.evaluate(el =>
+      el.animate(
+        [{transform: 'translateX(0px)'}, {transform: 'translateX(200px)'}],
+        {duration: 600},
+      ),
+    );
+    await page.waitForTimeout(500);
+    const presented = await cast.stop();
+    expect(presented.length).toBeGreaterThan(8);
+    // A window of the middle frames, zeroed on one of them.
+    const zero = presented[2].t;
+    const from = presented[3].t;
+    const to = presented[presented.length - 3].t;
+    const inside = presented.filter(f => f.t >= from && f.t <= to);
+    const seen: number[] = [];
+    const film = await castStoryboard(page, presented, {
+      name: 'harness-self-test-cast',
+      zero,
+      from,
+      to,
+      data: t => {
+        seen.push(t);
+        return {t};
+      },
+    });
+    expect(film.frames.map(f => f.t)).toEqual(inside.map(f => f.t - zero));
+    expect(film.frames.map(f => f.label)).toEqual(
+      inside.map(f => `t=${(f.t - zero).toFixed(1)} ms`),
+    );
+    expect(seen).toEqual(inside.map(f => f.t));
+    // Every cell is a PNG, whatever the screencast delivered.
+    for (const frame of film.frames) {
+      expect([...frame.png.subarray(0, 4)]).toEqual([0x89, 0x50, 0x4e, 0x47]);
+    }
+    const files = await film.write(testInfo);
+    expect(files['sheet.png']).toMatch(/sheet\.png$/);
+    // A window with no presented frame in it films nothing.
+    const empty = await castStoryboard(page, presented, {
+      name: 'harness-self-test-cast-empty',
+      zero,
+      from: to + 1,
+      to: to + 2,
+    });
+    expect(empty.frames).toEqual([]);
+    await expect(empty.write(testInfo)).rejects.toThrow(/no frames/);
+  });
 });