Repository navigation
feat(interaction): Chromium rendering traces, the compositor presentation record and the long-task log - #8548
Merged
Merged
Conversation
Adds internal/interaction-harness (@astryxdesign/interaction-harness) and playwright.interaction.config.ts: phone-sized touch projects in Chromium and WebKit for *.interaction.browser.spec.ts specs. Touch scripts are data (tap, drag with hold and rest, flick with an exact release velocity and velocity profiles, hold-then-drag, pinch, rotate, any path, a second finger mid-gesture), played as real engine input: CDP Input.dispatchTouchEvent with scripted timestamps in Chromium, WebKit's own input command in WebKit. The harness's self-test proves the timestamps, a native fling and multi-touch delivery; a pr-interaction CI job runs the proofs against each PR's Storybook build, and a container script runs the WebKit project where host libraries are missing.
… Chromium The Chromium driver sent a lift of one finger of several as a touchMove listing the fingers still down, assuming CDP releases a point a move omits. It does not: the lifted finger stayed down. A touchEnd that lists the lifting fingers releases exactly those while the others keep moving, so the driver sends that, and a self-test lifts one of two fingers and checks the release, the other finger's moves and the touchend's remaining touches. Playwright's WebKit sends that touchend with no remaining touches, which iOS never does. The harness does not work around it: drivers expose partialLifts, and PARTIAL_LIFT_SKIP is the reason a proof cites when it skips; the README lists it with the other engine limits.
… lift profiled flicks at their velocity - pr-interaction now always reports and the `test` join requires its success (AST-030 FR7, FR11), shaped like test-native-swift: a Linux interaction-scope job decides, with the base's scope helper and failing closed, whether the harness, a component, Storybook, internal/a11y-spec, the workflow, package.json or pnpm-lock.yaml changed. Only then do the proofs take a 4-core runner; otherwise, and in merge groups (as with pr-a11y), the job passes without work. - A profiled flick's last move runs at velocity times profile(1), so the finger lifts at the documented velocity, and releaseVelocity reads the last move instead of a 100 ms window mean. - Docs: drop the /math subpath this package does not export yet; scripted timestamps arrive to 0.1 ms; the job comment describes the self-test it runs.
…ed its previous position A sample that repeats a position is not a move: frames that merge() inserts for another finger, where this finger did not move, made the last move look shorter, so a 10 Hz drag merged with a tap read 375 px/s instead of 300 and a 60 Hz drag whose other finger lifted 5 ms before its last move read about 3.6x too fast. The previous sample is now the first one at the previous position, as the page sees it, with a merged case in scripts.test.ts.
Pause every Web Animation, seek it to a time and read the presented transform and opacity, then compare the curve with oracles that share no code with the motion under test: cubic-bezier, the SwiftUI spring (response, dampingFraction, initial velocity), UIScrollView deceleration and UIKit's rubber band. Storyboards photograph seeked animations at t = 0, 16.7, 33.3 ... ms or each step of a scripted touch, and write a PNG contact sheet plus the JSON trace a spec asserts, under the gitignored test-results/interaction-evidence. A timer gate holds a component's own timers while a filmstrip is taken. pnpm storyboards runs the tagged tests.
…sented values An animation that keeps its end value in a run of identical final keyframes (held so a later interruption can read it) reported its whole duration as the curve: storyboards filmed the hold, waitForAnimation's interval reached past the landing, and waitForRest, which also required every animation to leave the running state and read only transform, timed out. The curve now ends where that run starts (HeldAnimation.curveEndMs, waitForAnimation's to, the default storyboard times), and rest is no animation pending or before its curve's end plus the presented transform, translate, rotate, scale and opacity (and ::backdrop opacity) still for 3 frames, whatever the playState. sampleCurve composes translate, rotate and scale with transform, so curves of the individual properties are sampled.
…film no endless curve, let the page cancel gated callbacks - holdAnimations holds only running document-timeline animations: a finished fill animation keeps its end value, an animation the page paused stays paused (release resumes only what was held), and a scroll-driven one is not seeked. Each held animation keeps its offset from the earliest start, so staggered starts stay staggered. - A sample or storyboard frame at or past an end reports the time it was read at (presentedAt, half a millisecond inside the end), and curveOf reads oracles there. - An animation that repeats forever no longer sets a storyboard's default length (it ran the worker out of memory); with nothing else to film, animationStoryboard asks for times. - The timer gate queues held callbacks by id, so the page's clearTimeout and cancelAnimationFrame still cancel them; a held frame callback gets the time it runs; frames(), waitForRest and touchStoryboard use the real requestAnimationFrame, so they resolve while frozen. - The storyboard self-tests carry the @Storyboard tag that `pnpm storyboards` selects.
…s from the element's own animations, resume endless ones
- animationStoryboard's default times run from the earliest start to the
latest curve end among the animations that end, each seek at
origin + t: beside a spinner that started earlier the animation under
test no longer shows its start in every frame, and a second animation
that starts later is filmed to its end.
- A sample's presentedAt (per channel: transform, opacity) comes from the
sampled element's own animations, by keyframe property, not from any
animation on the page.
- releaseAnimations({finish: true}) resumes an animation that repeats
forever instead of finishing it (which throws), and always clears the
held set.
- The interaction proofs' scope includes packages/themes/*/src/.
…before it looked waitForAnimation polled getAnimations() after the action, so an entrance shorter than the first look (a cold page answers its first evaluate late) had already finished: "no animation started". The page now records every animation from the first call into curves.ts on it, and on every later document: a frame loop on its own requestAnimationFrame, the CSS animation and transition events, and Element.animate as it is called, pending animations included (WebKit can start and finish a short script animation between two frames of a fresh page). waitForAnimation also reports an animation that started after the last waitForRest or waitForAnimation and has finished, with finishedAt. watchAnimations(locator) arms the record before an action that may be the page's first contact with the harness.
…uest This pull request keeps the curve-sampling instruments: holding and seeking animations, sampling presented values, the oracles, waiting for an animation and for rest. The storyboards (storyboard.ts, the evidence output, the timer gate and the `pnpm storyboards` script) and their tests move to a pull request stacked on this one.
`holdAnimations` replaced the held set on each call, so a second hold left the first set paused after `releaseAnimations` (it resumed only the newest set), and `waitForRest` then reported the frozen frame as rest. A later hold now adds to the set: everything held stays held until the release, and offsets count from the earliest start of all of it. The self-test holds two elements in turn, releases, and checks the first runs to its end; on the old code it stays frozen 50 px into a 300 px move. The self-test's header and the `curves.ts` SYNC line no longer describe the storyboard and timer-gate layer, which moved to its own pull request.
animationStoryboard films held animations by seeking them (t = 0, 16.7, 33.3 … ms, from the earliest start to the latest curve end among the animations that end, each at its own offset); touchStoryboard takes a frame per scripted touch event. Each writes a PNG contact sheet and a JSON trace of its measurements to the evidence directory and attaches them to the report; the spec asserts on the trace. The timer gate (installTimerGate, freezeTimers, thawTimers) holds the page's timers and frame callbacks while a storyboard photographs, so a component that finishes an animation from a timer cannot do it behind the held animation's back; the page can still cancel a held callback, and the harness's own frame waits (and the animation recorder) keep the page's real requestAnimationFrame. `pnpm storyboards` runs only the tests tagged @Storyboard.
`touchStoryboard` said velocity-dependent behavior is unchanged in Chromium. That holds for velocity read from event timestamps, which keep their scripted times, but a handler that reads `performance.now()` or `Date.now()` (Item's and Toast's swipes) sees one event per photograph, about 100 ms apart, so a flick that dismisses when played can settle back when filmed. The doc and the README now state that limit and point such gestures at a screencast, and a self-test shows both clocks: scripted 16.7 ms gaps in `timeStamp`, more than two frames in the handler's clock.
Timestamped composited frames (Chromium's screencast and WebKit's own), a marker the frames can find, and the busy-main-thread proof: block the page's main thread mid-flight and show the element still moving every frame, with a requestAnimationFrame animation that freezes as the negative control. Chromium trace counts of Layout, Paint and style work inside an animation window, and long tasks. The exemplar spec proves the BottomSheet drag to a detent with all of it: the drag follows the finger, a flick dismisses, the settle matches its timing function, keeps advancing through a blocked main thread in Chromium and WebKit, does no Layout or Paint, and gets a drag and settle storyboard.
…ed machine The proof asserted a stall under 40 ms and near-exact frame counts, so a dropped screencast frame or two on a loaded runner failed it, and frames delivered out of order read as the element stepping backwards. busyWindowProof now returns a verdict that tolerates what load does to a screencast and nothing a frozen animation does: a median frame interval of at most 20 ms, no stall longer than three screencast intervals, and at least 90% of the window's whole frame intervals presenting a new position (one miss always allowed), with the raw numbers in a summary. Screencast frames are sorted by presentation time, the cadence comes from the frames around the window, and blockMainThread can block from inside the page in the frame an animation has run, so a short settle is still judged at length on a busy machine. The exemplar judges its settle until it is within 2 px of rest, since the screencast rounds to whole pixels. A unit test covers the limits; the rAF negative control still fails every criterion.
…ading them from the animation The settle-curve oracle took its easing and duration from the animation under test, so it passed a 900 ms ease-in or a 150 ms linear settle as readily as the real one. The spec now asserts the theme's values (the default --ease-standard, cubic-bezier(0.24, 1, 0.4, 1), over the neutral theme's --duration-medium, 300 ms) and builds the oracle from them; the header no longer claims 410 ms.
…ebKit frames on its clock - `blockMainThread`'s `during` path ran its loop inside the rAF callback of the frame that saw the animation. WebKit hands a newly started animation to its compositor in that frame's rendering update, which runs after rAF callbacks, so the loop held the hand-off and even a compositor animation froze (2 of 29 frames moved). The loop now runs from a task queued in that frame. - WebKit's screencast frames were rebased onto the wall clock by taking the first frame as the start, which shifted every frame earlier by however long that frame took to come (60–100 ms). They are now rebased by the measured offset between the wall clock and CLOCK_MONOTONIC, which WebKit stamps them on. - `busyWindowProof` fails a judged part shorter than four frames (the animation under test was not moving during the block) instead of passing it, takes the end value from `settledAt` when given (WebKit can present a stale position once an accelerated animation ends while the main thread is blocked), and reports where the longest stall starts. - The BottomSheet busy case arms the block before the release and reads where the settle ends from the page.
Retargeting a CSS transition cancels it. If that happened while it was held, `seekAnimations` and `sampleCurve` set its time again, which revived it over the page's own new transition, and `releaseAnimations` replayed it from the start. Seeking, sampling and releasing now drop an `idle` animation from the held set first, and times count from a fixed origin (the earliest start held until the release), so dropping one never moves the others. The self-test retargets a held transition, seeks, releases and waits for rest: the cancelled transition stays idle and the box ends where the page sent it. On the old code the cancelled transition is still held after the seek.
# Conflicts: # internal/interaction-harness/src/harness.interaction.browser.spec.ts
# Conflicts: # internal/interaction-harness/src/harness.interaction.browser.spec.ts
… instrument `touchStoryboard`'s doc and the README pointed a clock-timed gesture at `startScreencast` and `castStoryboard`, which this branch does not have. They now say a storyboard cannot show such a gesture and to record it in real time.
…cast instruments With frame evidence in the harness, a clock-timed gesture a storyboard cannot show is recorded with `startScreencast` and `castStoryboard`: the doc and the README name them.
`pause()` only schedules the pause: the engine takes the hold time at its next frame, so a held animation stayed held a frame past `heldAtMs`, and a read right after the hold varied by a frame between runs. Setting the current time right after `pause()` completes a pending pause at once, where `heldAtMs` read it. The self-test holds a 1 s move and checks it is still at `heldAtMs` two frames later; on the previous code it reads one frame later in Chromium (16.6–16.7 ms) and 15–70 ms later in WebKit.
# Conflicts: # internal/interaction-harness/src/harness.interaction.browser.spec.ts
# Conflicts: # internal/interaction-harness/src/harness.interaction.browser.spec.ts
`blockMainThread({during})` counted an animation as running as soon as its
current time reached `minProgressMs`. An animation created in the frame the
page looks is still pending, not on the compositor yet, and at
`minProgressMs: 0` the block started on it and froze it: in WebKit
deterministically, and in Chromium on a Lightbox page while the block still
ran inside the frame's callback. A pending animation no longer counts, and
`minProgressMs` is raised to one frame (16 ms), since in WebKit an
animation's start time resolves before its layer is flushed.
The self-test creates the animation in the frame the helper looks and
blocks at `minProgressMs: 0`. On the previous code it fails 6/6 in WebKit
(0 frames presented through the block); now it passes 10/10 in WebKit and
6/6 in Chromium.
![astracat-bot[bot]](https://avatars.githubusercontent.com/in/4796231?s=60&v=4){kind=small}
There was a problem hiding this comment.
Changes requested: the trace, compositor-record and long-task instruments pass their positive and negative controls in Chromium, and the earlier readLongTasks and README points are fixed, but two things remain before this can go in.
startCompositorTracekeeps everyDisplay::FrameDisplayedevent in the browser (a page in another context feeds it too), and those events carry no page identity, so with a second page animating, a 240 ms stall inside the block passesbusyWindowProofwithpresented(15 frames credited) while it fails without that page; please have it throw or return no record when another page is open, say that the record is browser-wide, and add that case as a negative control.- The diff is still 2,693 authored lines because it carries the open #8045 beside this pull request's own 846; please land #8045 first and update this branch so its diff is the trace layer alone.
[Automated review]
Comment on lines
+280
to
+283
| return events | ||
| .filter(e => e.name === 'Display::FrameDisplayed') | ||
| .map(e => epoch + (e.ts - mark.ts) / 1000) | ||
| .sort((a, b) => a - b); |
There was a problem hiding this comment.
Display::FrameDisplayed carries no page identity, and this keeps every display's frames. In Chromium, with a second page of the same context animating, a 240 ms stall inside the block passed busyWindowProof with this presented (15 frames credited as skipped by the screencast), while it failed without that page; a still page's record went from 0 to 30 frames in 500 ms beside an animating page, in the same context or another. Since the events can't be attributed to a page, consider throwing (or returning no record) when the browser has another page open at start or stop, and saying in the JSDoc and README that the record is browser-wide.
4 tasks done
startCompositorTrace keeps every Display::FrameDisplayed in the browser, and those events carry no page identity: with a second page animating, a stall inside the block passed busyWindowProof with presented, crediting the other page's frames. It now throws when another page is open as it starts or stops, and says the record is browser-wide (JSDoc and README). Negative control: another animating page makes it refuse to start, and a page opened mid-trace makes stop() refuse.
cixzhang
added a commit
that referenced
this pull request
Oct 26, 2026
… against its span In WebKit the rAF negative control could read "1/1 moved": one frame from before the block arrived 45 ms into it, and the moved criterion divided by the delivered frames. A part that delivered fewer than minJudgedFrames frames is now judged against its span, as one that delivered none was (busy.test.ts case; it fails on the previous rule). The README and the self-test header now say only the busy-main-thread proof carries a negative control beside its positive check.
…into n8548 # Conflicts: # internal/interaction-harness/README.md
The README's busy-main-thread paragraph and busyWindowProof's JSDoc said an undelivered frame is never a miss; when fewer than minJudgedFrames frames were delivered in the judged part, its span counts every frame. One clause in each says so.
vjeux
added a commit
to vjeux/astryx
that referenced
this pull request
Oct 26, 2026
…ss renders; the base moves to the foundation heads The base (3f66f76) is the sheet's Gboard ride (meta#8566, on meta#8559 6be79f4, meta#8484 and meta#8076 62ecda1) merged with meta#8554 e4f3c8d, on cixzhang's meta#8446 87e516b, meta#8549, meta#8548 and meta#8045, with no conflict. meta#8559 moved `useKeyboardHeight` here, to its first consumer: ChatLayout lays its dock and transcript padding out from it. Its subscribe and snapshot functions are now stable for the component's life (per window), so React keeps one subscription across renders; before, every render unsubscribed and resubscribed, and a lone reader detached the module's listeners and measured again on each keyboard change (INV11). Red first: "reads the layout once per keyboard change for a lone component" expected no listener re-added and one read, and got two `addEventListener` calls. The hook's tests (one re-render per change, a keyboard already up, StrictMode's replay leaving no listener) and the README's layout example come with it.
There was a problem hiding this comment.
Changes requested: the trace, compositor-record and long-task instruments pass their positive and negative controls in Chromium, and the compositor record now refuses another page that is open as it starts or stops, but two things remain before this can go in.
- A page opened after
startCompositorTraceand closed beforestop()is still credited: with a spinning page opened right after the record started and closed after a 700 ms block, a 240 ms hold passedbusyWindowProofwithpresentedin 6 of 7 runs while it failed on the screencast alone, so please also refuse any page that appears during the record (for example by watching for new pages from start to stop) and add that case to the negative control. - The diff is still 2,771 authored lines because it carries the open #8045 beside this pull request's own 912, so please land #8045 first and update this branch so its diff is the trace layer alone.
[Automated review]
| await cdp.send('Tracing.end'); | ||
| await complete; | ||
| await cdp.detach().catch(() => {}); | ||
| assertOnlyPage(page, 'as the trace stops'); |
There was a problem hiding this comment.
This check, like the one at start (:269), sees only the pages open at that instant, so a page opened after the record starts and closed before stop() is never seen, and its Display::FrameDisplayed events are kept. In Chromium, with a spinning page opened right after startCompositorTrace and closed after a 700 ms block, a 240 ms hold passed busyWindowProof with presented in 6 of 7 runs (13–15 frames credited), in the same context or another, while the screencast alone failed every time. Watching for new pages from start to stop (context.on('page') on every context, or Target.targetCreated on a browser CDP session) and throwing here when one appeared would close it.
…ile it ran A page opened after startCompositorTrace and closed before stop() was still credited: its frames are in the browser-wide record, and neither check saw it. The record now watches for new pages from start to stop, in the page's own context and through the browser's target discovery in every context, and stop() throws if one opened. Negative controls: a page opened and closed mid-record, and one in another context, both make stop() refuse; alone, it records.
There was a problem hiding this comment.
Changes requested, for scope only: the trace, compositor-record and long-task instruments pass their positive and negative controls in Chromium, and the compositor record now refuses a page that opens while it runs. The diff is still 2,847 authored lines because it carries the open #8045 beside this pull request's own 988, so please land #8045 first and then update this branch so its diff is the trace layer alone.
[Automated review]
Contributor
|
An offer for this stack's conflicts with main since #8045 merged: two branches, each your head plus one main merge, where the harness files take main's #8045 text with only each pull request's own changes on top (the same resolution as your main merge on #8446):
|
vjeux
added a commit
to vjeux/astryx
that referenced
this pull request
Oct 29, 2026
There was a problem hiding this comment.
No changes needed: the rendering trace, the compositor presentation record and the long-task log serve the spec:AST-094 FR5 browser evidence, and their positive and negative self-tests pass in Chromium, with the WebKit fallbacks behaving as documented. With #8045 landed, the diff is now this layer alone. Optional: the README recipe for startCompositorTrace could say to stop the record before trackMarker, which opens a scratch page and would otherwise make stop() throw.
[Automated review]
| still presents. Pass when the compositor presented each frame (epoch ms on the | ||
| page clock, from a trace of `Display::FrameDisplayed`) as `presented`. A gap between two moved | ||
| still presents. `startCompositorTrace(page)`, started with the screencast and | ||
| stopped after the block, records when the compositor presented each frame |
There was a problem hiding this comment.
Optional: trackMarker and the storyboards open a scratch page (src/frames.ts:308, src/storyboard.ts:188), so a proof that calls trackMarker before stop() gets "another page opened during the record". Saying "stopped after the block and before trackMarker" here would save authors that detour; the self-test already stops it in that order.
cixzhang
marked this pull request as draft
October 30, 2026 15:00
cixzhang
marked this pull request as ready for review
October 30, 2026 15:01
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #8045 (frame evidence), which is stacked on #8348. Until those land, this diff includes them; this pull request's own change is its single commit.
What it adds
Chromium-only instruments for interaction proofs, split out of #8045 for scope (the code is unchanged):
traceRendering(page, body)counts the main thread's rendering work (Layout, Paint, UpdateLayoutTree, PrePaint, …) between two marks, andcountBetweenreads an animation's steady middle: the "no Layout / Paint during a transform animation" proof.startCompositorTrace(page)records when the compositor presented each frame, whichbusyWindowProoftakes aspresentedso a frame the screencast skipped is not mistaken for a stall.installLongTaskLog/readLongTasksrecord the page's own long tasks. Harness reads mark themselves (console.timeStamp('harness-read')), so a trace reports them indevtoolsTasks, apart from the page'slongTasks.Who it helps
Authors of interaction proofs: a proof can show that an animation costs the main thread nothing per frame, and the busy-main-thread verdict stays correct on a loaded machine.
Testing
harness.interaction.browser.spec.ts): the trace counts no Layout for a transform animation and one per frame fortop; the long-task log records the page's own long task but not a DevTools evaluate; a harness read inside a trace is not a page long task; screencast-skipped frames the compositor presented are not a stall, and a frozen animation still fails. 49 tests × 2 repeats pass.busy.test.ts(29 pass), including the recorded skip fixture.internal/), nothing published: no changeset.Co-authored with vjeux.