Repository navigation
perf(chat): render a streaming reply's growing tail without the deferred pass - #23487
Pearcekieser wants to merge 1 commit into
Conversation
First Principles Review (Opus 5.5, fork) — ✅ PASSPremise-level review of First-Principles-Verdict: PASS Nothing blocks. Issue #23486 and the tests that fail on main show the defect is real, and the fix changes the deferral that caused it. The one note is a hook parameter that only one caller uses. Notes
What this change shipsInventory (5 items) — 4 justifiedIntent: let a person on a slow phone watch a streaming reply's text arrive live, and stop a second page render on every chunk on desktop. This is a FIX.
[FIRST-PRINCIPLES-REVIEWED] 0a130c2 Description read[DESCRIPTION-READ] 4c751e94f95f145e1fc30a2896719b78931c1990a66fe557c254d8d39afbcbab That sha256 names the PR description this verdict read. Recompute it with |
UX Review (Opus 5.5, fork) — ✅ PASSUX-level review of UX-Verdict: PASS The PR adds or changes no control, label or string, so there is nothing to check. Streamed text reaches rows that are already on screen sooner, and no element changes form or place. [UX-REVIEWED] 0a130c2 Description read[DESCRIPTION-READ] d47565bc5adeb25caa8c13f8be2ee94fba1ddadec81ca7209930f6cd6ee1d0dc That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Design Review (Opus 5.5, fork) — ✅ PASSDesign-level review of Design-Verdict: PASS Nothing blocks. The change only affects rendering performance and is easy to revert: Notes
[DESIGN-REVIEWED] 0a130c2 Description read[DESCRIPTION-READ] c6d7ca982ba74d41dcf8a1c3f89a847936eaea8936ebba14871f935cbd7b95bf That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with |
Opus 5.5 Review (fork) — ✅ no blocking findingsReviewed |
303a573 to
e7331fa
Compare
|
self-added: yes
|
|
|
|
GPT 6.1 Review (fork) — ✅ no blocking findingsReviewed Review detailsNo findings. |
e7331fa to
0a130c2
Compare
|
self-added: yes
|
|
self-added: yes
|
|
|
|
|
|
|
|
HOLD (unanswered review note) at 0a130c2. The change is sound. Tail-only growth now renders at urgent priority, the earlier refs-written-during-render findings are fixed with a regression test, and Opus and GPT report no findings at this head. CI and PR Readiness are green. One Design Review note at this head has no answer. While a deferred change of shape is pending (for example a tool row arriving mid-reply), later tail chunks keep showing the last live value until the background pass commits (useSlotDeferredValue.ts#L66-L75). On a throttled phone, that can bring the freeze back for that stretch. The current-head design disposition replies only to the PASS punchline. To clear: reply to that note on the PR. Either give a measurement (the benchmark with mid-stream tool rows) or state it as a known limitation in the description, and link an issue if follow-up work is planned. — crew-pr-reviewer (r123) |
|
iamwhatever
left a comment
There was a problem hiding this comment.
Requesting changes at 0a130c2: one planned follow-up has no issue.
The code is sound. Tail-only growth renders at urgent priority, a pending shape change shows the last live value instead of the stream-start frame, and the refs are written only on commit (useSlotDeferredValue.ts#L66-L75). Every AI lane passes at this head and CI is green.
Your design disposition accepts the stale-tail window as a known limitation and says the per-row content subscription is planned as the next change in this series. That is a deferred follow-up, and nothing tracks it. Please open an issue for it (the stale window while a shape change is pending, and the per-row subscription that removes it) and link it in the PR description.
— crew-pr-reviewer (r124)
|
question (non-blocking): on the r124 change request, the tracking issue. This PR leaves no work deferred, so I would rather not open an issue on its own. The window you name, where tail chunks wait while a background pass for a change of shape is pending, is not something this PR adds. It is main's behaviour for every chunk, narrowed here to the stretch between a shape change and its commit, and the hook never shows text older than what is already on screen. Nothing in this PR is owed to close it. The per-row content subscription is the next change in this series. It will arrive as its own PR with its own issue and benchmark numbers, and that issue will describe this window as its motivation. My earlier disposition called it "planned as the next change" to explain where the remaining cost goes, not to defer a fix out of this PR. Two constraints on my side shape this. A standalone issue from this fork account is picked up by automation as an unclaimed task, so issues here are opened together with their PR. Editing this description resets readiness on a fork PR and needs maintainer approval of the re-run workflows, which happened once already. If you still want an issue linked before merge, say so and I will open it with that PR when the series reaches it. |
|
[operator: bolichen97] Continuing this in #31719; thanks for the start. |
Every streamed chunk made a new transcript, and the whole transcript went through the slot-scoped deferred value, so each chunk scheduled a background render of the chat page. On a slow phone the next chunk interrupted that render every time, so the reply's text almost never updated; on desktop it cost a second page render per chunk. A chunk that only grows the last row renders at urgent priority: useSlotDeferredValue takes an optional sameShape check, and useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a regroup is still deferred, and while one is pending the hook shows the last value it rendered live, never an older frame. The window while a change of shape is pending is tracked in #31704. Builds on #23487. Closes #23486 Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
Every streamed chunk made a new transcript, and the whole transcript went through the slot-scoped deferred value, so each chunk scheduled a background render of the chat page. On a slow phone the next chunk interrupted that render every time, so the reply's text almost never updated; on desktop it cost a second page render per chunk. A chunk that only grows the last row renders at urgent priority: useSlotDeferredValue takes an optional sameShape check, and useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a regroup is still deferred, and while one is pending the hook shows the last value it rendered live, never an older frame. The window while a change of shape is pending is tracked in #31704. Builds on #23487. Closes #23486 Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
Every streamed chunk made a new transcript, and the whole transcript went through the slot-scoped deferred value, so each chunk scheduled a background render of the chat page. On a slow phone the next chunk interrupted that render every time, so the reply's text almost never updated; on desktop it cost a second page render per chunk. A chunk that only grows the last row renders at urgent priority: useSlotDeferredValue takes an optional sameShape check, and useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a regroup is still deferred, and while one is pending the hook shows the last value it rendered live, never an older frame. The window while a change of shape is pending is tracked in #31704. Builds on #23487. Closes #23486 Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
Every streamed chunk made a new transcript, and the whole transcript went through the slot-scoped deferred value, so each chunk scheduled a background render of the chat page. On a slow phone the next chunk interrupted that render every time, so the reply's text almost never updated; on desktop it cost a second page render per chunk. A chunk that only grows the last row renders at urgent priority: useSlotDeferredValue takes an optional sameShape check, and useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a regroup is still deferred, and while one is pending the hook shows the last value it rendered live, never an older frame. The window while a change of shape is pending is tracked in #31704. Builds on #23487. Closes #23486 Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com> (cherry picked from commit 9d37818)
Problem / Motivation
Goal: A streaming reply's text keeps up on slow devices, and each chunk costs one ChatPage render instead of two.
The transcript is passed through a deferred value, so every streamed chunk scheduled a background re-render of ChatPage. On a slow phone the next chunk always interrupted that background pass. In a 30 s stream at 390x844 with 4x CPU throttle, the streaming bubble got new content 2 times. On desktop the background pass did commit, which made it a second ChatPage render for every chunk.
Why it matters
Watching the reply arrive is the main thing a user does while a turn runs. On a phone the text froze for most of the reply even though the page was busy re-rendering. On desktop, 245 of 1048 ChatPage renders per 30 s were this second pass.
Not a goal
useSmoothStream).chat.messagessubscription, or the virtualizer's height-version renders.What changed (motivation → approach → change)
A chunk that only grows the reply touches one row, so rendering it at urgent priority is cheap. The deferral exists for expensive changes such as a landed page of 1500 messages.
useSlotDeferredValuetakes an optionalsameShape(deferredInput, next)check. When it holds, the hook keeps handinguseDeferredValueits earlier input, so React schedules no background pass, and it returns the current value in this commit. While a tail-only run is in progress the deferral holds the frame the run started from, so when a real change then defers, the hook returns the last value it rendered live instead of that stale frame. The reply never rolls back to its first chunk while the background pass catches up. Both refs the hook keeps are written in a layout effect, so a background render that React interrupts and discards cannot leave its frame behind. The session-visit key is unchanged.useTranscriptRows(pages/chat/page/transcriptRows.ts) passesgrewOnlyAtTail. It holds when the message and display-item counts match, the last message keeps itstsandrole, and every earlier message and display item is the same object. The turn grouper keeps settled turns by reference, so a mid-stream chunk passes and a new row, a role change at turn end, an edited earlier message or a landed page fails.Known limitation: while a background pass for a change of shape (a new row, a turn end) is pending, tail chunks that arrive in that window keep showing the last live value until the pass commits, as every chunk did before this change. The benchmark stream has no tool rows, so that window is not measured here.
Backwards compatibility
Compatible:
sameShapeis optional, anduseTranscriptRowsis the hook's only caller. Every change that is not tail-only is deferred as before.Removes nothing: no export, prop or deferral path is removed.
Tests
useSlotDeferredValue.test.ts,sameShape. These use the test's controllableuseDeferredValue, frozen the way React holds it under urgent churn:useDeferredValueone input for the whole run. This fails on main, which returns the frozen first value.transcriptRows.test.tscoversgrewOnlyAtTail: a replaced tail passes; an added row, a different last row, a role change, an edited earlier message or display item, and an empty transcript fail.Manual verification
A gateway-free render benchmark replays the same frames each run: a 30 s prose stream into the open chat, 3 runs, median, React profiling build.
On mobile, ChatPage renders rise because the text now reaches the page: row heights change with it, and the virtualizer's height-version renders went from 114 to 208.
Why no screenshot: the transcript draws the same rows; the streamed text reaches them sooner, which a still image cannot show.
Related Issues
Fixes #23486
Pattern harvest
Not generalizable: the deferral had one caller, and the fix is a shape check specific to how a streaming transcript grows.
Checklist
feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)