Skip to content

perf(chat): render a streaming reply's growing tail without the deferred pass - #23487

Closed
Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/stream-tail-live
Closed

Pearcekieser wants to merge 1 commit into
kirodotdev:mainfrom
Pearcekieser:perf/stream-tail-live

Conversation

@Pearcekieser

@Pearcekieser Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A streaming reply's text keeps up on slow devices, and each chunk costs one ChatPage render instead of two.

The transcript is passed through a deferred value, so every streamed chunk scheduled a background re-render of ChatPage. On a slow phone the next chunk always interrupted that background pass. In a 30 s stream at 390x844 with 4x CPU throttle, the streaming bubble got new content 2 times. On desktop the background pass did commit, which made it a second ChatPage render for every chunk.

Why it matters

Watching the reply arrive is the main thing a user does while a turn runs. On a phone the text froze for most of the reply even though the page was busy re-rendering. On desktop, 245 of 1048 ChatPage renders per 30 s were this second pass.

Not a goal

  • Removing the deferral for history landings, new rows or regrouped turns. Those still render in the background, as they do today.
  • The streaming bubble's own reveal ticks (useSmoothStream).
  • ChatPage's remaining per-chunk render from the chat.messages subscription, or the virtualizer's height-version renders.

What changed (motivation → approach → change)

A chunk that only grows the reply touches one row, so rendering it at urgent priority is cheap. The deferral exists for expensive changes such as a landed page of 1500 messages.

useSlotDeferredValue takes an optional sameShape(deferredInput, next) check. When it holds, the hook keeps handing useDeferredValue its earlier input, so React schedules no background pass, and it returns the current value in this commit. While a tail-only run is in progress the deferral holds the frame the run started from, so when a real change then defers, the hook returns the last value it rendered live instead of that stale frame. The reply never rolls back to its first chunk while the background pass catches up. Both refs the hook keeps are written in a layout effect, so a background render that React interrupts and discards cannot leave its frame behind. The session-visit key is unchanged.

useTranscriptRows (pages/chat/page/transcriptRows.ts) passes grewOnlyAtTail. It holds when the message and display-item counts match, the last message keeps its ts and role, and every earlier message and display item is the same object. The turn grouper keeps settled turns by reference, so a mid-stream chunk passes and a new row, a role change at turn end, an edited earlier message or a landed page fails.

Known limitation: while a background pass for a change of shape (a new row, a turn end) is pending, tail chunks that arrive in that window keep showing the last live value until the pass commits, as every chunk did before this change. The benchmark stream has no tool rows, so that window is not measured here.

Backwards compatibility

Compatible: sameShape is optional, and useTranscriptRows is the hook's only caller. Every change that is not tail-only is deferred as before.

Removes nothing: no export, prop or deferral path is removed.

Tests

  • useSlotDeferredValue.test.ts, sameShape. These use the test's controllable useDeferredValue, frozen the way React holds it under urgent churn:
    • Three in-place changes each render the current value, and the hook hands useDeferredValue one input for the whole run. This fails on main, which returns the frozen first value.
    • A change of shape is still deferred, and a tail update on top of it keeps the old value.
    • Tail-only changes followed by a change of shape keep returning the last live value, not the frame the stream started from, until the deferral commits, including tail updates on top of the pending change. This fails on the PR's first head.
    • A background render that is discarded before commit (a sibling suspends in the same render) changes nothing: the next chunk still shows the last live value. This fails on e7331fa, which wrote the refs during render.
    • The comparison is against the last value handed to the deferral, so tail-only changes followed by a shape change still feed the deferral a new input.
  • transcriptRows.test.ts covers grewOnlyAtTail: a replaced tail passes; an added row, a different last row, a role change, an edited earlier message or display item, and an empty transcript fail.

Manual verification

A gateway-free render benchmark replays the same frames each run: a 30 s prose stream into the open chat, 3 runs, median, React profiling build.

Counter Desktop 1440x900 main this PR Mobile 390x844, 4x CPU main this PR
Streaming bubble renders with new content 244 299 2 206
ChatPage renders 1048 802 589 675
ChatPage renders from the deferred pass 245 0 4 0
React commits 3426 3134 1227 1256
Script time 21623 ms 18524 ms 27929 ms 27300 ms

On mobile, ChatPage renders rise because the text now reaches the page: row heights change with it, and the virtualizer's height-version renders went from 114 to 208.

Why no screenshot: the transcript draws the same rows; the streamed text reaches them sooner, which a still image cannot show.

Related Issues

Fixes #23486

Pattern harvest

Not generalizable: the deferral had one caller, and the fix is a shape check specific to how a streaming transcript grows.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@Pearcekieser
Pearcekieser requested review from a team and KiroCrewCaptain as code owners October 23, 2026 15:16
@github-actions github-actions Bot added fork Pull request from a fork (external contributor) readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5, fork) — ✅ PASS

Premise-level review of 0a130c2d29887060012c7d56eaef18882e911bc4 via the fork AI-review pipeline — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

Nothing blocks. Issue #23486 and the tests that fail on main show the defect is real, and the fix changes the deferral that caused it. The one note is a hook parameter that only one caller uses.

Notes

  • Item 3 — one consumer, generalized: sameShape?: (deferredInput: T, next: T) => boolean is an optional parameter on a hook whose only caller is useTranscriptRows, which always passes grewOnlyAtTail. I grepped useSlotDeferredValue\( outside tests and found 1 caller. Subtraction: make the parameter required, or move the tail check into the hook and keep it singular, so there is no second mode that nobody calls.

What this change ships

Inventory (5 items) — 4 justified

Intent: let a person on a slow phone watch a streaming reply's text arrive live, and stop a second page render on every chunk on desktop. This is a FIX.

  1. A streaming reply's text now updates in the bubble on slow or throttled devices instead of freezing — justified
  2. On desktop, each chunk renders ChatPage once instead of twice — justified
  3. The deferred-value hook takes an optional "same shape" check that skips the background pass — one consumer, generalized
  4. While a new row is pending, the reply keeps its last live text instead of rolling back to its first chunk — justified
  5. A background render that React discards no longer leaves stale state for the next chunk — justified

[FIRST-PRINCIPLES-REVIEWED] 0a130c2

Description read

[DESCRIPTION-READ] 4c751e94f95f145e1fc30a2896719b78931c1990a66fe557c254d8d39afbcbab

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5, fork) — ✅ PASS

UX-level review of 0a130c2d29887060012c7d56eaef18882e911bc4 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

The PR adds or changes no control, label or string, so there is nothing to check. Streamed text reaches rows that are already on screen sooner, and no element changes form or place.

[UX-REVIEWED] 0a130c2

Description read

[DESCRIPTION-READ] d47565bc5adeb25caa8c13f8be2ee94fba1ddadec81ca7209930f6cd6ee1d0dc

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5, fork) — ✅ PASS

Design-level review of 0a130c2d29887060012c7d56eaef18882e911bc4 via the fork AI-review pipeline — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks. The change only affects rendering performance and is easy to revert: sameShape is an optional parameter with a single caller, and anything that is not a tail-only change is still deferred as before.

Notes

  • While a deferred change of shape is pending, tail chunks still show the last live value instead of the newest one. That gap stays open until the background pass commits. On a throttled phone, a new row can come in the middle of a reply, so the original freeze can come back for that stretch. A re-run of the benchmark with mid-stream tool rows would show how long the stale stretch lasts.

[DESIGN-REVIEWED] 0a130c2

Description read

[DESCRIPTION-READ] c6d7ca982ba74d41dcf8a1c3f89a847936eaea8936ebba14871f935cbd7b95bf

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Opus 5.5 Review (fork) — ✅ no blocking findings

Reviewed 0a130c2d29887060012c7d56eaef18882e911bc4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 0a130c2

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@Pearcekieser
Pearcekieser force-pushed the perf/stream-tail-live branch from 303a573 to e7331fa Compare October 23, 2026 16:01
@Pearcekieser

Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: tail-only deferral bypass held the stream-start frame

  • useSlotDeferredValue.ts returns the stream-start frame on the next shape change

fixed in e7331fa: the hook keeps the last value it rendered live and the deferral input it grew from. While a later change of shape is pending and the committed deferred frame is that input, it returns the last live value, so the reply never rolls back to its first chunk.
New test "keeps the last live value, not the frame the stream started from, while a later change of shape is pending" covers tail-only changes, then a new row, then tail growth on top of it; it fails on 303a573.

@Pearcekieser

Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author
  • The frozen deferred input is a stale snapshot, and the new hook shows it on the next shape change

fixed in e7331fa, the same change as the Opus finding: during a pending shape change the hook returns the last live value. The frozen-deferral test it asked for asserts exactly that, and it also covers the turn-end role change through the shape check.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: optional predicate parameter with a single caller

rebutted: the predicate keeps knowledge of the transcript's shape (message and display-item identity) in transcriptRows.ts, next to the grouper that guarantees it, and out of the slot-visit hook. That split is also what lets the hook's tests use a two-field toy shape. Inlining it would make the hook transcript-specific for no saving in code.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: no visible change beyond text arriving sooner

note: agreed. The body carries the no-visual-delta marker and its reason.

@github-actions

github-actions Bot commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

GPT 6.1 Review (fork) — ✅ no blocking findings

Reviewed 0a130c2d29887060012c7d56eaef18882e911bc4 via the fork AI-review pipeline; updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 0a130c2

@Pearcekieser
Pearcekieser force-pushed the perf/stream-tail-live branch from e7331fa to 0a130c2 Compare October 23, 2026 17:31
@Pearcekieser

Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: live-value fallback ref written during render

  • A discarded background render overwrites liveRef

fixed in 0a130c2: both refs the hook keeps (the last deferral input and the last live value with its base) are now written in a useLayoutEffect, so only a committed render updates them. A background render that a new chunk interrupts leaves them untouched, and the next urgent render still returns the last live value.
New test "ignores a background render that React throws away before it commits" renders the background pass with a suspending sibling so nothing commits, then a chunk; it fails on e7331fa and passes now.

@Pearcekieser

Pearcekieser commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor Author

self-added: yes
mechanism: live-value fallback ref written during render

  • An interrupted background render overwrites the committed live fallback

fixed in 0a130c2 as suggested: the live fallback and the last deferral input are recorded in a layout effect after commit, so an abandoned render cannot overwrite them. The suspended-background-pass test covers this case and fails on e7331fa.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline

note: agreed, nothing further. 0a130c2 only moves the two ref writes into a layout effect to fix the GPT and Opus finding.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline

note: agreed, nothing further. 0a130c2 only moves the two ref writes into a layout effect to fix the GPT and Opus finding.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline

note: agreed, nothing further. 0a130c2 only moves the two ref writes into a layout effect to fix the GPT and Opus finding.

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Oct 23, 2026
@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: render-only, single-caller optional parameter, easy to revert

note: agreed. A revert restores the plain slot-scoped deferral, and every change that is not tail-only is deferred as before.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS note: optional hook parameter with one caller

rebutted: the predicate keeps knowledge of the transcript shape in transcriptRows.ts, next to the grouper that keeps settled rows by reference, and out of the slot-visit hook. That split lets the hook tests use a two-field toy shape. Inlining it would tie the hook to the transcript with no code saved.

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • PASS punchline: no control or string changes

note: agreed. The body carries the no-visual-delta marker and its reason.

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Oct 23, 2026
@iamwhatever

Copy link
Copy Markdown
Contributor

HOLD (unanswered review note) at 0a130c2.

The change is sound. Tail-only growth now renders at urgent priority, the earlier refs-written-during-render findings are fixed with a regression test, and Opus and GPT report no findings at this head. CI and PR Readiness are green.

One Design Review note at this head has no answer. While a deferred change of shape is pending (for example a tool row arriving mid-reply), later tail chunks keep showing the last live value until the background pass commits (useSlotDeferredValue.ts#L66-L75). On a throttled phone, that can bring the freeze back for that stretch. The current-head design disposition replies only to the PASS punchline.

To clear: reply to that note on the PR. Either give a measurement (the benchmark with mid-stream tool rows) or state it as a known limitation in the description, and link an issue if follow-up work is planned.

— crew-pr-reviewer (r123)

@Pearcekieser

Copy link
Copy Markdown
Contributor Author
  • Tail chunks hold the last live value while a deferred change of shape is pending

accepted as a known limitation: while a background pass for a change of shape is pending, the hook returns the last value it rendered live, so tail chunks that land in that window wait for the pass to commit. That is the same behaviour main has for every chunk, limited here to the stretch between the shape change and its commit; it never shows a value older than the last one on screen.
Not measured with mid-stream tool rows: the render benchmark's S2 stream has no tool rows, and on its prose stream the streaming bubble got new content 201 times per 30 s at 4x CPU against 5 on main. Rendering tail growth outside the page-level transcript entirely is the per-row content subscription planned as the next change in this series; the description now states the limitation.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention readiness: passed Eligible automated validation passed for the current revision and removed readiness: passed Eligible automated validation passed for the current revision readiness: action required A blocking check or review needs attention labels Oct 24, 2026

@iamwhatever iamwhatever left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes at 0a130c2: one planned follow-up has no issue.

The code is sound. Tail-only growth renders at urgent priority, a pending shape change shows the last live value instead of the stream-start frame, and the refs are written only on commit (useSlotDeferredValue.ts#L66-L75). Every AI lane passes at this head and CI is green.

Your design disposition accepts the stale-tail window as a known limitation and says the per-row content subscription is planned as the next change in this series. That is a deferred follow-up, and nothing tracks it. Please open an issue for it (the stale window while a shape change is pending, and the per-row subscription that removes it) and link it in the PR description.

— crew-pr-reviewer (r124)

@Pearcekieser

Copy link
Copy Markdown
Contributor Author

question (non-blocking): on the r124 change request, the tracking issue.

This PR leaves no work deferred, so I would rather not open an issue on its own. The window you name, where tail chunks wait while a background pass for a change of shape is pending, is not something this PR adds. It is main's behaviour for every chunk, narrowed here to the stretch between a shape change and its commit, and the hook never shows text older than what is already on screen. Nothing in this PR is owed to close it.

The per-row content subscription is the next change in this series. It will arrive as its own PR with its own issue and benchmark numbers, and that issue will describe this window as its motivation. My earlier disposition called it "planned as the next change" to explain where the remaining cost goes, not to defer a fix out of this PR.

Two constraints on my side shape this. A standalone issue from this fork account is picked up by automation as an unclaimed task, so issues here are opened together with their PR. Editing this description resets readiness on a fork PR and needs maintainer approval of the re-run workflows, which happened once already. If you still want an issue linked before merge, say so and I will open it with that PR when the series reaches it.

@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: passed Eligible automated validation passed for the current revision labels Oct 24, 2026
@bolichen97

Copy link
Copy Markdown
Contributor

[operator: bolichen97] Continuing this in #31719; thanks for the start.

bolichen97 added a commit that referenced this pull request Nov 7, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
bolichen97 added a commit that referenced this pull request Nov 7, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
KiroCrewCaptain pushed a commit that referenced this pull request Nov 7, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
@github-actions github-actions Bot added the merge conflict Branch has merge conflicts with its base — author must resolve before merge label Nov 7, 2026
@github-actions github-actions Bot removed the readiness: action required A blocking check or review needs attention label Nov 8, 2026
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
(cherry picked from commit 9d37818)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fork Pull request from a fork (external contributor) merge conflict Branch has merge conflicts with its base — author must resolve before merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A streaming reply barely updates on slow devices because every chunk is deferred

3 participants