Skip to content

fix(chat): keep a streaming reply's text live on slow devices - #31719

Merged
KiroCrewCaptain merged 1 commit into
mainfrom
fix/23486-stream-tail-live
Nov 7, 2026
Merged

KiroCrewCaptain merged 1 commit into
mainfrom
fix/23486-stream-tail-live

Conversation

@bolichen97

Copy link
Copy Markdown
Contributor

Problem / Motivation

Goal: A streaming reply's text keeps up on slow devices, and each streamed chunk costs one chat page render instead of two.

On a slow phone a streaming reply's text barely moves. In a 30 s prose stream at 390x844 with 4x CPU throttle, the streaming bubble received new content 2 times while the page kept re-rendering. On desktop each chunk cost a second ChatPage render: 245 of 1048 ChatPage renders in the same stream at 1440x900.

Why it matters

Watching the reply arrive is the main thing a user does while a turn runs. On a phone the text froze for most of the reply even though the page was busy. On desktop about a quarter of ChatPage's renders during a stream were that second pass.

Not a goal

  • Removing the deferral for a landed page of history, a new row or a regrouped turn. Those still render in the background.
  • The window while one of those is pending, where tail chunks wait for it to commit. That is A streaming reply pauses on slow devices while a new row is still rendering #31704 (a per-row content subscription).
  • The streaming bubble's own reveal ticks (useSmoothStream), ChatPage's per-chunk render from its chat.messages subscription, and the virtualizer's height-version renders.

What changed (motivation → approach → change)

The reply's text now updates in the same commit as each chunk. The whole transcript went through one deferred value, so every chunk asked React for a background render of the chat page. On a slow device the next chunk interrupted that render every time, so the bubble kept its old text; on desktop the render did finish, which made it a second page render per chunk.

A chunk that only grows the reply's last row is cheap to draw, so it skips the deferral. Anything else stays deferred: a page of history landing, a new row, a regrouped turn. While one of those is pending, the transcript keeps showing the last text it drew live, never an older frame, so the reply cannot roll back to its first chunk.

Where Change
website/src/hooks/useSlotDeferredValue.ts optional sameShape(deferredInput, next): when it holds, the deferral keeps its earlier input and the current value renders now. Its two refs are written in a layout effect, so a render React throws away leaves nothing behind
pages/chat/page/transcriptRows.ts useTranscriptRows passes grewOnlyAtTail: same message and row counts, the last message keeps its ts and role, every earlier message and row is the same object

Known limitation: while a change of shape is pending (the streaming row first appearing, a tool row mid-reply), later tail chunks show the last live text until that background render commits, as every chunk did before. #31704 tracks it.

Backwards compatibility

Compatible: sameShape is optional and useTranscriptRows is the hook's only caller; without it the hook returns exactly what it returned before.

Removes nothing: no export, prop or deferral path is removed.

Tests

  • useSlotDeferredValue.test.ts, sameShape block, with the test's controllable useDeferredValue held back the way React holds it under urgent churn:
    • in-place changes each render the current value, and the deferral gets one input for the whole run (fails on main, which shows the frozen first value);
    • tail-only changes followed by a change of shape keep returning the last live value, not the stream-start frame, and a tail chunk on top of the pending change hands the deferral no new input (fails on main);
    • a render React throws away before it commits (a sibling suspends) changes nothing for the next chunk (fails on main, and fails if the live ref is written during render).
  • transcriptRows.test.ts, grewOnlyAtTail: holds for an in-place tail replacement and for a streaming flush through the real turn grouper and running state of a two-turn transcript; fails for an added row, a different or role-changed last row, an edited earlier message or row, and an empty transcript.
  • Determinism: both files ran 5 times in shuffled order (seeds 11, 23, 37, 41, 59), 15 of 15 passing each run. No timers or clock in either file.
  • Red on main: the new hook tests run against origin/main's useSlotDeferredValue.ts fail 3 of 9.

Proof: not provable -- frontend test (prove.py runs Python only)

Manual verification

The render benchmark replays the same frames each run: a 30 s prose stream into the open chat, gateway-free, 3 runs, median, React profiling build. These numbers were measured on #23487's head; this PR keeps its behaviour (the hook drops one field that never changed the result).

Counter Desktop 1440x900 main this change Mobile 390x844, 4x CPU main this change
Streaming bubble renders with new content 244 299 2 206
ChatPage renders 1048 802 589 675
ChatPage renders from the deferred pass 245 0 4 0

On mobile, ChatPage renders rise because the text reaches the page: row heights change with it, and the virtualizer's height-version renders go up. On this head npm --prefix website run bench:render -- --fail-on dispatches,selectorCalls passes every scenario.

Screenshots / video

Why no screenshot: the transcript draws the same rows; the streamed text reaches them sooner, which a still image cannot show.

Related Issues

Closes #23486

Builds on #23487 by @Pearcekieser.

Follow-up: #31704

Sibling sweep

Swept: git grep -n "useDeferredValue" -- website/src
Sibling: website/src/components/PromptLengthNotice.tsx:44 -- not affected -- it defers the composer's own typed text, which is the intended typing-path deferral, not a streaming transcript

Pattern harvest

Not generalizable: the transcript deferral has one caller, and the shape check is specific to how a streaming transcript grows.

Checklist

  • At most two commits (one is the norm), with a Conventional Commits title (feat|fix|docs|style|refactor|perf|test|chore|ci|build|revert: ...)
  • Existing tests pass and new tests added for new functionality
  • Self-review completed; code follows project style guidelines
  • Documentation updated (if applicable)
  • No secrets, credentials, or internal references in the diff

@bolichen97
bolichen97 requested review from a team and KiroCrewCaptain as code owners November 6, 2026 19:38
@bolichen97
bolichen97 requested a review from Zedmor November 6, 2026 19:38
@github-actions github-actions Bot added the readiness: checking Automated validation is still running label Nov 6, 2026
@bolichen97
bolichen97 enabled auto-merge (squash) November 6, 2026 19:43
@github-actions github-actions Bot added readiness: action required A blocking check or review needs attention and removed readiness: checking Automated validation is still running labels Nov 6, 2026
@bolichen97
bolichen97 force-pushed the fix/23486-stream-tail-live branch from 25f767e to 3804469 Compare November 7, 2026 01:52
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in website/src/hooks/ and website/src/pages/chat/page/ChatTranscriptView.tsx since this head went green (8dbdf04 -> 678769e); CI re-running.

The rebase also picks up #31625 and #31803, which fix the two backend tests that were red on main at the old base (test_the_package_stays_within_its_line_budget, test_only_the_allowed_workflows_use_pull_request_target). The diff is unchanged; the related frontend tests, tsc and eslint pass on the new head 3804469.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running readiness: action required A blocking check or review needs attention and removed readiness: action required A blocking check or review needs attention readiness: checking Automated validation is still running labels Nov 7, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
@bolichen97
bolichen97 force-pushed the fix/23486-stream-tail-live branch from 3804469 to 78f1d0e Compare November 7, 2026 05:24
@bolichen97

Copy link
Copy Markdown
Contributor Author

Rebased: main moved in website/src/hooks/ and website/src/pages/chat/page/ since this head went green (678769e -> 464b547); CI re-running.

The diff is unchanged. The related frontend tests, tsc and eslint pass on the new head 78f1d0e.

The Render Benchmark red on the old head (s1x5.renders: 10928 against budget 8150 (limit 9372) OVER) is not from this change: the same scenario reaches about 10900 renders in some runs on main with this PR's two source files reverted, so it is recorded in #31963.

[operator: bolichen97]

@github-actions github-actions Bot added readiness: checking Automated validation is still running and removed readiness: action required A blocking check or review needs attention labels Nov 7, 2026
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

UX Review (Opus 5.5) — ✅ PASS

UX-level review of 78f1d0ed51eb45414295b94dd85d4006aeeb469f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

UX-Verdict: PASS

Nothing to check. The diff adds no control, label or state; it only changes when the streamed text renders.

[UX-REVIEWED] 78f1d0e

Description read

[DESCRIPTION-READ] 80e7bfaa4fe5da4da116e04a4409a815c25ec3b33a8373430ad8304423fbd0c5

That sha256 names everything this verdict read: the PR description, plus the 3 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Design Review (Opus 5.5) — ✅ PASS

Design-level review of 78f1d0ed51eb45414295b94dd85d4006aeeb469f — updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

Design-Verdict: PASS

Nothing blocks: the root cause is that every chunk went through the deferral, and the fix takes in-place tail growth off that path while new rows, landed pages and regroups stay deferred, with no rollback to an older frame.

Notes

[DESIGN-REVIEWED] 78f1d0e

Description read

[DESCRIPTION-READ] e7913957c986524683f3ea066f6cfa2a7592a6a4e4102acfefef1c141806cbca

That sha256 names everything this verdict read: the PR description, plus the 1 evidence file(s) the workflow collected from it and handed this review. It covers that evidence because the media strip replaces every attachment URL with the same placeholder, so an edit swapping one attachment for another would leave the description's captured text identical while what this review judged changed. Recompute it with .github/scripts/pr-description-capture.sh over the same inputs; if it differs, the description or that evidence changed after this verdict was formed and any finding drawn from either is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

First Principles Review (Opus 5.5) — ✅ PASS

Premise-level review of 78f1d0ed51eb45414295b94dd85d4006aeeb469f — why this exists and whether the shipped surface is the smallest honest version. Updated in place on each push. A BLOCK verdict blocks PR readiness; PASS does not.

First-Principles-Verdict: PASS

The fix comes with an issue (#23486) and hook tests that fail on main. Its one wider piece, the sameShape parameter, has a single caller. That is a note, not a blocker.

Notes

  • Item 3 — one consumer, generalized: sameShape?: (deferredInput: T, next: T) => boolean. It is an optional predicate on a hook that has exactly 1 non-test caller, transcriptRows.ts:93, which always passes grewOnlyAtTail (grepped useSlotDeferredValue|grewOnlyAtTail in website/src). Subtraction: make the parameter required, or narrow useSlotDeferredValue to the transcript shape, so there is no "absent" branch for a caller that never exists.

What this change ships

Inventory (4 items) — 3 justified

Intent: on slow devices, a streaming reply's text keeps up as it arrives, without a second chat page render per chunk. This is a FIX.

  1. A streamed chunk that only grows the last row now shows in the same commit, with no background page render — justified
  2. While a new row or regroup is still waiting to render, the transcript shows the last live text instead of rolling back to the first chunk — justified
  3. useSlotDeferredValue gains an optional sameShape predicate parameter — one consumer, generalized: optional predicate, one caller
  4. New exported grewOnlyAtTail(prev, next) check in transcriptRows.ts — justified

[FIRST-PRINCIPLES-REVIEWED] 78f1d0e

Description read

[DESCRIPTION-READ] cce350973344d5f4ff866134421f05066394ec7a577aab24e53cc1534cb3b5e0

That sha256 names the PR description this verdict read. Recompute it with .github/scripts/pr-description-capture.sh; if it differs, the description changed after this verdict was formed and any finding drawn from the description is unproven.

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

GPT 6.1 Review — ✅ no blocking findings

GPT 6.1 completed its review of 78f1d0ed51eb45414295b94dd85d4006aeeb469f and found no blocking issues.

This comment is updated in place on each push.

Review details

No findings.
[GPT-REVIEWED] 78f1d0e

False positive or not applicable? A repository writer can comment:
/ai-review override gpt 78f1d0ed51eb45414295b94dd85d4006aeeb469f false-positive: <reason, 20 to 500 characters>

@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

Opus 5.5 Review — ✅ no blocking findings

Reviewed 78f1d0ed51eb45414295b94dd85d4006aeeb469f — this comment is updated in place on each push.

Review details

No findings.

[OPUS-REVIEWED] 78f1d0e

Verdict parsed from the review's SHA-scoped output markers for commit 78f1d0ed51eb45414295b94dd85d4006aeeb469f.

False positive or not applicable? A repository writer can comment:
/ai-review override fable 78f1d0ed51eb45414295b94dd85d4006aeeb469f false-positive: <reason, 20 to 500 characters>

@github-actions github-actions Bot added readiness: passed Eligible automated validation passed for the current revision and removed readiness: checking Automated validation is still running labels Nov 7, 2026

@KiroCrewCaptain KiroCrewCaptain left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved for the on-call. Review lanes pass on 78f1d0e, no conflict, no open change request.

@KiroCrewCaptain
KiroCrewCaptain merged commit 9d37818 into main Nov 7, 2026
97 checks passed
@KiroCrewCaptain
KiroCrewCaptain deleted the fix/23486-stream-tail-live branch November 7, 2026 06:06
buluoray pushed a commit that referenced this pull request Nov 8, 2026
Every streamed chunk made a new transcript, and the whole transcript
went through the slot-scoped deferred value, so each chunk scheduled a
background render of the chat page. On a slow phone the next chunk
interrupted that render every time, so the reply's text almost never
updated; on desktop it cost a second page render per chunk.

A chunk that only grows the last row renders at urgent priority:
useSlotDeferredValue takes an optional sameShape check, and
useTranscriptRows passes grewOnlyAtTail. A landed page, a new row or a
regroup is still deferred, and while one is pending the hook shows the
last value it rendered live, never an older frame. The window while a
change of shape is pending is tracked in #31704.

Builds on #23487.

Closes #23486

Co-authored-by: Pearcekieser <5055971+Pearcekieser@users.noreply.github.com>
(cherry picked from commit 9d37818)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

readiness: passed Eligible automated validation passed for the current revision

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A streaming reply barely updates on slow devices because every chunk is deferred

2 participants