Make streamed chat responses appear smoothly, like typing

[cpsievert]

Summary

Today, when a response streams into the chat, each new piece of text pops onto the screen the moment it arrives — so the reveal is only as smooth as the delivery. Pieces show up in bursts (network buffering, server pauses, tool calls), and fast providers send hundreds per second. Every arrival also makes the browser re-process the message so far — more work than it needs to do when the stream is fast.

This PR reveals streamed text at a steady pace, a few characters at a time, like typing — even when delivery is bursty. Under the hood that's two simple ideas working together: throttling (the message is re-processed at most ~20 times a second, no matter how fast pieces arrive — that's where the benchmark savings below come from) and pacing (each update reveals just enough new text to keep the reveal smooth). One behavior change to be aware of: the pulsing dot no longer trails along with incoming text. It now appears only when nothing has arrived for 1.5 seconds or no text has arrived at all, and it disappears the moment text starts moving again.

Demo

Side-by-side of the same synthetic token stream (3–8 tokens every 50–150ms, a realistic LLM cadence — the anthropic profile in the app below), before (main) and after (this PR):

smooth-demo.mp4

Try it

R app with a synthetic token stream (no API key needed)

library(shiny)
library(bslib)
library(coro)
library(shinychat)

SAMPLE <- paste0(
  "Here's a **quick** overview of how the `summarise()` function works in practice.\n\n",
  "1. First, group the data with `group_by()` so each species is handled separately.\n",
  "2. Then compute summary statistics like the mean and standard deviation.\n\n",
  "```r\n",
  "penguins |>\n",
  "  group_by(species) |>\n",
  "  summarise(mean_mass = mean(body_mass_g, na.rm = TRUE))\n",
  "```\n\n",
  "The result is a tidy tibble with one row per species. Gentoo penguins are ",
  "noticeably heavier on average, while Adélie and Chinstrap are similar. \U0001F427 You ",
  "might also want to look at _sex_ differences within each species before ",
  "drawing conclusions.\n"
)
SAMPLE <- paste0(SAMPLE, SAMPLE)

# Split before each space/newline (delimiter stays with the following word),
# and split words > 7 chars into a 5-char piece + remainder, mimicking
# subword tokenization.
tokenize <- function(text) {
  parts <- strsplit(text, "(?=[ \n])", perl = TRUE)[[1]]
  toks <- character(0)
  for (w in parts) {
    if (nchar(w) > 7) {
      toks <- c(toks, substr(w, 1, 5), substr(w, 6, nchar(w)))
    } else if (nzchar(w)) {
      toks <- c(toks, w)
    }
  }
  toks
}

profile_params <- function(profile) {
  switch(profile,
    anthropic = list(n = sample(3:8, 1), gap = 0.05 + runif(1) * 0.1),
    bursty = list(n = sample(15:40, 1), gap = 0.3 + runif(1) * 0.5),
    stall = list(n = sample(3:8, 1), gap = 0.05 + runif(1) * 0.1)
  )
}
STALL_AFTER_CHUNK <- 20
STALL_SECONDS <- 6.0

ui <- page_fillable(
  layout_columns(
    selectInput("profile", NULL, c("anthropic", "bursty", "stall")),
    actionButton("go", "Stream sample"),
    col_widths = c(3, 3),
    fill = FALSE
  ),
  chat_ui("chat")
)

server <- function(input, output, session) {
  fake_stream <- function(profile) {
    set.seed(7319)  # deterministic chunk sizes and timing
    toks <- tokenize(SAMPLE)

    async_generator(function() {
      await(async_sleep(0.3))
      i <- 1
      chunk <- 0
      while (i <= length(toks)) {
        prof <- profile_params(profile)
        n <- prof$n
        gap <- prof$gap
        yield(paste(toks[i:min(i + n - 1, length(toks))], collapse = ""))
        i <- i + n
        chunk <- chunk + 1
        if (profile == "stall" && chunk == STALL_AFTER_CHUNK) {
          gap <- STALL_SECONDS
        }
        await(async_sleep(gap))
      }
    })()
  }

  observeEvent(input$go, {
    chat_append("chat", fake_stream(input$profile))
  })
}

shinyApp(ui, server)

Click Stream sample. The default anthropic profile delivers 3–8 tokens every 50–150ms — a realistic LLM cadence, so the comparison against main (whole-chunk pops) is apples-to-apples. The stall profile pauses mid-stream for 6 seconds, which is when the pulsing "waiting" dot should appear; it should disappear the moment text starts moving again.

Benchmarks

We recorded one real streamed response from each of five providers, preserving its exact chunk sizes and timing, then replayed each recording in headless Chrome and measured how much main-thread CPU time the page burns while the response streams. Each number below is this PR's CPU time divided by today's, for the same recording: 0.38× is 62% less work; 1.58× is 58% more. The right column reruns the recordings with Chrome's CPU throttled to quarter speed, standing in for a weaker machine.

Provider (pieces/sec)	CPU vs today	CPU vs today, 4× throttled
OpenAI (86/sec)	0.49×	0.78×
DeepSeek (252/sec)	0.38×	0.68×
Anthropic (22/sec)	0.95×	0.94×
Groq (884/sec)	0.76×	0.93×
Google (8/sec)	1.58×	1.87×

Wall-clock time to the finished response is unchanged; the only addition is ~0.4s of "finishing typing" at the end.

The savings track delivery rate: the more pieces per second, the more re-processing the throttle eliminates. Google is the one regression — at 8 pieces per second there is little to throttle, and pacing spreads the same text across more re-processing passes, for about 1.6–1.9× today's CPU. We're accepting that cost for the smooth reveal; the pure-throttling variant explored earlier on this branch keeps Google at roughly today's cost if we ever want to switch.

Design decisions

• Pacing, not just throttling. A pure throttle (emit everything buffered on each tick) is cheaper on CPU — especially for Google's large, infrequent pieces — but keeps the jerky reveal. We chose the typing effect and accept the Google regression in the table.
• Reveal speed follows the backlog, not the provider. Each update reveals ~2% of whatever text is waiting, with a 30 chars/s floor, so a fast provider doesn't read faster than a slow one — bursts just catch up quicker. drainRate / minCharsPerSecond in StreamSmoother tune this.
• Responses finish "typing" (~0.4s) instead of popping the tail in at once. While that tail drains, server messages that assume the response is final (history updates, sibling navigation) are held and replayed in order; a new message or pressing Stop cuts the drain short, and Stop doesn't mark the already-finished reply as cancelled.
• The dot means "waiting", not "streaming". It appears only after 1.5s with no visible change (STREAM_IDLE_MS) or before any text arrives, and skips the pulse under prefers-reduced-motion. The threshold is a judgement call.