Skip to content

perf(cli): scan only the tag seam when appending stream text - #1548

Merged
obadakhalili merged 1 commit into
CodebuffAI:mainfrom
Salhi-zh:perf/cli-stream-seam
Oct 23, 2026
Merged

obadakhalili merged 1 commit into
CodebuffAI:mainfrom
Salhi-zh:perf/cli-stream-seam

Conversation

@Salhi-zh

Copy link
Copy Markdown
Contributor

What

The text streaming path in cli/src/utils/block-operations.ts concatenated the whole accumulated block with each incoming delta and then ran two String.includes scans over the result:

const combinedText = lastBlock.content + text
if (combinedText.includes(THINK_OPEN_TAG) || combinedText.includes(THINK_CLOSE_TAG)) { ... }

That made one reply cost O(n²) in its own length — every delta re-walked everything before it, allocating and flattening the concatenation each time.

Why it's safe to narrow the scan

A think tag can only be completed by the delta that completes it, at the seam between the block's tail and that delta. A tag already sitting inside the block was split out when it arrived. So the check now:

  1. looks at the last 7 characters of the block (the widest partial tag, </think), plus the delta itself;
  2. falls back to the original full-block scan whenever either contains <.

< in ordinary prose (a < b) still takes the full path, so nothing is lost.

Measurement (bun, same machine, before → after)

Scenario Before After
8,000 deltas, 470 KB, plain text 434 ms 1.1 ms
4,000 deltas, 234 KB 117 ms 0.9 ms
2,000 deltas, 117 KB 34 ms 0.7 ms
1,000 deltas, 59 KB 21 ms 0.4 ms
8 subagents × 600 deltas 38 ms 5.5 ms

Cost is now linear in stream length instead of quadratic (us/delta was rising 17 → 54 as the stream grew; it now sits flat below 0.5).

Memory: no leak found on this path. The batched updater's interval is already disposed when a turn ends (use-send-message.ts), so it was not changed.

Verification

  • Behavioural equivalence, not just tests: 432 corpus scenarios (18 texts × 12 chunk sizes, root and agent paths) dumped from both implementations — byte-identical output, sha256 6e06dde1….
  • bun test src/utils src/hooks — 2672 pass, 0 fail (157 files).
  • bun run typecheck — exit 0.
  • New block-operations-streaming.test.ts covers tags split inside the tag syntax at every seam position, the 7-char </think boundary, and long blocks keeping every character.

One pre-existing quirk surfaced while writing the tests and is left untouched: <think></think> with no content emits zero blocks, and a tag arriving after a long prefix folds the prefix into the reasoning block. Both behave identically before and after this change, hence the narrower assertions in the new test file.

Every text delta concatenated the whole accumulated block with the new
text and ran two substring scans over the result, so one reply cost
O(n²) in its own length: 8,000 deltas (470 KB) spent ~430 ms inside
this loop, and the cost kept growing for long output.

A think tag can only be *completed* at the seam between the block's
tail and the incoming delta — a tag already inside the block was split
out when it arrived. Check a 7-character seam first (the widest partial
tag, `</think`) and fall back to the full-block scan only when a `<`
could still be mid-tag.

The blocks produced are unchanged: 432 corpus scenarios (18 texts ×
12 chunkings, root and agent paths) are byte-identical before and
after, and the same suite passes on both trees.

8,000 deltas: 434 ms → 1.1 ms, and now linear in stream length.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <noreply@codebuff.com>
@codebuff-team

Copy link
Copy Markdown
Contributor

Nice catch. The old code re-concatenated and re-scanned the whole accumulated text block on every delta, which is quadratic over the life of a reply — your benchmark numbers line up with what you'd expect from that pattern.

The seam-based fix is sound: since lastBlock.textType === 'text' implies any earlier tag content has already been parsed out of this block, the only way a new match can form is at the boundary between the block's tail and the incoming delta, and the widest partial match (</think, missing only >) is 7 characters — exactly what seamLength = Math.max(THINK_OPEN_TAG.length, THINK_CLOSE_TAG.length) - 1 computes. The text.includes('<') fallback correctly catches tags that arrive entirely within a single delta, and falling back to the full scan whenever either side has a < preserves correctness for ordinary prose like a < b.

What makes this easy to trust rather than just plausible:

  • The new block-operations-streaming.test.ts exercises the seam boundary explicitly (the </think split case), chunked long streams, and the ordinary-<-stays-as-text case.
  • You ran a differential equivalence check (432 corpus scenarios, byte-identical hashes) against the old implementation rather than just asserting a few examples, which is the right level of rigor for a perf change that must not alter output.
  • The one pre-existing quirk you found (<think></think> emitting zero blocks) is correctly left alone and documented rather than silently 'fixed' as a drive-by.

This is in-scope (cli/src/utils), touches no forbidden paths, and is a clean, minimal diff for the problem it solves. Worth porting.

@codebuff-team codebuff-team added bot:triaged Classified by the community triage bot pr:port-candidate Worth porting into the private source tree labels Oct 23, 2026
@obadakhalili
obadakhalili merged commit 86858d0 into CodebuffAI:main Oct 23, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bot:triaged Classified by the community triage bot pr:port-candidate Worth porting into the private source tree

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants