Repository navigation
perf(cli): scan only the tag seam when appending stream text - #1548
Conversation
Every text delta concatenated the whole accumulated block with the new text and ran two substring scans over the result, so one reply cost O(n²) in its own length: 8,000 deltas (470 KB) spent ~430 ms inside this loop, and the cost kept growing for long output. A think tag can only be *completed* at the seam between the block's tail and the incoming delta — a tag already inside the block was split out when it arrived. Check a 7-character seam first (the widest partial tag, `</think`) and fall back to the full-block scan only when a `<` could still be mid-tag. The blocks produced are unchanged: 432 corpus scenarios (18 texts × 12 chunkings, root and agent paths) are byte-identical before and after, and the same suite passes on both trees. 8,000 deltas: 434 ms → 1.1 ms, and now linear in stream length. 🤖 Generated with Codebuff Co-Authored-By: Codebuff <noreply@codebuff.com>
|
Nice catch. The old code re-concatenated and re-scanned the whole accumulated text block on every delta, which is quadratic over the life of a reply — your benchmark numbers line up with what you'd expect from that pattern. The seam-based fix is sound: since What makes this easy to trust rather than just plausible:
This is in-scope ( |
What
The text streaming path in
cli/src/utils/block-operations.tsconcatenated the whole accumulated block with each incoming delta and then ran twoString.includesscans over the result:That made one reply cost O(n²) in its own length — every delta re-walked everything before it, allocating and flattening the concatenation each time.
Why it's safe to narrow the scan
A think tag can only be completed by the delta that completes it, at the seam between the block's tail and that delta. A tag already sitting inside the block was split out when it arrived. So the check now:
</think), plus the delta itself;<.<in ordinary prose (a < b) still takes the full path, so nothing is lost.Measurement (
bun, same machine, before → after)Cost is now linear in stream length instead of quadratic (
us/deltawas rising 17 → 54 as the stream grew; it now sits flat below 0.5).Memory: no leak found on this path. The batched updater's interval is already disposed when a turn ends (
use-send-message.ts), so it was not changed.Verification
6e06dde1….bun test src/utils src/hooks— 2672 pass, 0 fail (157 files).bun run typecheck— exit 0.block-operations-streaming.test.tscovers tags split inside the tag syntax at every seam position, the 7-char</thinkboundary, and long blocks keeping every character.One pre-existing quirk surfaced while writing the tests and is left untouched:
<think></think>with no content emits zero blocks, and a tag arriving after a long prefix folds the prefix into the reasoning block. Both behave identically before and after this change, hence the narrower assertions in the new test file.