You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
fix(shiki): tokenize an unchanged fence once while a document streams - #616
The Shiki plugin now keeps each fence's tokens by its raw node, the same way pierre-highlights does since #612, so a fence the streaming parser hands back unchanged is tokenized once. Output nodes are still rebuilt on every update. Blocks with transformers go through codeToHast as before.
Why
While a document streams, the parser reuses its previous, unhighlighted nodes, and the plugin highlighted all of them again on every update. Streaming 20 short TypeScript fences 64 characters at a time called codeToTokens 182 times, once for each fence in each update. With this change it's 75, most of them for the fence that is still arriving.
I found it while profiling a streamed chat in comark-ui. The new test fails on main (2 calls instead of 1) and passes here. pnpm --filter comark run test, pnpm typecheck, oxlint and oxfmt on the changed files all pass.
Summary by CodeRabbit
Performance
Improved code highlighting during streaming updates by reusing results for unchanged code blocks, reducing repeated processing.
Bug Fixes
Updated code blocks are reprocessed when their content or language changes, and edits to previously rendered output no longer affect subsequent renders.
The CLI's render reads .github/pr-lens.yml and applies your renames, exclusions and lane pins at draw time
🪧 More tips
Run npx skills add coldteadotai/pr-lens, then tell your coding agent: "Diagram the change you just made with PR Lens and attach it to the pull request."
Run npx @coldtea/pr-lens-cli analyze --base origin/main on a branch, then npx @coldtea/pr-lens-cli render .pr-lens/graph.json. Same lenses, your own model key, before the pull request exists
Click the link under each diagram to open it on a canvas you can zoom, pan and step through
The diagrams are links. Click one to open it on the canvas, then press W or click play to walk through the change
Open a diagram on the canvas, then press W or click play to walk through the change one step at a time
Set github.comment.collapsed: true in .github/pr-lens.yml to fold the comment behind one View architecture… row. Drawing still runs as before
Set github.draw: on-demand in .github/pr-lens.yml and PR Lens stops drawing pull requests on its own. Comment @pr-lens draw on a pull request when you want that one drawn
Add .github/workflows/pr-lens.yml with coldteadotai/pr-lens/packages/action@v0 and your model provider's key as its api-key to run PR Lens from your own CI. Any /chat/completions endpoint works
After a push, tick Redraw to draw the latest commit
Switch GitHub to dark mode and the diagrams follow. The moving dots are this pull request's data in motion
Set github.description: false in .github/pr-lens.yml to stop PR Lens adding buttons to your PR descriptions
Thanks for using PR Lens! It's free for OSS. If it helped, a star helps other teams find it.
Shiki code-block highlighting now accepts a token cache and reuses cached tokenization results when the code and language match. Each plugin instance maintains its own cache. A streaming test checks reuse across parses and distinct output nodes.
Changes: Shiki token caching
Layer / File(s)
Summary
Cache contract and token reuse packages/comark/src/internal/shiki.ts
highlightCodeBlocks accepts an optional cache. The tokenization path reuses cached results when both the code and language match; otherwise, it tokenizes and updates the cache.
Plugin integration and streaming test packages/comark/src/internal/shiki.ts, packages/comark/test/plugins/shiki-streaming.test.ts
Each plugin instance passes its cache to post-processing. The streaming test checks that unchanged code is tokenized once and that successive parses return distinct nodes.
Streaming Shiki fences are now tokenized once per unchanged block. A stale-color edge case exists only if a caller mutates theme options in place mid-stream, which is unlikely. The change is low risk to merge.
Pre-merge checks: 4
Check name
Status
Explanation
Description Check
Check skipped - CodeRabbit’s high-level summary is enabled.
Title check
The title says the change makes an unchanged fence tokenize once while a document streams. The raw summary confirms a per-fence token cache keyed by the raw code-block node, reused only when code and …
Linked Issues check
Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check
Check skipped because no linked issues were found for this pull request.
Full details: Title check
Explanation
The title says the change makes an unchanged fence tokenize once while a document streams. The raw summary confirms a per-fence token cache keyed by the raw code-block node, reused only when code and language match, plus a streaming test for that behavior. The PR objectives describe the same goal. The title is specific and matches the main change. The 'fix(shiki)' scope and conventional-commit form are acceptable.
Finishing Touches
Generate unit tests (beta)
Create a new PR
Fix failing CI checks
Create a new PR
Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
System calls cannot be simulated consistently and are left out of the measure. The reported value understates the real cost. Switch to the Walltime instrument to measure them.
The reason will be displayed to describe this comment to others. Learn more.
Additional comments: 1
packages/comark/src/internal/shiki.ts-407-407: 1
407-407: Minor · Functional Correctness · Quick win
Unverified finding
Verification ran but could not confirm this finding. It is shown for review, not as a verified issue.
Invalidate cached tokens when the highlighting configuration changes.
If a caller replaces the retained options.themes between streaming updates, this condition still accepts the previous theme's result. getHighlighter and themeOptions use the new configuration, but the rebuilt block retains the previous token colors and themeName. Previously, each update tokenized with the current themes. Shiki derives both values from the selected themes. (raw.githubusercontent.com)
Include the effective light/dark theme selection and highlighter identity in cache validity checks. Keep cache hits when that context is unchanged.
Review info
Run configuration
Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: defaults
Review profile: CHILL
Plan: Advanced
Run ID: aefeed1f-1f5c-4a61-ba2c-1d58e64aed7e
Commits
Reviewing files that changed from the base of the PR and between a320fbf and c92cafb.
Included review availability: This review used your included allowance. Your plan provides up to 4 included reviews per hour; 3 remain after this review.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The Shiki plugin now keeps each fence's tokens by its raw node, the same way
pierre-highlightsdoes since #612, so a fence the streaming parser hands back unchanged is tokenized once. Output nodes are still rebuilt on every update. Blocks withtransformersgo throughcodeToHastas before.Why
While a document streams, the parser reuses its previous, unhighlighted nodes, and the plugin highlighted all of them again on every update. Streaming 20 short TypeScript fences 64 characters at a time called
codeToTokens182 times, once for each fence in each update. With this change it's 75, most of them for the fence that is still arriving.I found it while profiling a streamed chat in comark-ui. The new test fails on
main(2 calls instead of 1) and passes here.pnpm --filter comark run test,pnpm typecheck, oxlint and oxfmt on the changed files all pass.Summary by CodeRabbit