Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .changeset/perky-plums-dig.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
---
'@fuzdev/fuz_code': minor
---

feat: add rust lexer
13 changes: 9 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ fuz_code is a **syntax highlighting library**:

- Runtime HTML generation with CSS classes
- Hand-written single-pass lexers (zero regex), one per language
- 8 built-in languages (TS, JS, CSS, HTML, JSON, Svelte, Markdown, Bash)
- 9 built-in languages (TS, JS, CSS, HTML, JSON, Svelte, Markdown, Shell, Rust)
- Extensible by writing a lexer (`SyntaxLang`)
- Optional Svelte component (`Code.svelte`)

Expand All @@ -69,7 +69,7 @@ src/
│ ├── syntax_styler.ts # SyntaxStyler class: registry + lex/stylize facade
│ ├── syntax_styler_global.ts # pre-configured global instance
│ ├── lexer.ts # lexer substrate: Lexer, TokenTypeRegistry, flat events, HTML render
│ ├── lexer_*.ts # hand-written lexers (json, ts, css, bash, markup, svelte, md)
│ ├── lexer_*.ts # hand-written lexers (json, ts, css, bash, markup, svelte, md, rust)
│ ├── Code.svelte # main Svelte component
│ ├── CodeHighlight.svelte # experimental CSS Highlight API
│ ├── CodeTextarea.svelte # experimental live-highlighted textarea
Expand Down Expand Up @@ -141,6 +141,11 @@ One `SyntaxLang` lexer per language, registered via `add_lang`:
info matching and any-length closers, headings, blockquotes, lists, hr)
with a per-block inline scan (emphasis, inline code, links, entities, raw
markup via the markup scanner); fences embed their languages
- `lexer_rust.ts` - Rust, registered as `rust` with `rs` as an alias: one
flat scan loop (nested block comments and raw strings resolve with counters;
attribute interiors lex inline under an `[`/`]` depth counter — no frame
machine), lifetime-vs-char `'` disambiguation, `name!` macros, doc-vs-plain
comment split, the r/b/c string prefixes

Embedded languages resolve lazily by name through the registry (markdown
fences → any language, markup `<script>`/`<style>`/`style=`/`on*=`, svelte
Expand Down Expand Up @@ -174,7 +179,7 @@ can't overflow the call stack; past the cap a region stays plain text.

## Supported languages

`ts`, `js`, `css`, `html`, `json`, `svelte`, `md`, `sh`
`ts`, `js`, `css`, `html`, `json`, `svelte`, `md`, `sh`, `rust`/`rs`

## Testing

Expand Down Expand Up @@ -296,7 +301,7 @@ Theme uses CSS variables from fuz_css:
3. **Progress discipline** - every scan loop advances position or emits+advances
4. **Nesting discipline** - no JS-call-stack recursion that scales with input
nesting. Flat single-pass loops wherever the grammar allows (css, markup,
svelte, json, and md need no stacks at all); an explicit pooled frame
svelte, json, md, and rust need no stacks at all); an explicit pooled frame
machine only where a construct's interior re-enters the same grammar
(`lexer_ts.ts` templates/generics/annotations, `lexer_bash.ts`
substitutions — use these as the template, including their inline fast
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ Highlights:
- a minimal, explicit API to generate stylized HTML — `stylize(code, lang)`
- stateless ES modules, instead of globals with side effects
- written in TypeScript, with no runtime dependencies
- eight built-in languages (see below), extensible by writing a lexer
- nine built-in languages (see below), extensible by writing a lexer

Two optional integrations:

Expand Down Expand Up @@ -112,6 +112,7 @@ Registered by default in `syntax_styler_global` — one hand-written lexer each:
- [`css`](src/lib/lexer_css.ts)
- [`json`](src/lib/lexer_json.ts) (with comments — jsonc)
- [`sh`](src/lib/lexer_bash.ts) (POSIX/bash family — also serves `bash`/`shell`)
- [`rust`](src/lib/lexer_rust.ts) (also serves `rs`)

Add a language by writing a `SyntaxLang` lexer and registering it with
`add_lang` — see the existing `lexer_*.ts` modules.
Expand Down
2 changes: 1 addition & 1 deletion src/lib/code_sample.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,6 @@ export interface CodeSample {
}

// Languages ordered from simple to complex
export const sample_langs = ['json', 'css', 'ts', 'html', 'svelte', 'md', 'sh'] as const;
export const sample_langs = ['json', 'css', 'ts', 'rs', 'html', 'svelte', 'md', 'sh'] as const;

export type SampleLang = (typeof sample_langs)[number];
2 changes: 2 additions & 0 deletions src/lib/lexer.ts
Original file line number Diff line number Diff line change
Expand Up @@ -480,6 +480,8 @@ export const matches_ci = (text: string, from: number, word: string): boolean =>

export const is_digit = (c: number): boolean => c >= 48 && c <= 57;

export const is_upper = (c: number): boolean => c >= 65 && c <= 90;

export const is_ascii_alnum = (c: number): boolean =>
(c >= 48 && c <= 57) || ((c | 0x20) >= 97 && (c | 0x20) <= 122);

Expand Down
2 changes: 2 additions & 0 deletions src/lib/lexer_md.ts
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ const T_LANG_JSON = token_type('lang_json');
const T_LANG_SVELTE = token_type('lang_svelte');
const T_LANG_BASH = token_type('lang_bash');
const T_LANG_MD = token_type('lang_md');
const T_LANG_RUST = token_type('lang_rust');
const T_HEADING = token_type('heading');
const T_HEADING_PUNCTUATION = token_type('punctuation', 'heading_punctuation');
const T_BLOCKQUOTE = token_type('blockquote');
Expand Down Expand Up @@ -96,6 +97,7 @@ add_fence_lang('json', 'json', T_LANG_JSON);
add_fence_lang('svelte', 'svelte', T_LANG_SVELTE);
add_fence_lang('sh bash shell', 'sh', T_LANG_BASH);
add_fence_lang('md markdown', 'md', T_LANG_MD);
add_fence_lang('rust rs', 'rust', T_LANG_RUST);

// line-internal whitespace — `is_space` from the toolkit includes newlines
const is_line_space = (c: number): boolean => c === 32 || c === 9;
Expand Down
Loading