Repository navigation

docs: document state-machine tokenizer architecture - #13

Merged

imrim12 merged 2 commits into

mainfrom

claude/docs-fsm-tokenizer

May 1, 2026

Merged

docs: document state-machine tokenizer architecture#13
imrim12 merged 2 commits into
mainfrom
claude/docs-fsm-tokenizer

Commits on Apr 27, 2026
- docs: document state-machine tokenizer architecture and benchmarks
  claude
  committed
  4bad109
Commits on May 1, 2026
- refactor(parser): collapse to a single Tokenizer, drop regex variant
  claude
  committed
  75d094f