Skip to content

Add a buildCst: false parser option (draft for discussion) - #2272

Draft
cdietrich wants to merge 5 commits into
eclipse-langium:mainfrom
cdietrich:perf/2263-build-cst-option
Draft

cdietrich wants to merge 5 commits into
eclipse-langium:mainfrom
cdietrich:perf/2263-build-cst-option

Conversation

@cdietrich

@cdietrich cdietrich commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Refs #2263

Draft. This PR is here so there is concrete code to look at for #2263. It isn't meant to be merged as is. The open questions are at the end.

What it does

LangiumParser.parse(text, { buildCst: false }) builds only the AST. AST nodes get no $cstNode, references get no $refNode, and hidden tokens (comments) are dropped. The AST, the lexer errors and the parser errors are the same as with a CST, including during error recovery. The option is off by default.

LangiumDocumentFactory.fromString and fromTextDocument already pass ParserOptions to the parser, so documents can be created this way without further changes. parseHelper works too, through its parserOptions.

While parsing, LangiumParser reads the CST in three places. Without a CST, each one is replaced:

CST use during parsing Without a CST
consume: converter.convert(image, leafNode), which only reads leafNode.grammarSource The converter gets a stand-in object { grammarSource }, a new one per conversion.
subrule: the result is only assigned if cstNode.length > 0 A counter of consumed valid tokens, compared at subrule entry and exit. An assigned action ({infer X.left=current}) moves the subrule's composite node below a new node, so only tokens consumed before the first such action count. The counter is cut at the same point.
construct for datatype rules: convert(value, obj.$cstNode), which reads the composite's grammarSource (the invoking rule call) A stack of the invoking grammar elements, maintained in subrule.

Also, consume skips the hidden-token handling, action() skips the CST rewiring, and buildReference receives undefined as refNode.

The two points from the issue comment:

  1. Value converter opt-in. ValueConverter.supportsCstFreeConversion?: boolean (readonly, optional). DefaultValueConverter has a getter that returns true only when the instance's class is DefaultValueConverter itself. A subclass might read more of the CST node, for example cstNode.text in runConverter, so it has to opt in explicitly with override get supportsCstFreeConversion() { return true; }. parse builds the CST anyway unless the converter opts in. So buildCst: false is a request: with an existing custom converter it falls back to today's behaviour and can't cause runtime errors. The fallback happens without notice. The ParserOptions.buildCst JSDoc says so, and callers can check $cstNode on the result. See open question 4.
  2. Builder state. CstNodeBuilder keeps the last tree until the next buildRootNode. CstNodeBuilder.clear() drops it, and parse calls it at the start of every CST-free parse. Without this, a parser used in both modes would keep the last CST alive after its document is gone.

The default path (no option, or buildCst: true) behaves exactly as before. The code changes there only rearrange consume and action(), which now share their code with the CST-free path. The tests pin this (see below).

What a CST-free document can and can't do

Works:

  • Parsing, $container links, and ParseResult errors, the same as with a CST.
  • Exporting symbols, scope computation, linking: ref.ref resolves as usual. Exported AstNodeDescriptions have no nameSegment or selectionSegment.
  • Validation runs, but diagnostics without a CST fall back to the range 0:0–0:0 (getDiagnosticRange). This includes linking errors.
  • JsonSerializer serializes the AST, without $textRegion or $sourceText.

Doesn't work (needs the CST):

  • Anything that maps between AST and text: NameProvider.getNameNode, findNodeForProperty/findNodeForKeyword, CstUtils, and streamAst(root, { range }), which skips nodes without a CST node.
  • Comments: CommentProvider returns undefined, so documentation in hover and completion is missing too.
  • Formatting, and the LSP services that start from a position or return ranges: definition, references, hover, highlight, rename, semantic tokens, folding, document symbols, call and type hierarchy. Such documents aren't meant to be open in an editor.
  • Generator tracing (generate/generator-tracing.ts) gets no source regions for these nodes.
  • The reference index: DefaultReferenceDescriptionProvider.createInfoDescriptions skips references without $refNode. So IndexManager.isAffected doesn't know what a CST-free document depends on, and an incremental DocumentBuilder.update doesn't relink it when its targets change, unless it has linking errors. A test pins this current behaviour. See open question 1.
  • DefaultLangiumDocumentFactory.update uses the CST's fullText to skip re-parsing unchanged text. A CST-free document listed as changed is therefore always re-parsed. The re-parse goes through AsyncParser, which has no options, so it builds a CST.

API surface added

All of it is additive:

  • ParserOptions.buildCst?: boolean
  • ValueConverter.supportsCstFreeConversion?: boolean
  • DefaultValueConverter.supportsCstFreeConversion (getter)
  • CstNodeBuilder.clear()

The rest is private to LangiumParser.

Tests

packages/langium/test/parser/build-cst-option.test.ts:

  • Differential. Each input is parsed with and without a CST, and the test compares the ASTs (ignoring $cstNode, $refNode and container links; references compared by text; values tagged with their type) and the lexer errors, lexer report and parser errors (message, token, rule stack). It also checks that no AST node has a $cstNode and no reference has a $refNode. The corpus:

    • a value-conversion grammar: ID (including ^), STRING with escapes, INT, and number, boolean, bigint and Date conversion, each through a terminal and through a datatype rule; nested datatype rules; keyword values; ?=; single, multi and datatype-rule cross-references; a fragment;
    • an expression grammar with assigned actions, an optional subrule that can be empty, fragments and unassigned rule calls;
    • an infix-operator grammar;
    • Langium's own langium-grammar.langium and langium-types.langium, plus the test grammars, parsed with the grammar language.

    That makes 44 hand-written inputs (25 of them with syntax errors on purpose, e.g. the eval + 3; cases where an assigned action follows an operand that consumed no valid token), plus 40 deterministic mutations of each (deletions, stray characters, truncations, duplicated slices): 1,760 mutated inputs, 1,512 of them with syntax errors. A datatype rule as entry rule ({ rule: 'QualifiedName' }) is covered too.

  • Concrete values. Parity alone passes when both modes are wrong in the same way, so the converted values of every built-in kind are also asserted in each mode.

  • Datatype rule as entry rule. This pins existing behaviour that buildCst doesn't change. Parsed as entry rule ({ rule: 'NumberDT' }), a datatype rule returns its text as a string in both modes. With a CST, the converter gets the root CST node, which has no grammar element. Without a CST, no rule call invoked the rule. The same rule called from another rule converts to its type (number, boolean, bigint, Date). The terminals inside it are still converted (^ and quotes removed). I checked this on main before the change.

  • Default unchanged. No option and buildCst: true give the same CST, AST and errors on the whole corpus. Inline snapshots of the CST (including leading, inner and trailing comments, an action and error recovery) were recorded on main before the change and pass unchanged after it.

  • Opt-in. DefaultValueConverter opts in, a plain subclass doesn't, and a subclass can opt in with a getter. A subclass that reads cstNode.text and doesn't opt in still gets a CST (same CST as a normal parse). So does a converter that only implements the interface. An opted-in converter receives only { grammarSource }. A converter that keeps every node it receives still sees each node's own grammar element afterwards, in the same order as with a CST.

  • Builder. After a CST parse followed by a CST-free parse, the builder holds no tree, and the earlier result's CST is intact. A CST-free parse in between doesn't change the next CST.

  • Documents. CST-free documents link, both as source and as target. The limitation of the reference index is pinned (see above).

Mutation checks: I broke each part in turn (in the compiled output) and ran the new test file. The last four rows were run after the review changes; parity tests stay green for all four, so only the new tests catch them:

Broken Failing tests
consume passes no grammar element to the converter all differential tests, value kinds, opt-in stand-in
…only for tokens inside datatype rules Values differential (both), value kinds, datatype entry rule
subrule always assigns 8 tests: all corpora, empty optional subrule
action cut ignored Expressions differential (both), Langium mutated
token counter not incremented 13 tests
datatype rule converted without its rule call Values differential (both), value kinds
converter opt-in ignored "without the opt-in…", "interface only…"
subclasses inherit the opt-in "opts in for its own class only", "without the opt-in…"
clear() not called / doesn't drop the root "builder does not keep the tree"
default parse becomes CST-free 13 tests, including the snapshot pins
trailing comments dropped in the default path the snapshot pin
one stand-in object reused for every conversion "a converter that keeps the nodes it receives…"
number conversion off by one, in both modes value kinds, datatype entry rule pin
boolean conversion inverted, in both modes value kinds, datatype entry rule pin
entry datatype rules converted to number, in both modes datatype entry rule pin

npm run build, npm run lint and npm test -- packages/langium/ pass.

A changelog entry will follow before this is merged.

Performance

Numbers on main will follow. Benchmarks to run: packages/langium/test/parser/cst-parse-memory-benchmark.ts and parser-time-benchmark.ts (both need a { buildCst: false } variant; today they parse with the default), and examples/statemachine/test/workspace-build-benchmark.ts (would need the option passed through the document factory).

For reference, the measurements from #2263 on a 7,137-document workspace (Langium 4.4.2, prototype, --optimize-for-size, two full build-and-compile runs):

Allocated Promoted GC pause
CST built and kept (default) 4.4 GB 736 MiB 2.9 s
CST built, discarded after parse 4.0 GB 474 MiB 1.7 s
buildCst: false 3.0 GB 311 MiB 1.3 s

Retained heap after a build: about 131 MiB, against 345 MiB with the CST kept.

Open questions for review

  1. Reference index and incremental builds. This question decides whether the option fits long-lived workspaces at all, so it may be worth settling before the parser-level details below. References without $refNode aren't indexed, so IndexManager.isAffected can't see what a CST-free document depends on, and an incremental DocumentBuilder.update misses it when its targets change (see above). That's fine for a one-shot batch build, but wrong for a long-lived workspace that applies updates. Options:

    • document it as a limitation, i.e. the option is for one-shot batch builds only;
    • make ReferenceDescription.segment optional and index these references without a segment, which touches find-references and every consumer of segment;
    • track dependencies separately.

    The post-parse discard many adopters do today (clearing $cstNode and $refNode) has the same gap.

  2. Scope of the option. A per-call ParserOptions flag, as here, or a parser mode in the services? The per-call flag lets one workspace mix documents with and without a CST, which is the use case in Parse without building a CST (buildCst: false) for headless batch use #2263: the documents that get compiled keep their CST. A service-level mode would be simpler to reason about, but it would need a second parser instance for mixed workspaces.

  3. The value converter contract. The stand-in is cast to CstNode. Each conversion gets a new one, so a converter may keep it. The opt-in is a getter, and TypeScript makes a subclass override it with a getter: a field gives TS2610. Alternatives considered in the issue comment:

    • only documenting it: existing converters that read more of the node would break silently;
    • passing the grammar element instead of the CST node: breaks every converter;
    • a detached leaf per token: brings back much of the allocation this avoids.

    The opt-in is also stricter than the most common case. Many languages subclass DefaultValueConverter only to override runConverter, and such a subclass silently gets a CST with buildCst: false, because the getter returns true for DefaultValueConverter itself only. Possible remedies:

    • log a one-time warning when buildCst: false is requested but not honoured;
    • a separate method that takes the grammar element, e.g. convertValue(input, feature), with convert(input, cstNode) delegating to it. Subclasses that only override runConverter would then work without a CST. That's more API, though.

    Which do you prefer?

  4. Silent fallback. If the converter doesn't opt in, buildCst: false builds a CST without notice. A batch job sized for CST-free memory then gets full CSTs. For now this is only documented in the ParserOptions.buildCst JSDoc, and callers can check $cstNode on the result. Should the parser report it, e.g. a field on ParseResult saying whether a CST was built, or a one-time log? This is tied to question 3: a remedy there would make the fallback rarer, but not remove it.

  5. Document lifecycle. update() always re-parses a CST-free document when it is listed as changed, and the re-parse builds a CST, because AsyncParser and the worker-thread parser take no options. Should documents keep CST-free status across updates (e.g. the factory remembers the options per URI)? Should AsyncParser support the option?

  6. Consumer contract. Most of Langium already copes with $cstNode === undefined, because fromModel documents have none either. Where should the contract be documented: the ParserOptions JSDoc only, or the docs site?

  7. Equivalence of the subrule check. The token counter matches cstNode.length > 0 because a valid token always has a non-empty range (isValidTokenRange), and hidden nodes are only attached right before a consumed token. So a subrule's composite node can't hold hidden nodes alone. Are there cases I'm missing, e.g. custom token builders or parser subclasses that override consume/subrule?

  8. Core or extension point? Instead of the option, LangiumParser could expose protected hooks around the CST builder calls, so adopters can implement CST-free parsing themselves. That's less to maintain in core, but it's a larger and more fragile API.

  9. Fastbelt. You mentioned Fastbelt as the long-term direction. Does that change whether this belongs in Langium 4.x?

Differences from the prototype

The prototype is a patch on the compiled 4.4.2 output. Compared with it:

🤖 Generated with Claude Code

https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m

cdietrich and others added 3 commits November 17, 2026 13:20
`LangiumParser.parse(text, { buildCst: false })` builds only the AST: no
`$cstNode` on AST nodes, no `$refNode` on references, no hidden nodes.
The option is also available through `LangiumDocumentFactory.fromString`
and `fromTextDocument`, which already pass `ParserOptions` on.

During parsing, the parser reads the CST in three places. Without a CST:
- value conversion gets a reused stand-in node that only carries
  `grammarSource`;
- "did the subrule consume a token" compares a counter of consumed valid
  tokens, cut at the first assigned action like the CST rewiring does;
- datatype rules are converted with the invoking rule call, tracked on a
  stack in `subrule`.

The option is only honoured if the value converter declares
`supportsCstFreeConversion`. `DefaultValueConverter` returns true for
the class itself only, so a subclass that may read more of the CST node
keeps getting a CST until it opts in.

`CstNodeBuilder.clear()` drops the previous tree; it is called before
every CST-free parse so the builder does not keep the last CST alive.

Refs eclipse-langium#2263

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
Both modes construct the previous node, push the new one and assign the
previous node to the action's feature; only the CST rewiring (with a CST)
and the cut of the enclosing subrule's token count (without a CST) differ.
Without a CST, `last.$cstNode` is undefined, so the same `assign` call
serves both.

Also explain at `subruleWithoutCst` why only the first assigned action's
cut counts.

Refs eclipse-langium#2263

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
- Pass a new `{ grammarSource }` stand-in to the value converter for each
  conversion instead of reusing one object, so a converter that keeps the
  nodes it receives still sees the grammar element of each conversion.
- Assert the converted values in both modes, not only that both modes
  agree, and pin the existing behaviour for datatype rules parsed as entry
  rule: they return their text as a string in both modes, since no grammar
  element invoked them.
- Document in `ParserOptions.buildCst` what a CST-free document lacks, that
  the option falls back to building a CST unless the value converter opts
  in, and how callers can check it.
- Comment the `CstNodeBuilder` invariant after `clear()`, and clarify how
  the `DefaultValueConverter` opt-in behaves with duplicated package copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
@cdietrich

Copy link
Copy Markdown
Contributor Author

Rough benchmark numbers for this branch vs. main (da1beac): Node 24 on a laptop that wasn't idle, so ballpark figures rather than a careful benchmark. Allocation, scavenges and GC time are process totals, recorded with a small --import preload using v8.GCProfiler (allocation estimated as the heap before each GC minus the heap after the previous one).

packages/langium/test/parser/parser-time-benchmark.ts (~1.5 MB input, 30 measured parses; totals include 10 warm-up parses and 10 lexer runs), 2 runs each:

median parse allocated scavenges GC
main 625 ms 38.7 GB 629 8.5 s
this branch, default 626 ms 38.7 GB 630 8.2 s
this branch, buildCst: false 464 ms (−26 %) 31.4 GB (−19 %) 470 (−25 %) 2.7 s (−68 %)

The buildCst: false row comes from a local copy of the benchmark that passes { buildCst: false } to parse (not part of this PR); it builds 0 CST nodes instead of 722,305, with the same 194,910 AST nodes. The default path is unchanged. GC time drops most, because the CST no longer has to be traced while it is alive.

🤖 Generated with Claude Code

@cdietrich
cdietrich requested review from msujew and a balanced review from Copilot November 19, 2026 06:51

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants