Repository navigation
Conversation
`LangiumParser.parse(text, { buildCst: false })` builds only the AST: no
`$cstNode` on AST nodes, no `$refNode` on references, no hidden nodes.
The option is also available through `LangiumDocumentFactory.fromString`
and `fromTextDocument`, which already pass `ParserOptions` on.
During parsing, the parser reads the CST in three places. Without a CST:
- value conversion gets a reused stand-in node that only carries
`grammarSource`;
- "did the subrule consume a token" compares a counter of consumed valid
tokens, cut at the first assigned action like the CST rewiring does;
- datatype rules are converted with the invoking rule call, tracked on a
stack in `subrule`.
The option is only honoured if the value converter declares
`supportsCstFreeConversion`. `DefaultValueConverter` returns true for
the class itself only, so a subclass that may read more of the CST node
keeps getting a CST until it opts in.
`CstNodeBuilder.clear()` drops the previous tree; it is called before
every CST-free parse so the builder does not keep the last CST alive.
Refs eclipse-langium#2263
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
Both modes construct the previous node, push the new one and assign the previous node to the action's feature; only the CST rewiring (with a CST) and the cut of the enclosing subrule's token count (without a CST) differ. Without a CST, `last.$cstNode` is undefined, so the same `assign` call serves both. Also explain at `subruleWithoutCst` why only the first assigned action's cut counts. Refs eclipse-langium#2263 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
- Pass a new `{ grammarSource }` stand-in to the value converter for each
conversion instead of reusing one object, so a converter that keeps the
nodes it receives still sees the grammar element of each conversion.
- Assert the converted values in both modes, not only that both modes
agree, and pin the existing behaviour for datatype rules parsed as entry
rule: they return their text as a string in both modes, since no grammar
element invoked them.
- Document in `ParserOptions.buildCst` what a CST-free document lacks, that
the option falls back to building a CST unless the value converter opts
in, and how callers can check it.
- Comment the `CstNodeBuilder` invariant after `clear()`, and clarify how
the `DefaultValueConverter` opt-in behaves with duplicated package copies.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
|
Rough benchmark numbers for this branch vs.
The 🤖 Generated with Claude Code |
Refs #2263
Draft. This PR is here so there is concrete code to look at for #2263. It isn't meant to be merged as is. The open questions are at the end.
What it does
LangiumParser.parse(text, { buildCst: false })builds only the AST. AST nodes get no$cstNode, references get no$refNode, and hidden tokens (comments) are dropped. The AST, the lexer errors and the parser errors are the same as with a CST, including during error recovery. The option is off by default.LangiumDocumentFactory.fromStringandfromTextDocumentalready passParserOptionsto the parser, so documents can be created this way without further changes.parseHelperworks too, through itsparserOptions.While parsing,
LangiumParserreads the CST in three places. Without a CST, each one is replaced:consume:converter.convert(image, leafNode), which only readsleafNode.grammarSource{ grammarSource }, a new one per conversion.subrule: the result is only assigned ifcstNode.length > 0{infer X.left=current}) moves the subrule's composite node below a new node, so only tokens consumed before the first such action count. The counter is cut at the same point.constructfor datatype rules:convert(value, obj.$cstNode), which reads the composite'sgrammarSource(the invoking rule call)subrule.Also,
consumeskips the hidden-token handling,action()skips the CST rewiring, andbuildReferencereceivesundefinedasrefNode.The two points from the issue comment:
ValueConverter.supportsCstFreeConversion?: boolean(readonly, optional).DefaultValueConverterhas a getter that returnstrueonly when the instance's class isDefaultValueConverteritself. A subclass might read more of the CST node, for examplecstNode.textinrunConverter, so it has to opt in explicitly withoverride get supportsCstFreeConversion() { return true; }.parsebuilds the CST anyway unless the converter opts in. SobuildCst: falseis a request: with an existing custom converter it falls back to today's behaviour and can't cause runtime errors. The fallback happens without notice. TheParserOptions.buildCstJSDoc says so, and callers can check$cstNodeon the result. See open question 4.CstNodeBuilderkeeps the last tree until the nextbuildRootNode.CstNodeBuilder.clear()drops it, andparsecalls it at the start of every CST-free parse. Without this, a parser used in both modes would keep the last CST alive after its document is gone.The default path (no option, or
buildCst: true) behaves exactly as before. The code changes there only rearrangeconsumeandaction(), which now share their code with the CST-free path. The tests pin this (see below).What a CST-free document can and can't do
Works:
$containerlinks, andParseResulterrors, the same as with a CST.ref.refresolves as usual. ExportedAstNodeDescriptions have nonameSegmentorselectionSegment.getDiagnosticRange). This includes linking errors.JsonSerializerserializes the AST, without$textRegionor$sourceText.Doesn't work (needs the CST):
NameProvider.getNameNode,findNodeForProperty/findNodeForKeyword,CstUtils, andstreamAst(root, { range }), which skips nodes without a CST node.CommentProviderreturnsundefined, so documentation in hover and completion is missing too.generate/generator-tracing.ts) gets no source regions for these nodes.DefaultReferenceDescriptionProvider.createInfoDescriptionsskips references without$refNode. SoIndexManager.isAffecteddoesn't know what a CST-free document depends on, and an incrementalDocumentBuilder.updatedoesn't relink it when its targets change, unless it has linking errors. A test pins this current behaviour. See open question 1.DefaultLangiumDocumentFactory.updateuses the CST'sfullTextto skip re-parsing unchanged text. A CST-free document listed as changed is therefore always re-parsed. The re-parse goes throughAsyncParser, which has no options, so it builds a CST.API surface added
All of it is additive:
ParserOptions.buildCst?: booleanValueConverter.supportsCstFreeConversion?: booleanDefaultValueConverter.supportsCstFreeConversion(getter)CstNodeBuilder.clear()The rest is private to
LangiumParser.Tests
packages/langium/test/parser/build-cst-option.test.ts:Differential. Each input is parsed with and without a CST, and the test compares the ASTs (ignoring
$cstNode,$refNodeand container links; references compared by text; values tagged with their type) and the lexer errors, lexer report and parser errors (message, token, rule stack). It also checks that no AST node has a$cstNodeand no reference has a$refNode. The corpus:ID(including^),STRINGwith escapes,INT, andnumber,boolean,bigintandDateconversion, each through a terminal and through a datatype rule; nested datatype rules; keyword values;?=; single, multi and datatype-rule cross-references; a fragment;langium-grammar.langiumandlangium-types.langium, plus the test grammars, parsed with the grammar language.That makes 44 hand-written inputs (25 of them with syntax errors on purpose, e.g. the
eval + 3;cases where an assigned action follows an operand that consumed no valid token), plus 40 deterministic mutations of each (deletions, stray characters, truncations, duplicated slices): 1,760 mutated inputs, 1,512 of them with syntax errors. A datatype rule as entry rule ({ rule: 'QualifiedName' }) is covered too.Concrete values. Parity alone passes when both modes are wrong in the same way, so the converted values of every built-in kind are also asserted in each mode.
Datatype rule as entry rule. This pins existing behaviour that
buildCstdoesn't change. Parsed as entry rule ({ rule: 'NumberDT' }), a datatype rule returns its text as a string in both modes. With a CST, the converter gets the root CST node, which has no grammar element. Without a CST, no rule call invoked the rule. The same rule called from another rule converts to its type (number,boolean,bigint,Date). The terminals inside it are still converted (^and quotes removed). I checked this onmainbefore the change.Default unchanged. No option and
buildCst: truegive the same CST, AST and errors on the whole corpus. Inline snapshots of the CST (including leading, inner and trailing comments, an action and error recovery) were recorded onmainbefore the change and pass unchanged after it.Opt-in.
DefaultValueConverteropts in, a plain subclass doesn't, and a subclass can opt in with a getter. A subclass that readscstNode.textand doesn't opt in still gets a CST (same CST as a normal parse). So does a converter that only implements the interface. An opted-in converter receives only{ grammarSource }. A converter that keeps every node it receives still sees each node's own grammar element afterwards, in the same order as with a CST.Builder. After a CST parse followed by a CST-free parse, the builder holds no tree, and the earlier result's CST is intact. A CST-free parse in between doesn't change the next CST.
Documents. CST-free documents link, both as source and as target. The limitation of the reference index is pinned (see above).
Mutation checks: I broke each part in turn (in the compiled output) and ran the new test file. The last four rows were run after the review changes; parity tests stay green for all four, so only the new tests catch them:
consumepasses no grammar element to the convertersubrulealways assignsclear()not called / doesn't drop the rootnumberconversion off by one, in both modesbooleanconversion inverted, in both modesnumber, in both modesnpm run build,npm run lintandnpm test -- packages/langium/pass.A changelog entry will follow before this is merged.
Performance
Numbers on
mainwill follow. Benchmarks to run:packages/langium/test/parser/cst-parse-memory-benchmark.tsandparser-time-benchmark.ts(both need a{ buildCst: false }variant; today they parse with the default), andexamples/statemachine/test/workspace-build-benchmark.ts(would need the option passed through the document factory).For reference, the measurements from #2263 on a 7,137-document workspace (Langium 4.4.2, prototype,
--optimize-for-size, two full build-and-compile runs):buildCst: falseRetained heap after a build: about 131 MiB, against 345 MiB with the CST kept.
Open questions for review
Reference index and incremental builds. This question decides whether the option fits long-lived workspaces at all, so it may be worth settling before the parser-level details below. References without
$refNodearen't indexed, soIndexManager.isAffectedcan't see what a CST-free document depends on, and an incrementalDocumentBuilder.updatemisses it when its targets change (see above). That's fine for a one-shot batch build, but wrong for a long-lived workspace that applies updates. Options:ReferenceDescription.segmentoptional and index these references without a segment, which touches find-references and every consumer ofsegment;The post-parse discard many adopters do today (clearing
$cstNodeand$refNode) has the same gap.Scope of the option. A per-call
ParserOptionsflag, as here, or a parser mode in the services? The per-call flag lets one workspace mix documents with and without a CST, which is the use case in Parse without building a CST (buildCst: false) for headless batch use #2263: the documents that get compiled keep their CST. A service-level mode would be simpler to reason about, but it would need a second parser instance for mixed workspaces.The value converter contract. The stand-in is cast to
CstNode. Each conversion gets a new one, so a converter may keep it. The opt-in is a getter, and TypeScript makes a subclass override it with a getter: a field gives TS2610. Alternatives considered in the issue comment:The opt-in is also stricter than the most common case. Many languages subclass
DefaultValueConverteronly to overriderunConverter, and such a subclass silently gets a CST withbuildCst: false, because the getter returnstrueforDefaultValueConverteritself only. Possible remedies:buildCst: falseis requested but not honoured;convertValue(input, feature), withconvert(input, cstNode)delegating to it. Subclasses that only overriderunConverterwould then work without a CST. That's more API, though.Which do you prefer?
Silent fallback. If the converter doesn't opt in,
buildCst: falsebuilds a CST without notice. A batch job sized for CST-free memory then gets full CSTs. For now this is only documented in theParserOptions.buildCstJSDoc, and callers can check$cstNodeon the result. Should the parser report it, e.g. a field onParseResultsaying whether a CST was built, or a one-time log? This is tied to question 3: a remedy there would make the fallback rarer, but not remove it.Document lifecycle.
update()always re-parses a CST-free document when it is listed as changed, and the re-parse builds a CST, becauseAsyncParserand the worker-thread parser take no options. Should documents keep CST-free status across updates (e.g. the factory remembers the options per URI)? ShouldAsyncParsersupport the option?Consumer contract. Most of Langium already copes with
$cstNode === undefined, becausefromModeldocuments have none either. Where should the contract be documented: theParserOptionsJSDoc only, or the docs site?Equivalence of the subrule check. The token counter matches
cstNode.length > 0because a valid token always has a non-empty range (isValidTokenRange), and hidden nodes are only attached right before a consumed token. So a subrule's composite node can't hold hidden nodes alone. Are there cases I'm missing, e.g. custom token builders or parser subclasses that overrideconsume/subrule?Core or extension point? Instead of the option,
LangiumParsercould expose protected hooks around the CST builder calls, so adopters can implement CST-free parsing themselves. That's less to maintain in core, but it's a larger and more fragile API.Fastbelt. You mentioned Fastbelt as the long-term direction. Does that change whether this belongs in Langium 4.x?
Differences from the prototype
The prototype is a patch on the compiled 4.4.2 output. Compared with it:
consumeandaction()share their code between both modes instead of duplicating it.ruleCallStack,ruleCallStart,ruleCallCut).CstNodeBuilder.rootNodeis typed| undefined.main'slinkCstNodeneeded no change, since it is a no-op without$cstNode.DefaultValueConverter.runConverter), the build passes, and so dobuild-cst-option.test.ts, the value converter tests and all ofpackages/langium/.🤖 Generated with Claude Code
https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m