Repository navigation
perf: avoid per-node promises, streams and iterators when linking and indexing references - #2273
Conversation
`interruptAndCheck` is async, so every call creates a promise and every await costs a microtask, although it only yields once per interruption period and only throws once the token is cancelled. `interruptionDue` evaluates the same conditions synchronously, so that loops can await `interruptAndCheck` only when it would yield or throw. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
… indexing references `DefaultLinker.link` and `DefaultReferenceDescriptionProvider.createDescriptions` walked the AST with `streamAst`, awaited `interruptAndCheck` for every node and streamed its references with `streamReferences`. That costs a promise, two `Object.keys` arrays, two streams and an iterator result per child and per reference for every AST node, twice per document build. The new `AstUtils.walkReferences` visits the same references in the same order with an explicit stack, awaits `interruptAndCheck` only when `interruptionDue` says it would yield or throw, and allocates only the `ReferenceInfo` per reference. It reads the children of a node when the node is visited, so the callback must not add, remove or replace contained AST nodes; linking and indexing don't. The profiled path of `link` is unchanged. The same pattern as in the validation walk (eclipse-langium#2264). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
…d interruptionDue Both functions now read the "interruption period elapsed" condition from one private helper, so that they can't drift apart. `interruptAndCheck` behaves as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
…ode in document order `pushChildrenInDocumentOrder` pushes the children of a node onto an explicit stack and reverses that part of the stack, so that they are popped in the order of `streamContents`, without a temporary array. It is meant for the stack-based walks of the validator and, later, the linker and the reference index, so that they share one implementation. The helper lives in the new module `utils/ast-walk.ts`, which is not re-exported from `utils/index.ts`, so it does not become part of `AstUtils` or the public API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
…Order `walkReferences` moves from `ast-utils.ts` to the new module `utils/walk-references.ts`, which is not re-exported from `utils/index.ts`. So it is no longer part of `AstUtils`, and its restriction that `action` must not add, remove or replace contained AST nodes doesn't become public API. `DefaultLinker` and `DefaultReferenceDescriptionProvider` import it from there. It now pushes the children of a node with the shared internal helper `pushChildrenInDocumentOrder`, like the validation walk. The order of the visited references is unchanged. The tests import `walkReferences` and everything they combine it with from the sources, so that they share one instance of `promise-utils`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
|
Rough benchmark numbers for this branch vs.
Small in this benchmark, and the time difference is within its noise (about ±3 % between runs). Each of its builds allocates about 41 GB and takes about 19 s, mostly because of the deliberately unresolved references; the walk itself is a small share of that. On a workspace where linking and indexing visit many more AST nodes per reference, the two removed per-node loops were about 1.5 GB per full build each (see the description). 🤖 Generated with Claude Code |
msujew
left a comment
There was a problem hiding this comment.
Thanks, implementation looks good, just some comments on API - the functions currently are not usable by adopters of Langium.
…tUtils Both helpers were kept in internal modules, so adopters could not use them. Move them into ast-utils.ts, so that they are exported through AstUtils, and drop the internal ast-walk.ts and walk-references.ts modules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m
Refs #2264 (same pattern, in the linker and the reference index)
Contains the first three commits of #2269 (
interruptionDue, the shared interruption period check, andpushChildrenInDocumentOrder), so it can land first; #2269 then only adds the validation walk on top.DefaultLinker.linkandDefaultReferenceDescriptionProvider.createDescriptionsboth run this loop over every document:For every AST node that costs a promise and an async resumption, two
Object.keysarrays, two streams with their closures and iterator state, and an iterator result per child and per reference. A document build runs it twice per document.Changes:
AstUtils.walkReferences(root, action, cancelToken). It visits the same references in the same order as the loop above, with an explicit stack. It awaitsinterruptAndCheckonly wheninterruptionDuesays it would yield or throw, pushes the children of a node withpushChildrenInDocumentOrderfrom perf: avoid per-node promises, closures and arrays in the validation walk #2269 (the same helper as the validation walk), and allocates only theReferenceInfoper reference.walkReferencesandpushChildrenInDocumentOrderlive inast-utils.ts, so adopters can use them throughAstUtils.DefaultLinker.link(the path without profiler) andDefaultReferenceDescriptionProvider.createDescriptionsuse it. The profiled path oflinkis unchanged.interruptionDue, the shared period check andpushChildrenInDocumentOrdercome from perf: avoid per-node promises, closures and arrays in the validation walk #2269. The commits are identical in both branches, so they merge cleanly in either order; this PR movespushChildrenInDocumentOrderfrom perf: avoid per-node promises, closures and arrays in the validation walk #2269's internal module intoAstUtils.What stays the same:
$-prefixed properties are skipped as withObject.keysandstreamReferences.OperationCancelledat the same nodes as before. The remaining references of the node where the token was cancelled are still visited, as withstreamReferences(node).forEach.One assumption: the walk reads the children of a node when it visits the node, after the callback ran for the node's references.
streamAstreads each child only once the subtree of the previous child is done. The order is therefore the same as long as the callback doesn't add, remove or replace contained AST nodes. Linking and indexing don't; the JSDoc ofwalkReferencesstates it as a requirement foraction.Tests (
test/utils/walk-references.test.ts), on a grammar with single references, reference arrays, a multi-reference and nested nodes. The test importswalkReferences, and everything it combines it with, from the sources, so that all of them share one instance ofpromise-utils:walkReferencesandpushChildrenInDocumentOrderare part ofAstUtils;ReferenceInfos in the same order as thestreamAstloop, for the root and for every subtree;$-prefixed properties are skipped, as by thestreamAstloop;CancellationToken.Noneand with a token while no interruption is due;OperationCancelledbefore the first reference for a cancelled token, and at the next node when the token is cancelled during the walk;DefaultLinkerlinks every reference once, in thestreamAstorder, to the expected targets;DefaultReferenceDescriptionProvidercreates the same descriptions in the same order as with thestreamAstloop; both stop on cancellation.The linker and provider tests don't use
walkReferencesand also pass onmain.Effect: in a workspace of 9,795 documents of a large Langium DSL (Langium 4.4.2, the same walk as a service override), the full build allocated 1,549 MB less while linking and 1,545 MB less while indexing references, with identical link targets and diagnostics.
Benchmark numbers for
mainvs. this branch will follow in a comment.🤖 Generated with Claude Code
https://claude.ai/code/session_01XkvVhAAk5NRLH75qMHMc3m