Repository navigation
perf: share one ExpressionRuntime across DataFactoryElement evaluations - #182
Open
Florian BURELLI (florian-burelli) wants to merge 1 commit into
Open
Florian BURELLI (florian-burelli) wants to merge 1 commit into
Florian BURELLI (florian-burelli) wants to merge 1 commit into
Conversation
`DataFactoryElement.evaluate` built a new `ExpressionRuntime` on every
call, and each one compiles the Lark grammar of the expression language
again. The compile costs about 50 ms, roughly four times the evaluation
itself, so a test suite spent most of its time rebuilding the same parser.
Build the runtime once, lazily, and reuse it. It holds no per-evaluation
state: the transformer only keeps the compiled parser, and the evaluator
is a stateless facade over the .NET evaluator.
Measured on one expression (`@concat('a', 'b')`), 30 iterations:
before 65 ms per evaluation, after 12 ms. The test suite of this
repository goes from 18 s to 12 s.
Fixes microsoft#178
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Author
|
@microsoft-github-policy-service agree company="AWARE" |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #178
Problem
DataFactoryElement.evaluatebuilds a newExpressionRuntimeon every call, andExpressionRuntime.__init__builds anExpressionTransformer, which compiles the Lark grammar of the expression language again. On this machine the compile costs about 50 ms, roughly four times the evaluation itself, so suites with many expressions spend most of their time rebuilding the same parser.Change
_data_factory_element.py: a module-level_get_expression_runtime()behindfunctools.lru_cachebuilds the runtime once, lazily, andevaluatereuses it. The runtime holds no per-evaluation state: the transformer only keeps the compiled parser (DataFactoryExpressionTransformerandReconstructorare still created per call), and the evaluator is a stateless facade over the .NETEvaluator.ExpressionRuntimestays importable from_data_factory_element, and the factory looks it up at call time, so code that patches it there keeps working.Numbers
DataFactoryElement.evaluate,@concat('a', 'b'), mean of 30The remaining suite time is dominated by the ~25 tests that construct their own
ExpressionRuntime(); left untouched here.Tests
test_evaluate_datafactory_element.py: evaluating several elements constructsExpressionRuntimeonce and still returns the right results.Full suite: 222 passed. Ruff check and format clean.
🤖 Generated with Claude Code