Repository navigation
refactor: read analysis natively, plan SQL tests on a native thread and trim hot spots - #1118
Merged
Merged
Conversation
InferredNullability becomes a native StrEnum registered with the compiler enums, so Rust reads it through the typed registry instead of importing sqlbuild.compiler.lineage.types in thirteen places. Python callers import it from sqlbuild._native and the shape-only module is deleted. Stage captures keep the original captured shape name.
Semantic shapes, coverage, type recovery, completion, metadata and lineage read each compiled model's inferred columns, declared columns, star flags, unchecked outputs, lineage, binding validation and dynamic proof through typed accessors. Native model views answer from their owner's Rust facts without building Python projections; hand-built records are converted at the read.
…d and future SQL test planning ran on a Python threading.Thread whose target was the native planning function, reporting through a concurrent.futures.Future. It now runs on a Rust thread that attaches to the interpreter and sends its outcome over a channel; the planner and `start_sql_test_planning` read it through the native `SqlTestPlanning` handle. The atexit hook waits, without the GIL and for at most five seconds, until no planning thread runs, so finalization never ends one inside native code. The planning entry is still looked up on the module, so tests can replace it.
…the module Model plan entries, audit entries, relation contexts and cursor resolution looked up native helpers on `sqlbuild._native` and called them with keyword dictionaries once or more per model. They now call the Rust functions directly: authored config strings and cursor bounds, the model plan action, SQL, hooks, full-refresh and microbatch limit resolution, model plan entry assembly, audit selection, attachment kind, effective run scope and lineage reads. Results are unchanged.
Dialect non-code scanning is called once per byte by many SQL test and analysis scanners. Return early when the byte cannot open a block comment, line comment, dollar quote or quoted text, instead of running prefix comparisons at every offset. Results are unchanged.
… starts Lint and format site patterns can only match at '@', '$' or '_'; skip the anchored regex search at every other code byte.
Projection-name validation scanned each branch once per clause keyword. Scan once and stop at the first top-level FROM or clause keyword with the same name and IS [NOT] DISTINCT FROM rules.
The store's whole-body checksum already verifies the stdout record stored under its digest, so reading a project reuse entry no longer hashes the full compile output a second time. Persisted bytes are unchanged.
Add rustc-hash (already in the lock file) as the workspace fast hasher. Index hosted shape names once instead of scanning every shape per lookup, match UNION BY NAME outputs through a name index instead of quadratic scans, and use FxHash for hot sets and maps in function-name validation, CTE column rules, lint additional facts and SQL test identifier tokens. None of these maps' iteration order reaches output.
The parent pickled the whole host payload into one bytes object before writing it, holding a second copy of the payload (about 285 MB for a 10,000-model project) alive during the rules phase. Pickle straight into the partial file instead; the atomic rename and the payload the hosts read are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
After the port, Rust stages still read compiled-model analysis and ran SQL test planning through Python objects and threads, and a profile showed avoidable CPU in pure-Rust hot spots.
Changes
Verification