Repository navigation
refactor: route native lookups through a typed registry and one host module - #1117
Merged
Merged
Conversation
Rust no longer imports sqlbuild modules by path. Classes, enums, errors and functions that sqlbuild._native registers are reached through records::registry (RecordId, ErrorId, NativeAttr), checked against the registration tables when the module initialises. Staying-Python entry points (adapter, user code, executor and runtime, third-party, public facade) are reached through the native_host table and sqlbuild._native_host, which resolves each name lazily. shape_classes.rs and hosted_native_name are removed. The native module path scanner checks every host table row.
Add `sqlbuild_core::graph_walk` with walks generic over the key type and the visited set: breadth-first reach (optional depth), directed path nodes, shortest path to a goal, Kahn order with a caller tie-break queue, depth-first post-order and depth-first cycle search. Replace the per-crate copies: closures and path nodes over DAG artifacts, the project graph, lineage commands, dbt graphs, Python selection, diff selection, Python-keyed graph edges, loader graphs, SQL-test CTE reads and helper scopes, macro export dependencies, rule import closures and type recovery; Kahn orders for planner keys, Dagster events and macro module loads; post-orders for loaders, SQL-test model chains and migration discovery; and cycle searches for Python nodes and macros. Each caller keeps its traversal order, tie-breaks and messages.
Scenario workers are daemon threads that run native scenario planning, which calls back into the adapter. When a run is abandoned, the main thread exits while a worker can still be inside that native frame, and finalization ending the thread there aborts the process with "exception not rethrown". Register an exit hook when workers start that joins the named scenario workers within a bounded time, so they leave native code first. Add a fresh-process regression test that abandons workers inside a native frame calling slow Python.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Rust looked up its own native classes and Python entry points by string at about 2,500 call sites, re-importing modules on every call; renaming a Python target only failed when that path ran. Graph walks were copied per caller, and abandoned scenario workers could abort the interpreter at exit.
Changes
sqlbuild._native_host: one lazily resolved host module listing every staying-Python entry point Rust may call, mirrored by one Rust table with typed per-edge wrappers; an import test fails on any rename.sqlbuild-corereplace about 30 per-caller copies (closures, paths, shortest path, dependency order, cycles) with each caller's ordering preserved.Verification