Repository navigation
refactor: host remaining Rust-only shapes, event export and janitor planning natively - #1115
Merged
Merged
Conversation
…ecords and errors natively
…tion relation naming to the executor
…ecord the staying compile view shapes
Add the destination-neutral export policy derived from the lifecycle catalogue, severity comparison and narrowing, exception-name sanitisation, the bounded finite-priority queue with displacement, per-sink routing and accounting snapshots to sqlbuild-core, with unit tests.
Move the event exporter dispatcher (priority queue, per-sink accounting, drain deadlines, invocation timeouts, failure notifications and periodic and final health summaries), the export policy, severity narrowing, error-name sanitisation and the command scope with its context variable into Rust. The dispatcher hosts its work on named Python daemon threads under a Python condition, joins its health thread at shutdown and stops its threads at interpreter exit. Each lifecycle sink call runs on a thread whose target is the Python invocation shim, so user code never runs above native frames that finalization would abort. Python keeps the public sink decorators and readers, the shapes re-exported by sqlbuild.sinks and the two shims that call user sink functions.
A command-output sink still running when the interpreter finalized ran above native frames on the capture worker thread, and CPython ending that daemon thread aborted the process. Each command-output sink call now runs on its own daemon thread whose target is the Python invocation shim, which reports completion back to the native export. Writes from those threads stay uncaptured. At exit, hooks set a shared exit signal that ends native waits, then stop and join the capture worker and the event exporter threads with a bounded wait. Delivered records and accounting are unchanged.
When planning failed before the SQL test plan was awaited, the background planning thread could still be running native code as the interpreter finalized, and CPython ending that daemon thread aborted the process instead of reporting the planning error. Execution planning now joins the planning thread on every exit path, and an exit hook joins any still-running planning thread with a bounded wait. Planned output is unchanged.
…ound a module shim
…cher Each command-output sink call started its own daemon thread, so a provider used only by output sinks that creates a thread-bound resource on first use (such as a SQLite connection) failed on the next record. All command-output sink calls for a dispatcher now run in order on a single long-lived thread whose target is a Python shim waiting on a Python queue, so no native frame is on that thread when interpreter finalization can end it. Records, ordering, failure counting, filtering and provider teardown are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Most remaining movable Python was data shapes used only by Rust, plus runtime logic (event export, janitor planning, discovery, Dagster parsing, loader records) that only had to stay near user code or the executor.
Changes
Verification