Skip to content

refactor: move runtime capture, history and orchestration logic to Rust - #1114

Merged
kvlonge merged 12 commits into
mainfrom
feat/native-group-15
Oct 15, 2026
Merged

kvlonge merged 12 commits into
mainfrom
feat/native-group-15

Conversation

@kvlonge

@kvlonge kvlonge commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Why

Runtime and orchestration logic was exempt from the Rust move because it lived next to user-code or executor code; only the user-code shims, capture glue, public facades and third-party glue should stay Python.

Changes

  • Native compute log storage, command-output capture dispatcher, and SQLite and Postgres execution history backends; tees, logging handlers, public facades and driver glue stay Python.
  • Native dbt subprocess runner, JSON event stream, project init and profile/project validation; PyYAML parsing stays Python.
  • Native custom-rule orchestration: parallel phases, host dispatch and splitting, result cache, cancellation and CPU count. User rules still run only in the Python host; interrupting a split run now stops every host.
  • Native compile timings, built-in audit templates and loaded-code identity.
  • Multi-host coverage restored, including interrupt, payload publish failures and host fact equivalence.

Verification

  • 11,385 unit/integration tests passed; make check-ci passed with all optional dependencies available.
  • Compiler differential: 371 projects, unchanged goldens. Planner differential: 11 projects matched the main baseline.
  • Compile outputs and exit codes match the reference on synthetic and private compile-only projects.
  • Old and new history backends read each other's databases; compute logs, output capture, rule outputs and dbt output matched recorded captures.
  • Independent review: no correctness findings.

Move LocalFilesystemComputeLogStorage, metadata encoding and validation,
inventory and pruning into sqlbuild-cli's compute log module and expose the
storage as a native class. The replay path and the storage now share one
metadata encoder, inventory and prune implementation. Python keeps the stream
tees, the scoped capture and the public shapes.
Constructing a Python exception imports the exceptions module, which can release
the GIL. Doing that while holding the storage state mutex let another thread
take the GIL and block on the mutex, deadlocking a late diagnostic append
against storage close. Every Python call now happens after the guard is
dropped.
Move command-output text normalization, chunking, line buffering, the bounded
priority queue, record validation, wire JSON, the invocation context and the
delivery dispatcher into Rust. The dispatcher keeps Python threading primitives
so delivery still runs on a named Python daemon thread. Python keeps the
stream tees, the scope that installs them, and the public record shapes.
@kvlonge
kvlonge enabled auto-merge (squash) October 15, 2026 12:19
@kvlonge
kvlonge merged commit 845980a into main Oct 15, 2026
40 checks passed
@kvlonge
kvlonge deleted the feat/native-group-15 branch October 15, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant