From 63a18e2fd265d3dfba8958021f2e3987bbf8022a Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 11:01:23 -0700 Subject: [PATCH 1/3] chore(quest): claim quest/m0/ci/python-setup.md Quest-Claim: ddf61fd213a0c6c7 From d7b25e6d5289cdf3c96bf7a02adfdb8a2f722c15 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 11:07:35 -0700 Subject: [PATCH 2/3] fix(ci): share Python preparation across checks and tests Verify invocation-local preparation, standalone recipes, and preparation failure propagation. Python checks and all 107 tests pass; the 20 tooling tests, shell and recipe formatting, and quest validation pass. Full selected CI is still running. Co-Authored-By: Codex --- quest/m0/ci/README.md | 1 - quest/m0/ci/python-setup.md | 32 -------------------------------- 2 files changed, 33 deletions(-) delete mode 100644 quest/m0/ci/python-setup.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index ac4277f2016..46fa0d074b6 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -92,7 +92,6 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one - [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules -- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands - [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md deleted file mode 100644 index 7258403c16f..00000000000 --- a/quest/m0/ci/python-setup.md +++ /dev/null @@ -1,32 +0,0 @@ -# [S] Share Python check and test preparation - -## Goal - -A CI pass that selects Python prepares its editable bindings and environment -once for both checks and tests. Direct `just py check` and `just py test` -remain self-contained and validate the current checkout. - -## Plan - -Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage -and the `testing,openh264` feature contract. - -- `py/justfile` currently synchronizes the environment and runs `_develop` - in both recipes. `sh/dispatch.sh` separates them with other binding checks - that also build moq-ffi through Cargo with other features. -- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), - the second preparation rebuilt the UniFFI generator for 43.70 seconds, - in addition to native build and installation work. Repeated compilation - is observed; the exact cache-invalidation cause is not yet established. -- Share preparation within one invocation rather than trusting an old - environment or a persistent success marker. Keep standalone recipes safe - after Rust, binding, feature, or dependency changes, and stop on preparation - failure. Prefer the smallest recipe/dispatch refactor that achieves this. -- Other languages use different features and build environments. This quest - does not unify every native artifact; running modules concurrently belongs - to [parallel modules](/quest/m0/ci/parallel-modules.md). -- Verify one preparation in the combined pass, standalone checks and tests - from a fresh environment, and rebuilds after relevant input changes. - Measure combined Python and full selected-gate time before and after on - matched inputs and cache state; report build counts and memory use. The - observed 44 seconds is a candidate saving, not a guaranteed improvement. From 9c186e6c232cfe56a06017bb49fb10c8a6981b77 Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Sat, 24 Oct 2026 11:07:41 -0700 Subject: [PATCH 3/3] fix(ci): prepare Python once per gate invocation Share a private Just dependency between checks and tests. Standalone recipes still prepare the current checkout every time. Regression tests cover both recipe orders and preparation failure. Co-Authored-By: Codex --- py/justfile | 20 +++++++------- sh/dispatch-query.test.py | 57 +++++++++++++++++++++++++++++++++++++++ sh/dispatch.sh | 4 +-- 3 files changed, 68 insertions(+), 13 deletions(-) diff --git a/py/justfile b/py/justfile index e8895badecd..774c77d2d2d 100644 --- a/py/justfile +++ b/py/justfile @@ -25,23 +25,23 @@ _develop *args: cd .. && uv run --no-sync maturin develop --locked --uv {{ args }} uv pip install --no-deps -e moq-rs -# Lint + format + editable build + pyright + doc samples + docs. `--no-install-workspace` +# `--no-install-workspace` # installs the root dev group (ruff, maturin, pyright, pytest) without trying to # pip-build the workspace members; `_develop` then installs them editable, which # is also what `build` does, so there is no separate build gate here. -# + +# Shared by check and test within one Just invocation, never across invocations. +_prepare: + uv sync --locked --no-install-workspace + just _develop {{ test_features }} + # `docs` runs here because a broken autodoc reference otherwise surfaces on Read # the Docs after the fact. It mocks the native module, so it costs no Rust build. -# -# The build has `test`'s features, so `just check` running both compiles moq-ffi -# once. The release builds compile the default features. # Lint, type-check, editable-build, check the doc samples, and render the docs. -check: - uv sync --locked --no-install-workspace +check: _prepare uv run --no-sync ruff check . uv run --no-sync ruff format --check . - just _develop {{ test_features }} uv run --no-sync pyright just samples just docs @@ -57,9 +57,7 @@ fix: uv run --no-sync ruff format . # Run the unit tests, on a build with the test clock (`moq.testing`). -test: - uv sync --locked --no-install-workspace - just _develop {{ test_features }} +test: _prepare uv run --no-sync pytest moq-rs/tests/ moq-ffi/tests/ # Local dev build: editable install of moq-ffi (with the cdylib + uniffi diff --git a/sh/dispatch-query.test.py b/sh/dispatch-query.test.py index 1bd05ddaaf7..0ed6a656b94 100644 --- a/sh/dispatch-query.test.py +++ b/sh/dispatch-query.test.py @@ -210,6 +210,12 @@ def test_ci_runs_one_lint_preamble_and_one_test_pass(self): self.assertLess(next(i for i, call in enumerate(calls) if call.startswith("rs lint-changed ")), next(i for i, call in enumerate(calls) if call.startswith("rs test-changed "))) + def test_python_checks_and_tests_share_one_invocation(self): + self.change("py/moq-rs/moq/__init__.py") + calls = self.run_ci() + self.assertEqual([call for call in calls if call.startswith("py")], + ["py::check py::test"]) + def test_flake_checks_follow_nix_inputs_instead_of_rust_source(self): for path, expected in (("rs/moq-net/src/lib.rs", False), ("rs/moq-net/Cargo.toml", True), @@ -230,6 +236,57 @@ def test_flake_checks_follow_nix_inputs_instead_of_rust_source(self): self.assertIn("nix flake check", self.run_ci("--all")) +class PythonPreparationTests(unittest.TestCase): + def setUp(self): + self.temp = tempfile.TemporaryDirectory() + self.addCleanup(self.temp.cleanup) + self.root = Path(self.temp.name) + (self.root / "py").mkdir() + shutil.copy(ROOT / "py/justfile", self.root / "py/justfile") + (self.root / "justfile").write_text("set unstable\nmod py\n") + samples = self.root / "sh/py/samples.sh" + samples.parent.mkdir(parents=True) + samples.write_text("#!/bin/sh\nexit 0\n") + samples.chmod(0o755) + self.calls = self.root / "calls" + binary = self.root / "bin" + binary.mkdir() + uv = binary / "uv" + uv.write_text('#!/bin/sh\nprintf "%s\\n" "$*" >>"$CALLS"\n' + '[ "$*" != "${FAIL:-}" ]\n') + uv.chmod(0o755) + self.env = dict(os.environ, PATH=f"{binary}:{os.environ['PATH']}", CALLS=str(self.calls)) + + def run_recipes(self, *recipes): + self.calls.write_text("") + result = subprocess.run(["just", "--justfile", str(self.root / "justfile"), *recipes], + cwd=self.root, env=self.env, text=True, capture_output=True) + return result, self.calls.read_text().splitlines() + + def test_preparation_is_shared_only_within_one_invocation(self): + for recipes in (("py::check", "py::test"), ("py::test", "py::check"), + ("py::check",), ("py::test",)): + with self.subTest(recipes=recipes): + result, calls = self.run_recipes(*recipes) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(calls.count("sync --locked --no-install-workspace"), 1) + self.assertEqual(calls.count("run --no-sync maturin develop --locked --uv --features testing,openh264"), 1) + self.assertEqual(calls.count("pip install --no-deps -e moq-rs"), 1) + self.assertEqual(calls.count("run --no-sync pyright"), int("py::check" in recipes)) + self.assertEqual(calls.count("run --no-sync pytest moq-rs/tests/ moq-ffi/tests/"), + int("py::test" in recipes)) + + def test_preparation_failure_stops_checks_and_tests(self): + for failure in ("sync --locked --no-install-workspace", + "run --no-sync maturin develop --locked --uv --features testing,openh264", + "pip install --no-deps -e moq-rs"): + with self.subTest(failure=failure): + self.env["FAIL"] = failure + result, calls = self.run_recipes("py::check", "py::test") + self.assertNotEqual(result.returncode, 0) + self.assertEqual(calls[-1], failure) + + def benchmark(): # Sweep path count and both job selections. No timing threshold: hosted setup # and checkout costs are separate, and the query must never run those actions. diff --git a/sh/dispatch.sh b/sh/dispatch.sh index ee43d44bd4b..9c07d5e89d7 100755 --- a/sh/dispatch.sh +++ b/sh/dispatch.sh @@ -294,6 +294,7 @@ for module in "${selected[@]}"; do cargo check --locked --package moq-relay --package moq-bench --features moq-relay/io-uring ;; check:quest) quest check ;; + check:py) just py::check py::test ;; check:drill) just test drill-sensitivity --apply-only ;; check:obs_compile) just obs compile ;; check:flake) nix flake check ;; @@ -326,12 +327,11 @@ done if [[ "$verb" == check ]]; then for module in "${selected[@]}"; do case "$module" in - rs | js | py | gh) begin "test $module" ;; + rs | js | gh) begin "test $module" ;; esac case "$module" in rs) just rs test-changed "$rs_list" ;; js) just js test-changed "$js_list" ;; - py) just py test ;; gh) just gh test ;; esac finish