diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2b699f61031..ba215a9d1d8 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -93,6 +93,8 @@ The historical samples locate candidates; they are not controlled benchmarks. - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one - [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules - [One nextest build in the queue](/quest/m0/ci/one-test-build.md) - `test rs` compiles once instead of three times +- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands +- [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands diff --git a/quest/m0/ci/obs-parallel.md b/quest/m0/ci/obs-parallel.md new file mode 100644 index 00000000000..473c36223e5 --- /dev/null +++ b/quest/m0/ci/obs-parallel.md @@ -0,0 +1,40 @@ +# [S] Parallelize OBS source checks + +## Goal + +When OBS's syntax checks are on a queue entry's critical path after modules run +in parallel, they check independent translation units concurrently after +preparing their shared headers, without losing diagnostics or exceeding the +fanout bound modules share. Failure and cancellation leave no child checks +running. + +## Plan + +Decided 2026-10-23: module-level parallelism lands first, and this adds +translation-unit fanout inside `obs_compile` only where it still shortens the +entry. + +- `sh/obs/compile.sh` checks 20 source and test translation units sequentially. + The complete OBS compile module took 46 seconds in + [run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) + and 92 seconds in + [run 39993252909](https://github.com/moq-dev/moq/actions/runs/39993252909), + including header preparation, which these samples do not separate. +- Once modules overlap, measure whether `obs_compile` ends an entry, such as + one touching only `cpp/obs/`. If it never does, delete this quest. +- Finish `sh/obs/includes.sh` and its generated C++ package preparation before + starting concurrent syntax checks. Preserve all source/test coverage, + compiler flags, per-source diagnostics, and the aggregate failure result. +- Take concurrency from the bound parallel modules and local admission share; + do not create a second scheduling policy or unbounded fanout. +- Verify a failing source makes the command fail, all sources are checked, and + interruption reaps children. Compare serial and parallel runs on the same + inputs, reporting header preparation separately from compilation. + +## Required + +- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - decides whether OBS stays on the critical path, and owns the fanout bound + +## Related + +- [Share local check capacity](/quest/m0/ci/local-gate.md) - coordinate internal fanout with the shared local command budget diff --git a/quest/m0/ci/parallel-modules.md b/quest/m0/ci/parallel-modules.md index e0d986107ca..78f0261424d 100644 --- a/quest/m0/ci/parallel-modules.md +++ b/quest/m0/ci/parallel-modules.md @@ -15,7 +15,14 @@ overlap the Rust build, an estimated 45% off those entries. - Bound concurrency so modules do not starve each other, and keep each module's output readable when they interleave. -- A failure still fails the run and names its module. +- A failure still fails the run and names its module, and cancels the + modules still running, so a cheap check that fails ends the entry in its own + time. In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) + the tooling tests failed after 18 s, but only after about 16 minutes of + modules ahead of them. Verify an injected `just gh test` failure ends the run + before the Rust build finishes, and that cancellation leaves no module + running. Decided 2026-10-23: this replaces a separate quest to move cheap + checks ahead of builds in the serial loop. Public API: none. Wire: none. diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md new file mode 100644 index 00000000000..7258403c16f --- /dev/null +++ b/quest/m0/ci/python-setup.md @@ -0,0 +1,32 @@ +# [S] Share Python check and test preparation + +## Goal + +A CI pass that selects Python prepares its editable bindings and environment +once for both checks and tests. Direct `just py check` and `just py test` +remain self-contained and validate the current checkout. + +## Plan + +Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage +and the `testing,openh264` feature contract. + +- `py/justfile` currently synchronizes the environment and runs `_develop` + in both recipes. `sh/dispatch.sh` separates them with other binding checks + that also build moq-ffi through Cargo with other features. +- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), + the second preparation rebuilt the UniFFI generator for 43.70 seconds, + in addition to native build and installation work. Repeated compilation + is observed; the exact cache-invalidation cause is not yet established. +- Share preparation within one invocation rather than trusting an old + environment or a persistent success marker. Keep standalone recipes safe + after Rust, binding, feature, or dependency changes, and stop on preparation + failure. Prefer the smallest recipe/dispatch refactor that achieves this. +- Other languages use different features and build environments. This quest + does not unify every native artifact; running modules concurrently belongs + to [parallel modules](/quest/m0/ci/parallel-modules.md). +- Verify one preparation in the combined pass, standalone checks and tests + from a fresh environment, and rebuilds after relevant input changes. + Measure combined Python and full selected-gate time before and after on + matched inputs and cache state; report build counts and memory use. The + observed 44 seconds is a candidate saving, not a guaranteed improvement.