From 4f8b4c4fc343528228a0d6b8c5c4a2f5b3d5ce36 Mon Sep 17 00:00:00 2001 From: Luke Curley <432854+kixelated@users.noreply.github.com> Date: Fri, 23 Oct 2026 13:36:26 -0700 Subject: [PATCH 1/2] docs(quest): plan merge queue latency improvements Record early tooling checks, shared Python preparation, and bounded OBS compilation with observed timings and validation criteria. Co-Authored-By: GPT-6 --- quest/m0/ci/README.md | 3 +++ quest/m0/ci/obs-parallel.md | 34 ++++++++++++++++++++++++++++++++++ quest/m0/ci/python-setup.md | 33 +++++++++++++++++++++++++++++++++ quest/m0/ci/tooling-first.md | 33 +++++++++++++++++++++++++++++++++ 4 files changed, 103 insertions(+) create mode 100644 quest/m0/ci/obs-parallel.md create mode 100644 quest/m0/ci/python-setup.md create mode 100644 quest/m0/ci/tooling-first.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index d8401a348f6..e8af2d51f7b 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -80,6 +80,9 @@ The historical samples locate candidates; they are not controlled benchmarks. ## Required +- [Fail early on repository tooling errors](/quest/m0/ci/tooling-first.md) - cheap checks reject a broken queue entry before expensive builds +- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands +- [Bound and parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently within the available resource budget - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin diff --git a/quest/m0/ci/obs-parallel.md b/quest/m0/ci/obs-parallel.md new file mode 100644 index 00000000000..de9c536d0e5 --- /dev/null +++ b/quest/m0/ci/obs-parallel.md @@ -0,0 +1,34 @@ +# [S] Bound and parallelize OBS source checks + +## Goal + +OBS syntax checks use available CPU concurrently after preparing their shared +headers, reducing queue latency without losing diagnostics or exceeding the +runner's resource budget. Failure and cancellation leave no child checks running. + +## Plan + +Decided 2026-10-23: parallelize independent translation units rather than +whole binding suites that share mutable Cargo outputs. + +- `sh/obs/compile.sh` checks 20 source and test translation units sequentially. + The complete OBS compile module took 46 seconds in + [run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) + and 92 seconds in + [run 39993252909](https://github.com/moq-dev/moq/actions/runs/39993252909). + These include header preparation and different workloads, so they identify + a candidate rather than establish an expected parallel speedup. +- Finish `sh/obs/includes.sh` and its generated C++ package preparation before + starting concurrent syntax checks. Preserve all source/test coverage, + compiler flags, per-source diagnostics, and the aggregate failure result. +- Choose bounded concurrency from measurements under the effective CPU and + memory limits, including competing work. Honor repository command admission + when present; do not create a second scheduling policy or unbounded fanout. +- Verify a failing source makes the command fail, all selected sources are + checked, and interruption reaps children. Compare serial and bounded + parallel runs on the same inputs, reporting header preparation separately + from compilation, wall time, peak memory, and impact on concurrent jobs. + +## Related + +- [Share local check capacity](/quest/m0/ci/local-gate.md) - coordinate internal fanout with the shared local command budget diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md new file mode 100644 index 00000000000..cbeba1f6e78 --- /dev/null +++ b/quest/m0/ci/python-setup.md @@ -0,0 +1,33 @@ +# [S] Share Python check and test preparation + +## Goal + +A CI pass that selects Python prepares its editable bindings and environment +once for both checks and tests. Direct `just py check` and `just py test` +remain self-contained and validate the current checkout. + +## Plan + +Decided 2026-10-23: remove repeated Python preparation before attempting +parallel execution of whole binding suites. Keep existing coverage and the +`testing,openh264` feature contract. + +- `py/justfile` currently synchronizes the environment and runs `_develop` + in both recipes. `sh/dispatch.sh` separates them with other binding checks + that also build through Cargo. +- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), + the second preparation rebuilt the UniFFI generator for 43.70 seconds, + in addition to native build and installation work. Repeated compilation + is observed; the exact cache-invalidation cause is not yet established. +- Share preparation within one invocation rather than trusting an old + environment or a persistent success marker. Keep standalone recipes safe + after Rust, binding, feature, or dependency changes, and stop on preparation + failure. Prefer the smallest recipe/dispatch refactor that achieves this. +- Other languages use different features and build environments. This quest + does not unify every native artifact or run suites concurrently against + shared mutable Cargo outputs. +- Verify one preparation in the combined pass, standalone checks and tests + from a fresh environment, and rebuilds after relevant input changes. + Measure combined Python and full selected-gate time before and after on + matched inputs and cache state; report build counts and memory use. The + observed 44 seconds is a candidate saving, not a guaranteed improvement. diff --git a/quest/m0/ci/tooling-first.md b/quest/m0/ci/tooling-first.md new file mode 100644 index 00000000000..9f2544ffa35 --- /dev/null +++ b/quest/m0/ci/tooling-first.md @@ -0,0 +1,33 @@ +# [S] Fail early on repository tooling errors + +## Goal + +Every merge-queue Check runs the cheap repository checks and tooling tests +before expensive builds, so a tooling regression releases the runner quickly. +Keep the same selected coverage and run each check once. + +## Plan + +Decided 2026-10-23: prioritize earlier failure feedback. This changes the cost +of failing runs; it does not promise a faster successful gate. + +- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), + the tooling tests failed after 18 seconds, but only after about 16 minutes + of preceding work. The selector fix landed in + [#6555](https://github.com/moq-dev/moq/pull/6555). +- The full `just gh test` suite passed locally in 36 seconds with + `GITHUB_ACTIONS=true` on `b2899cc08` plus the benchmark-coverage correction + already included in [#6480](https://github.com/moq-dev/moq/pull/6480). + This is a diagnostic sample, not a controlled runner benchmark. +- Start from `sh/dispatch.sh` and `.github/justfile`. Put cheap repository + lints and tooling tests ahead of builds while preserving upfront tool + validation, path selection, local entrypoints, and timing reports. Avoid + duplicating the tooling tests introduced by the prerequisite. +- Verify that an injected tooling failure prevents expensive work from + starting, and that successful dispatch still runs every selected check + exactly once. Compare time to failure and successful gate duration on the + same runner and cache state; record both in the implementation PR. + +## Required + +- [Tooling tests gate every merge](/quest/m1/gh-test-gate.md) - establish the always-selected tooling test gate before changing its order From c1f74b7a3a42c415ea48075da746445065b86faf Mon Sep 17 00:00:00 2001 From: Luke Curley Date: Fri, 23 Oct 2026 20:13:11 -0700 Subject: [PATCH 2/2] quest: align queue latency quests with the structural plan Fold the cheap-checks-first quest into parallel modules, whose fail-fast cancellation delivers the same early failure, and drop its link to the completed tooling-test gate. Block OBS translation-unit fanout on parallel modules and keep it only if OBS stays on the critical path. Rank both remaining quests after the structural changes main prioritized. Co-Authored-By: Claude Opus 5.5 --- quest/m0/ci/README.md | 5 ++--- quest/m0/ci/obs-parallel.md | 38 +++++++++++++++++++-------------- quest/m0/ci/parallel-modules.md | 9 +++++++- quest/m0/ci/python-setup.md | 11 +++++----- quest/m0/ci/tooling-first.md | 33 ---------------------------- 5 files changed, 37 insertions(+), 59 deletions(-) delete mode 100644 quest/m0/ci/tooling-first.md diff --git a/quest/m0/ci/README.md b/quest/m0/ci/README.md index 2f21685ef75..ba215a9d1d8 100644 --- a/quest/m0/ci/README.md +++ b/quest/m0/ci/README.md @@ -88,14 +88,13 @@ The historical samples locate candidates; they are not controlled benchmarks. ## Required -- [Fail early on repository tooling errors](/quest/m0/ci/tooling-first.md) - cheap checks reject a broken queue entry before expensive builds -- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands -- [Bound and parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently within the available resource budget - [Require the release gate](/quest/m0/ci/release-gates.md) - condition: the next cut carries `release-gate.yml`; then release PRs require it - [Share local check capacity](/quest/m0/ci/local-gate.md) - gate expensive phases and bound internal fanout - [Build queue entries concurrently](/quest/m0/ci/queue-concurrency.md) - the queue builds two or more entries at once instead of one - [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - `just ci` overlaps its selected modules - [One nextest build in the queue](/quest/m0/ci/one-test-build.md) - `test rs` compiles once instead of three times +- [Share Python check and test preparation](/quest/m0/ci/python-setup.md) - prepare editable bindings once per combined CI pass while preserving standalone commands +- [Parallelize OBS source checks](/quest/m0/ci/obs-parallel.md) - check independent translation units concurrently when OBS is on the critical path - [Measure nightly compiler levers](/quest/m0/ci/nightly-levers.md) - the parallel frontend and Cranelift, adopted only if the numbers beat their risks - [Bump to the manifest fix](/quest/m0/ci/mbx-bump.md) - condition: an mbx release contains jdx/mr-boxington#798; then bump the Rust cache's pin - [Verify the R2 Rust cache](/quest/m0/ci/mbx-r2-verify.md) - measure the hosted Nightly jobs' hits, restore time, and expiry once the bump lands diff --git a/quest/m0/ci/obs-parallel.md b/quest/m0/ci/obs-parallel.md index de9c536d0e5..473c36223e5 100644 --- a/quest/m0/ci/obs-parallel.md +++ b/quest/m0/ci/obs-parallel.md @@ -1,33 +1,39 @@ -# [S] Bound and parallelize OBS source checks +# [S] Parallelize OBS source checks ## Goal -OBS syntax checks use available CPU concurrently after preparing their shared -headers, reducing queue latency without losing diagnostics or exceeding the -runner's resource budget. Failure and cancellation leave no child checks running. +When OBS's syntax checks are on a queue entry's critical path after modules run +in parallel, they check independent translation units concurrently after +preparing their shared headers, without losing diagnostics or exceeding the +fanout bound modules share. Failure and cancellation leave no child checks +running. ## Plan -Decided 2026-10-23: parallelize independent translation units rather than -whole binding suites that share mutable Cargo outputs. +Decided 2026-10-23: module-level parallelism lands first, and this adds +translation-unit fanout inside `obs_compile` only where it still shortens the +entry. - `sh/obs/compile.sh` checks 20 source and test translation units sequentially. The complete OBS compile module took 46 seconds in [run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) and 92 seconds in - [run 39993252909](https://github.com/moq-dev/moq/actions/runs/39993252909). - These include header preparation and different workloads, so they identify - a candidate rather than establish an expected parallel speedup. + [run 39993252909](https://github.com/moq-dev/moq/actions/runs/39993252909), + including header preparation, which these samples do not separate. +- Once modules overlap, measure whether `obs_compile` ends an entry, such as + one touching only `cpp/obs/`. If it never does, delete this quest. - Finish `sh/obs/includes.sh` and its generated C++ package preparation before starting concurrent syntax checks. Preserve all source/test coverage, compiler flags, per-source diagnostics, and the aggregate failure result. -- Choose bounded concurrency from measurements under the effective CPU and - memory limits, including competing work. Honor repository command admission - when present; do not create a second scheduling policy or unbounded fanout. -- Verify a failing source makes the command fail, all selected sources are - checked, and interruption reaps children. Compare serial and bounded - parallel runs on the same inputs, reporting header preparation separately - from compilation, wall time, peak memory, and impact on concurrent jobs. +- Take concurrency from the bound parallel modules and local admission share; + do not create a second scheduling policy or unbounded fanout. +- Verify a failing source makes the command fail, all sources are checked, and + interruption reaps children. Compare serial and parallel runs on the same + inputs, reporting header preparation separately from compilation. + +## Required + +- [Run the queue's modules in parallel](/quest/m0/ci/parallel-modules.md) - decides whether OBS stays on the critical path, and owns the fanout bound ## Related diff --git a/quest/m0/ci/parallel-modules.md b/quest/m0/ci/parallel-modules.md index e0d986107ca..78f0261424d 100644 --- a/quest/m0/ci/parallel-modules.md +++ b/quest/m0/ci/parallel-modules.md @@ -15,7 +15,14 @@ overlap the Rust build, an estimated 45% off those entries. - Bound concurrency so modules do not starve each other, and keep each module's output readable when they interleave. -- A failure still fails the run and names its module. +- A failure still fails the run and names its module, and cancels the + modules still running, so a cheap check that fails ends the entry in its own + time. In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979) + the tooling tests failed after 18 s, but only after about 16 minutes of + modules ahead of them. Verify an injected `just gh test` failure ends the run + before the Rust build finishes, and that cancellation leaves no module + running. Decided 2026-10-23: this replaces a separate quest to move cheap + checks ahead of builds in the serial loop. Public API: none. Wire: none. diff --git a/quest/m0/ci/python-setup.md b/quest/m0/ci/python-setup.md index cbeba1f6e78..7258403c16f 100644 --- a/quest/m0/ci/python-setup.md +++ b/quest/m0/ci/python-setup.md @@ -8,13 +8,12 @@ remain self-contained and validate the current checkout. ## Plan -Decided 2026-10-23: remove repeated Python preparation before attempting -parallel execution of whole binding suites. Keep existing coverage and the -`testing,openh264` feature contract. +Decided 2026-10-23: remove repeated Python preparation. Keep existing coverage +and the `testing,openh264` feature contract. - `py/justfile` currently synchronizes the environment and runs `_develop` in both recipes. `sh/dispatch.sh` separates them with other binding checks - that also build through Cargo. + that also build moq-ffi through Cargo with other features. - In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), the second preparation rebuilt the UniFFI generator for 43.70 seconds, in addition to native build and installation work. Repeated compilation @@ -24,8 +23,8 @@ parallel execution of whole binding suites. Keep existing coverage and the after Rust, binding, feature, or dependency changes, and stop on preparation failure. Prefer the smallest recipe/dispatch refactor that achieves this. - Other languages use different features and build environments. This quest - does not unify every native artifact or run suites concurrently against - shared mutable Cargo outputs. + does not unify every native artifact; running modules concurrently belongs + to [parallel modules](/quest/m0/ci/parallel-modules.md). - Verify one preparation in the combined pass, standalone checks and tests from a fresh environment, and rebuilds after relevant input changes. Measure combined Python and full selected-gate time before and after on diff --git a/quest/m0/ci/tooling-first.md b/quest/m0/ci/tooling-first.md deleted file mode 100644 index 9f2544ffa35..00000000000 --- a/quest/m0/ci/tooling-first.md +++ /dev/null @@ -1,33 +0,0 @@ -# [S] Fail early on repository tooling errors - -## Goal - -Every merge-queue Check runs the cheap repository checks and tooling tests -before expensive builds, so a tooling regression releases the runner quickly. -Keep the same selected coverage and run each check once. - -## Plan - -Decided 2026-10-23: prioritize earlier failure feedback. This changes the cost -of failing runs; it does not promise a faster successful gate. - -- In [queue run 39996587979](https://github.com/moq-dev/moq/actions/runs/39996587979), - the tooling tests failed after 18 seconds, but only after about 16 minutes - of preceding work. The selector fix landed in - [#6555](https://github.com/moq-dev/moq/pull/6555). -- The full `just gh test` suite passed locally in 36 seconds with - `GITHUB_ACTIONS=true` on `b2899cc08` plus the benchmark-coverage correction - already included in [#6480](https://github.com/moq-dev/moq/pull/6480). - This is a diagnostic sample, not a controlled runner benchmark. -- Start from `sh/dispatch.sh` and `.github/justfile`. Put cheap repository - lints and tooling tests ahead of builds while preserving upfront tool - validation, path selection, local entrypoints, and timing reports. Avoid - duplicating the tooling tests introduced by the prerequisite. -- Verify that an injected tooling failure prevents expensive work from - starting, and that successful dispatch still runs every selected check - exactly once. Compare time to failure and successful gate duration on the - same runner and cache state; record both in the implementation PR. - -## Required - -- [Tooling tests gate every merge](/quest/m1/gh-test-gate.md) - establish the always-selected tooling test gate before changing its order